Evaluation Models

Types of Evaluation

Program evaluation consists of a set of practices that can be used to monitor and improve programs as they are being implemented (formative) and to assess the impact of programs as compared to the original goals over time or upon program completion (summative).

For CISE REU, evaluation operates on several levels, from the institutional to the national. The diagram below illustrates these layers.

• On the national level, the broad scope evaluation question might be: to what extent does research experience for undergraduate students in CISE areas lead them into graduate school?
• On the local level, the questions might be: to what extent are our participants motivated to continue in CISE majors and graduate school, and how effective is our REU site at engaging students in research? What programming elements work best for our site?

Summative and formative evaluation strategies need to be threaded throughout your site evaluation. The purpose of program evaluation at CISE REU sites is to examine how well the program has impacted retention in CISE majors and recruitment in CISE graduate programs.

Formative (program improvement):

• How is program implemented?
• Are activities delivered as intended? Fidelity of implementation?
• Are participants being reached as intended?
• What are participant reactions?

Summative (program outcomes and accountability):

• To what extent can changes be attributed to the program?
• What are the net effects?
• What are final consequences?

Impact evaluation:

• To what extent are desired changes occurring? Goals met?
• Who is benefiting/not benefiting? How?
• What seems to work? Not work?
• What are unintended outcomes?

[This section is an adaptation from the ADVANCE portal (http://www.portal.advance.vt.edu/), the NSF User-Friendly Handbook for Project Evaluation [pdf], and Assessing Campus Diversity Initiatives (See References: Garcia, et al.)]

Logic Models

A logic model is a conceptual framework that describes the pieces of the project and expected connections among them. A typical model has four broad categories of project elements that are connected by directional arrows: inputs, activities, short term outcomes, long term outcomes [NSF User-Friendly Handbook for project Evaluation].

• Project inputs= the various funding sources, resource streams and contributions that provide support to the project.
• Activities= the services, materials, actions, and events that characterize the project’s thrusts, or reach the target audience.
• Short-term impact= immediate results of the activities.
• Long-term outcomes= the broader and more enduring impacts on the system (i.e. individuals, groups, communities, and organizations).

These impacts will reflect CISE REU strategic outcomes discussed in the Evaluation & Reporting section entitled “NSF Required Indicators.” A logic model identifies these program elements and shows expected connections among them. PIs may find this model useful not only for evaluation but also for program management. It provides a framework for monitoring the flow of work and checking whether required activities are being put in place. [NSF User-Friendly Handbook for Project Evaluation]

University of Wisconsin Extension includes the following diagram and has a downloadable template at their website: UWEX. [NSF ADVANCE]

Logical Model in Evaluation