Chapter 8: Evaluation Concepts & Strategies
Evaluation Plan Phases
Evaluation Plan Phases
There are several critical phases you will need to consider in developing your evaluation strategy – planning, implementation, data analysis, reporting and communication, and project management.Â
Planning
The most important considerations during the planning phase of your engagement evaluation are:
- Developing a theory of change about how the engagement processes will lead to the desired outcomes
- Confirming scope and purpose of evaluation
- Identifying short, medium-term, and long-term metrics of interest to your engagement process
- Prioritizing evaluation instruments and metrics
- Defining and acquiring needed skills and assistance
- Designing your evaluation process including plans for data collection and analysis
- Identifying your target audience(s)
- Determining role your participants or stakeholders will have in the development process
- Assessing the feasibility of your potential methods for your target audience(s)
- Planning for how you will adapt the engagement process in relation to evaluation findings
Definition: Theory of Change
A theory of change is a description of how particular activities will achieve the desired change. This comprehensive explanation provides a framework for (1) identifying the specific types of outcomes desired and then (2) working backward to identify all the actions (such as behaviors or programs or policies) that need to be put in place to achieve those outcomes, and (3) providing an explanation of how the engagement efforts and related activities will create or lead to those actions.
Implementation
This is the carrying out of your evaluation plan. Although it will vary from project to project, this stage is where you will collect and manage evaluation data your project has identified as most critical. There are two main types of evaluation efforts which, when combined, provide a comprehensive assessment of the engagement process. First, formative and process evaluation strategies (see next section for definitions) seek to improve the engagement practice and to ensure the engagement effort is being implemented consistent with its overall goals. Second, summative evaluation efforts seek to document the outcomes resulting from the engagement processes. These outcomes can be short-term (such as knowledge gained), medium-term (such as sustained changes to behaviors) or long-term (such as changes in environmental conditions).
Data Analysis
Upon completion of data collection, the next step is to analyze the data and compare the results to the objectives of the engagement project.
Reporting & Communication
To tell your story effectively, it will be important for you to consider what you want to communicate about the results or processes of your project, which audiences are most important to communicate with, and what are the most appropriate methods for reaching these audiences. It is also particularly important to share formative evaluation findings with the project team and participants to identify any adaptations or changes that need to be made to the engagement process.
Project ManagementÂ
In addition to planning and conducting the evaluation, project leaders need to consider the management of the evaluation process, including time, skills and resources needed, job responsibilities across the team, and timelines (e.g., developing a Gantt chart or a similar planning tool).
Tools & worksheets
A Checklist to Help Focus Your Evaluation
A checklist, created by the CDC, to support you in crafting and assessing specific evaluation questions within a broader program or project evaluation.
Developing Your Engagement Plan
Guide and worksheet for developing an effective evaluation plan
Evaluations Questions Checklist for Program Evaluation
The purpose of this checklist is to aid in developing effective and appropriate evaluation questions and in assessing the quality of existing questions. It identifies characteristics of good evaluation questions, based on the relevant literature and our own experience with evaluation design, implementation, and use.
Additional resources
Developing a Logic Model: Teaching and Training Guide
This guide, written by Ellen Taylor-Powell, PhD and Ellen Henert, describes logic models and guides the reader through how to create and implement one. Contains many useful worksheets as appendices.
Phases of Data Analysis
This brief, written by Glenn D. Israel, covers the phases of data analysis for evaluation of an extension program.
Linking Extension Program Design with Evaluation Design for Improved Evaluation
Radhakrishna R, Chaudhary AK, Tobin D (2019) 3/16/2020 Linking Extension Program Design with Evaluation Design for Improved Evaluation, Journal of Extension 57(4).
Abstract: We present a framework to help those working in Extension connect program designs with appropriate evaluation designs to improve evaluation. The framework links four distinct Extension program domains—service, facilitation, content transformation, and transformative education—with three types of evaluation design—preexperimental, quasi-experimental, and true experimental. We use examples from Extension contexts to provide detailed information for aligning program design and evaluation design. The framework can be of value to various audiences, including novice evaluators, graduate students, and non-social scientists, involved in carrying out systematic evaluation of Extension programs.
Collecting Evaluation Data: An Overview of Sources and Methods
This brief, written by Ellen Taylor-Powell and Sara Steele, offers an overview of sources and methods in evaluation-related data collection, with a focus on extension.
Capturing Change: Comparing Pretest-Posttest and Retrospective Evaluation Methods
This brief, written by Jessica L. O’Leary and Glenn D. Israel compares two models of extension evaluation: “pretest-posttest” and “retrospective”.
Evaluation Models
In this monograph by Daniel L. Stufflebeam, the author reviews the dominant evaluation models used in the United States between 1960 and 1999 and argues which should be brought forward into the 21st century and which should be left behind.
Evaluability Assessment: Examining the Readiness of a Program for Evaluation
“The purpose of this briefing is to introduce program managers to the concept of Evaluability Assessment.”
Developing a Concept of Extension Program Evaluation
This brief offers an introduction to extension program evaluation and the many dimensions of its design and implementation.
Guiding Principles for Evaluators
This brief guide from the American Evaluation Association “can help you identify the basic ethical behavior to expect of yourself and of any evaluator”.
Developing an Effective Evaluation Plan: A guidebook
Developing an Effective Evaluation Plan. Atlanta, Georgia: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health; Division of Nutrition, Physical Activity, and Obesity, 2011.
This guidebook, produced by the CDC, provides a framework laying out a six-step process for the decisions and activities involved in conducting an evaluation.