Archive: evaluator

Blog: What I’ve Learned about Evaluation: Lessons from the Field

Posted on June 21, 2020 by  in Blog ()

Coordinator in Educational Leadership, San Francisco State University

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

What I’ve Learned about Evaluation_ Lessons from the FieldI’m completing my second year as the external evaluator of a three-year ATE project. As a first-time evaluator, I have to confess that I’ve had a lot to learn.

The first surprise was that, in spite of my best intentions, my evaluation process seems always a bit messy. A grant proposal is just that: a proposed plan. It is an idealized vision of what may come. Therefore, the evaluation plan based on that vision is also idealized. Over time, I have had to reconsider my evaluation as grant activities and circumstances evolved—what data is to be collected, how it is to be collected, or whether that data is to be collected at all.

I also thought that my evaluations would somehow reveal something startling to my project team. In reality, my evaluations have served as a mirror to them, acknowledging what they have done and mostly confirming what they already suspect to be true. In a few instances, the manner in which I’ve analyzed data has allowed the team to challenge some assumptions made along the way. In general, though, my work is less revelatory than I had expected.

Similarly, I anticipated my role as a data analyst would be more important. However, this project was designed to use iterative continuous improvement and so the team has met frequently to analyze and consider anecdotal data and impromptu surveys. This more immediate feedback on project activities was regularly used to guide changes. So while my planned evaluation activities and formal data analysis has been important, it has been a less significant contribution than I had expected.

Instead, I’ve added the greatest value to the team by serving as a critical colleague. Benefiting from distance from the day-to-day work, I can offer a more objective, outsider’s view of the project activities. By doing so, I’m able to help a talented, innovative, and ambitious team consider their options and determine whether or not investing in certain activities promotes the goals of the grant or moves the team tangentially. This, of course, is critical for a small grant on a small budget.

Over my short time involved in this work, I see that by being brought into the project from the beginning, and encouraged to offer guidance along the way, I’ve assessed the progress made in achieving the grant goals, and I have been able to observe and document how individuals work together effectively to achieve those goals. This insight highlights another important service evaluators can offer: to tell the stories of successful teams to their stakeholders.

As evaluators, we are accountable to our project teams and also to their funders. It is in the funders’ interest to learn how teams work effectively to achieve results. I had not expected it, but I now see that it’s in the teams’ interest for the external evaluators to understand their successful collaboration and bring it to light.

Blog: SWOT Analysis: What Is It? How Can It Be Useful?

Posted on August 6, 2019 by  in Blog ()

Doctoral Candidate, University of North Carolina at Greensboro

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Hello! My name is Cherie Avent, and I am a graduate student at the University of North Carolina at Greensboro. As a member of an external evaluation team, I recently helped facilitate a SWOT analysis for program managers of a National Science Foundation project to aid them in understanding their strengths, areas of improvement, and potential issues impacting the overall success of the project. In this blog, I will share what a SWOT analysis is, how it can benefit evaluations, and how to conduct one.

What is a SWOT Analysis?

The acronym “SWOT” stands for strengths, weaknesses, opportunities, and threats. A SWOT analysis examines the current performance and the potential future of a program or project. Strengths and weaknesses are controllable factors internal to a program, while opportunities and threats are uncontrollable external factors potentially impacting the circumstances of the project (Chermack & Kasshanna, 2007). More specifically, a SWOT analysis is used to achieve more effective decision making, assessing how strengths can be utilized for new opportunities and how weaknesses can hinder programmatic progress or highlight threats (Helms & Nixon, 2010). The goal is to take advantage of strengths, address weaknesses, maximize opportunities, and limit the impact of threats (Chermack & Kasshanna, 2007).

How can a SWOT be useful?

As evaluators, we can facilitate SWOT analyses with program managers to assist them in 1) understanding current project actions that are working well or need improving, 2) identifying opportunities for leveraging, 3) limiting areas of challenge, and 4) refining decision making for the overall success of the program. Many of the projects we serve involve various objectives and actions for achieving the overarching program goal. Therefore, a SWOT analysis provides an opportunity for program managers to assess why specific strategies or plans work and others do not.

How does one conduct a SWOT analysis?

There are multiple ways to conduct a SWOT analysis. Here are a few steps we found useful (Chermack & Kasshanna, 2007):

  1. Define the objective of the SWOT analysis with participants. What do program managers or participants want to gain by conducting the SWOT analysis?
  2. Provide an explanation of SWOT analysis procedures to participants.
  3. Using the two-by-two matrix below, ask each participant to consider and write strengths, weaknesses, opportunities, and threats of the project. Included are questions they may think about for each area.

SWOT Analysis

  1. Combine the individual worksheets into a single chart or spreadsheet. You can use a Google document or a large wall chart so everyone can participate.
  2. Engage participants in a dialogue about their responses for each category, discussing why they chose those responses and how they see the descriptions impacting the project. Differing perspectives will likely emerge. Ask participants how weaknesses can become strengths and how opportunities can become threats.
  3. Lastly, develop an action plan for moving forward. It should consist of concrete and achievable steps program managers can take concerning the programmatic goals.

 

References:

Chermack, T. J., & Kasshanna, B. K. (2007). The use and misuse of SWOT analysis and implications for HRD professionals. Human Resource Development International, 10(4), 383–399. doi:10.1080/13678860701718760

Helms, M. M., & Nixon, J. (2010). Exploring SWOT analysis—where are we now? A review of academic research from the last decade. Journal of Strategy and Management, 3(3), 215–251. doi:10.1108/17554251011064837

Keywords: evaluators, programmatic performance, SWOT analysis

Resource: Evaluation Responsibility Diagram

Posted on March 14, 2018 by  in Resources ()

This diagram provides an overview of evaluation responsibilities for the project staff, external evaluator, and combined responsibilities. This example is an excerpt from the Evaluation Basics for Non-evaluators webinar. Access slides, recording, handout, and additional resources from bit.ly/mar18-webinar.

File: Click Here
Type: Doc
Category: Getting Started
Author(s): Lori Wingate

Template: Evaluator Biographical Sketch

Posted on July 13, 2017 by  in Resources ()

This template was created by EvaluATE. It is based on the National Science Foundation’s guidelines for preparing biographical sketches for senior project personnel. The information about what evaluators should include in Products and Synergistic Activities sections are EvaluATE’s suggestions, not NSF requirements. The biosketch must not exceed two pages.

File: Click Here
Type: Worksheet
Category: Resources
Author(s): Lori Wingate

Newsletter: The Power of Evaluator and PI Collaboration

Posted on April 1, 2014 by  in Newsletter - ()

The PI of an ATE center or project has the responsibility of keeping a strong communication flow with the evaluator. This begins even before the project is funded and continues in a dynamic interchange throughout the funding cycle. There are easy ways that PI and evaluator can add value to a project. Simply asking for help is sometimes overlooked.

A recent example demonstrates how an ATE center used the expertise of the evaluator to get some specific feedback on the use of clearinghouse materials. The co-PI asked the evaluator for assistance and a very nice survey was created that allowed the evaluator to gather additional information about curriculum and instructional materials usage and the center PI’s to gain valuable input about the use of its existing materials.

Second, it is important to actually use the information gained from the evaluation data. What a natural and built in opportunity for the PI and the team to take advantage of impact data to drive the future direction of the  center or project. Using data to make decisions provides an opportunity to test assumptions and to learn if current practices and products are working.

Third, the evaluation develops evidence to be used to obtain further funding, advance technical education and contribute field of evaluation. By regular communication and collaboration, the project, the PI and the evaluator all gain value and can more effectively contribute to the design of the current and future projects. Together, the PI and the Evaluator can learn about impact, trends, and key successes that are appropriate for scaling. Thus evaluation is more than reporting but becomes a tool for strategic planning.

The Bio-Link evaluator, Candiya Mann, provides not only a written document that can be used for reporting and planning but also works with me to expand my connections with other projects and people that have similar interests in the use of data to drive actions and achieve broader impact. Removing isolation contributes new ideas for metrics and can actually make evaluation fun.

Learn more about Bio-Link at www.bio-link.org.