Archive: basic evaluation

Blog: Shorten the Evaluation Learning Curve: Avoid These Common Pitfalls*

Posted on September 16, 2020 by  in Blog ()

Executive Director, The Evaluation Center at Western Michigan University

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

This EvaluATE blog is focused on getting started with evaluation. It’s oriented to new ATE principal investigators who are getting their projects off the ground, but I think it holds some good reminders for veteran PIs as well. To shorten the evaluation learning curve, avoid these common pitfalls:

Searching for the truth about “what NSF wants from evaluation.” NSF is not prescriptive about what an ATE evaluation should or shouldn’t look like. So, if you’ve been concerned that you’ve somehow missed the one document that spells out exactly what NSF wants from an ATE evaluation—rest assured, you haven’t overlooked anything. But there is information that NSF requests from all projects in annual reports and that you are asked to report on the annual ATE survey. So it’s worthwhile to preview the Research.gov reporting template (bit.ly/nsf_prt) and the ATE annual survey questions (bit.ly/ATEsurvey16). And if you’re doing research, be sure to review the Common Guidelines for Education Research and Development – which are pretty cut-and-dried criteria for different types of research (bit.ly/cg-checklist). Most importantly, put some time into thinking about what you, as a project leader, need to learn from the evaluation. If you’re still concerned about meeting expectations, talk to your program officer.

Thinking your evaluator has all the answers. Even for veteran evaluators, every evaluation is new and has to be tailored to context. Don’t expect your evaluator to produce a detailed, actionable evaluation plan on Day 1. He or she will need to work out the details of the plan with you. And if something doesn’t seem right to you, it’s OK to ask for something different.

 Putting off dealing with the evaluation until you are less busy. “Less busy” is a mythical place and you will probably never get there. I am both an evaluator and a client of evaluation services, and even I have been guilty of paying less attention to evaluation in favor of “more urgent” matters. Here are some tips for ensuring your project’s evaluation gets the attention it needs: (a) Set a recurring conference call or meeting with your evaluator (e.g., every two to three weeks); (b) Put evaluation at the top of your project team’s meeting agendas, or hold separate meetings to focus exclusively on evaluation matters; (c) Give someone other than the PI responsibility for attending to the evaluation—not to replace the PI’s attention, but to ensure the PI and other project members are staying on top of the evaluation and communicating regularly with the evaluator; (d) Commit to using the evaluation results in a timely way—if you do something on a recurring basis, make sure you gather feedback from those involved and use it to improve the next activity.

Assuming you will need your first evaluation report at the end of Year 1. PIs must submit their annual reports to NSF 90 days prior to the end of the current budget period. So if your grant started on September 1, your first annual report is due around June 1. And it will take some time to prepare, so you should probably start writing in early May. You’ll want to include at least some of your evaluation results, so start working with your evaluator now to figure what information is most important to collect right now.

Veteran PIs: What tips do you have for shortening the evaluation learning curve?  Submit a blog to EvaluATE and tell your story and lessons learned for the benefit of new PIs.

*Blog is a reprint of the 2015 newsletter article

Newsletter: From ANCOVA to Z Scores

Posted on January 1, 2014 by  in Newsletter - ()

EvaluATE Blog Editor

The Evaluation Glossary App features more than 600 terms related to evaluation and assessment. Designed for both evaluators and those who work with evaluators, the app provides three ways to access the terms. The first way allows the user to browse alphabetically, like a dictionary. The second option is to view the terms by one of eight categories: 1) data analysis; 2) data collection; 3) ethics and guidelines; 4)evaluation design; 5) miscellaneous; 6)  program planning; 7) reporting and utilization; and 8) types of evaluation. The categories are a great starting point for users who are less familiar with evaluation lingo. The final option is a basic search function, which can be useful to anyone who needs a quick definition for an evaluation term. Each entry provides a citation for the definition’s source and crossreferences related terms in the glossary.

App author: Kylie Hutchinson of Community Solutions. Free for Android, iOS. Available wherever you purchase apps for your Android or Apple mobile device or from  communitysolutions.ca/web/evaluation-glossary/.

Newsletter: What grant writers need to know about evaluation

Posted on July 1, 2013 by  in Newsletter - ()

District Director of Grants and Educational Services, Coast Community College District

Fellow grant writers: Do you ever stop and ask yourselves, “Why do we write grants?” Do you actually enjoy herding cats, pulling teeth, or the inevitable stress of a looming proposal deadline? I hope not. Then what is the driver? We shouldn’t just write a grant to simply get funded or to earn prestige for our colleges. Those benefits may be motivators, but we should write to get positive results for the faculty, students, and institutions involved. And we should be able to evaluate those results in useful and meaningful ways so that we can identify ways to improve and demonstrate the project’s value.

Evaluation isn’t just about satisfying a promise or meeting a requirement to gather and report data, it’s about gathering meaningful data that can be utilized to determine the effectiveness of an activity and the impact of a project. When developing a grant proposal, one often starts with the goals, then thinks of the objectives and then plans the activities, hoping that in the end, the evaluation data will prove that the goals were met and the project was a success. That is putting a lot of faith in “hope.” I find it more promising to begin with the end in mind from an evaluation perspective: What is the positive change that we hope to achieve and how will it be evidenced? What does success mean? How can we tell? When will we know? And, how can we get participants to provide the information we will need for the evaluation?

The role of a grant writer is too often like that of a quilt maker—sections of the grant solicitation are delegated to different members of the institution with the evaluation section often outsourced to a third-party evaluator. Each party submits their content, then the grant writer scrambles to patch it all together. Now instead of a quilt, consider the construction of a tapestry. Instead of chunks of material stitched together in independent sections, each thread is carefully woven in a thoughtful way to create a larger, more cohesive overall design. It is important that the entire development team work together to fully understand each aspect of the proposal in order to collaboratively develop a coherent plan to obtain the desired outcomes. The project workplan, budget, and evaluation components should not be designed or executed independently—they occur simultaneously and are dependent upon each other.

I encourage you to think like an evaluator as you develop your proposals. Prepare yourself and challenge your team to be able to justify the value of each goal, objective, and activity and be able to explain how that value will be measured. If at all possible, involve your external or internal evaluator early on in the proposal development. The better the evaluator understands your overall concept and activities, the better he or she can tailor the evaluation plan to derive the desired results. A strong workplan and evaluation plan will help proposal reviewers connect the dots and see the potential of your proposal. It will also serve as a roadmap to success for your project implementation team.