An example evaluation timeline across 3 years for an ATE project. This example is an excerpt from the Evaluation Basics for Non-evaluators webinar. Access slides, recording, handout, and additional resources from bit.ly/mar18-webinar.
This diagram provides an overview of evaluation responsibilities for the project staff, external evaluator, and combined responsibilities. This example is an excerpt from the Evaluation Basics for Non-evaluators webinar. Access slides, recording, handout, and additional resources from bit.ly/mar18-webinar.
This checklist is designed to help with the creation of a project vita. Similar to an individual’s professional vita or resume, a project vita is a comprehensive index of factual information about a project’s activities and achievements. It documents past performance and demonstrates capacity for future endeavors. Tracking this information over the life of a project will make it easier to complete annual reports to sponsors, respond to information requests, and document achievements in funding applications. If the document is easy to find on the project’s website, stakeholders and other interest parties can easily see how productive (or not) the project has been. For a more dynamic vita, include links to supporting documents, staff biographies, or related web pages; this will allow users to quickly locate items referenced in the vita. For an example of a project vita, see evalu-ate.org/vita. This checklist suggests what to include in a vita and how to organize the information. Projects should tailor their vitae to their specific needs.
DRAFT: This checklist is designed to help project staff create a project resume. A project resume is a list of all key activities or accomplishments of a project. This document can easily be created in a word processing document, then uploaded to the project’s website. Make the resume easy to find on the project’s website, such as in the “About” section. For a more dynamic resume, include links to supporting documents, staff biographies, or personal Web pages, this will allow users to quickly locate items referenced on the project’s resume. Tracking all activities over the life of a project will make it easier to complete annual reports, apply for future funding, and respond to information requests. For an example of our project resume see (About > EvaluATE’s Resume).
This was a question submitted anonymously to EvaluATE by an ATE principal investigator (PI), so I do not know the specific nature of the recommendation in question. Therefore, my response isn’t about the substance of whatever this recommendation may have been about, but on the interpersonal and political dynamics of the situation.
Let’s put the various players’ roles into perspective:
As PI, you are ultimately responsible for your project—delivering what you outlined in your grant proposal/negotiations and making decisions about how best to conduct the project based on your experience, expertise, and input from various advisors. You are in the position of authority when it comes to how your project is implemented and what recommendations from what sources to implement in order to ensure the success of your project.
Your NSF program officer (PO) monitors your project, primarily based on information you provide in your annual report, submitted to him or her via research.gov. Your NSF
program officer may provide extremely valuable guidance and advice, but the PO’s role is to comment on your project as described in the report. You are not obligated to accept the advice. However, the PO does approve the report, based on his or her assessment of whether the project is sufficiently meeting the expectations of the grant. If you choose not to accept your program officer’s recommendations—which is completely acceptable—you should be able to provide a clear rationale for your decision in a respectful and diplomatic way by addressing each of the issues raised. Such a response should be documented, such as in your annual report and/or a response to the evaluation report.
Your evaluator is a consultant you hired to provide a service to your project in exchange for compensation. You are not obligated to accept this person’s recommendations, either. Again,
however, you should give your evaluator’s recommendations—especially those based on evidence—careful consideration and express why or why not you believe the recommendations are or are not appropriate for your project. An evaluator should never “ding” your project for not implementing the evaluation recommendations.
If you are really not sure who is right and neither person’s position (the PO’s recommendation or the evaluator’s disagreement with it) especially resonates with you and your understanding of what your project needs, you should seek additional information. If you have an advisory panel, this is exactly the type of tricky situation they can help with. If you don’t, you might consult an experienced person at your institution or another ATE project or center PI. Whichever way you go, you should be able to provide a clear rationale for your position and communicate it to both parties. This is not a popularity contest between your evaluator and your program officer. This is about making the right decisions for your project.
(originally published as blog at ltd.edc.org/strong-pievaluator-partnerships-users-guide on January 10, 2013)
Evaluation can be a daunting task for PIs. It can seem like the evaluator speaks another language, and the stakes for the project can seem very high. Evaluators face their own challenges. Often working with a tight budgets and timeframes, expectations are high that they deliver both rigor and relevance, along with evidence of project impact. With all this and more in the mix, it’s no surprise that tension can mount and miscommunication can drive animosity and stress.
As the head of evaluation for the ITEST Learning Resource Center and as a NSF program officer, I saw dysfunctional relationships between PIs and their evaluators contribute to missed deadlines, missed opportunities, and frustration on all sides. As an evaluator, I am deeply invested in building evaluators’ capacity to communicate their work and in helping program staff understand the value of evaluation and what it brings to their programs. I was concerned that these dysfunctional relationships would thwart the potential of evaluation to provide vital information for program staff to make decisions and demonstrate the value of their programs.
To help strengthen PI/evaluator collaborations, I’ve done a lot of what I called “evaluation marriage counseling” for PI/evaluator pairs. Through these “counseling sessions,” I learned that evaluation relationships are not so different from any other relationships. Expectations aren’t always made clear, communication often breaks down, and, more than anything else, all relation-ships need care and feeding.
As a program officer, I had the chance to help shape and create a new resource that supports PIs and evaluators in forming strong working relationships. Rick Bonney of the Cornell Lab of Ornithology and I developed a guide to working with evaluators, written by PIs, for PIs. Although it was designed for the Informal Science Education community, the lessons translate to just about any situation in which program staff are working with evaluators. The Principal Investigator’s Guide: Managing Evaluation in Informal STEM Education Projects is available at bit.ly/1l28nTt.
DMPs are not evaluation plans, but they should address how evaluation data will be handled and possibly shared.
DMPs are required for all NSF proposals, uploaded as a Supplementary Document in the FastLane.gov proposal submission system. They can be up to two pages long and should describe
- the kind of data you will gather
- how you’ll format the data and metadata (metadata is documentation about what your primary data are)
- what policies will govern the access and use of your data by others
- how you’ll protect privacy, confidentiality, security, and intellectual property
- who would be interested in accessing your data and in what ways they might use it
- your plans for archiving the data and preserving access to them
To learn more about data management plans, check out these resources: