Archive: project management

Blog: Evaluation Plan Cheat Sheets: Using Evaluation Plan Summaries to Assist with Project Management

Posted on October 10, 2018 by , in Blog ()
Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

We are Kelly Robertson and Lyssa Wilson Becho, and we work on EvaluATE as well as several other projects at The Evaluation Center at Western Michigan University. We wanted to share a trick that has helped us keep track of our evaluation activities and better communicate the details of an evaluation plan with our clients. To do this, we take the most important information from an evaluation plan and create a summary that can serve as a quick-reference guide for the evaluation management process. We call these “evaluation plan cheat sheets.”

The content of each cheat sheet is determined by the information needs of the evaluation team and clients. Cheat sheets can serve the needs of the evaluation team (for example, providing quick reminders of delivery dates) or of the client (for example, giving a reminder of when data collection activities occur). Examples of items we like to include on our cheat sheets are shown in Figures 1-3 and include the following:

  • A summary of deliverables noting which evaluation questions each deliverable will answer. In the table at the top of Figure 1, we indicate which report will answer which evaluation question. Letting our clients know which questions are addressed in each deliverable helps to set their expectations for reporting. This is particularly useful for evaluations that require multiple types of deliverables.
  • A timeline of key data collection activities and report draft due dates. On the bottom of Figure 1, we visualize a timeline with simple icons and labels. This allows the user to easily scan the entirety of the evaluation plan. We recommend including important dates for deliverables and data collection. This helps both the evaluation team and the client stay on schedule.
  • A data collection matrix. This is especially useful for evaluations with a lot of data collection sources. The example shown in Figure 2 identifies who implements the instrument, when the instrument will be implemented, the purpose of the instrument, and the data source. It is helpful to identify who is responsible for data collection activities in the cheat sheet, so nothing gets missed. If the client is responsible for collecting much of the data in the evaluation plan, we include a visual breakdown of when data should be collected (shown at the bottom of Figure 2).
  • A progress table for evaluation deliverables. Despite the availability of project management software with fancy Gantt charts, sometimes we like to go back to basics. We reference a simple table, like the one in Figure 3, during our evaluation team meetings to provide an overview of the evaluation’s status and avoid getting bogged down in the details.

Importantly, include the client and evaluator contact information in the cheat sheet for quick reference (see Figure 1). We also find it useful to include a page footer with a “modified on” date that automatically updates when the document is saved. That way, if we need to update the plan, we can be sure we are working on the most recent version.

 

Figure 1. Cheat Sheet Example Page 1. (Click to enlarge.)

Figure 2. Cheat Sheet Example Page 2. (Click to enlarge)

Figure 3. Cheat Sheet Example Page 2 (Click to enlarge.)

 

Evaluation Responsibility Diagram

Posted on March 14, 2018 by  in Resources ()

This diagram provides an overview of evaluation responsibilities for the project staff, external evaluator, and combined responsibilities. This example is an excerpt from the Evaluation Basics for Non-evaluators webinar. Access slides, recording, handout, and additional resources from bit.ly/mar18-webinar.

File: Click Here
Type: Doc
Category: Getting Started
Author(s): Lori Wingate

Checklist: Project Vita

Posted on July 12, 2017 by  in Resources ()

This checklist is designed to help with the creation of a project vita. Similar to an individual’s professional vita or resume, a project vita is a comprehensive index of factual information about a project’s activities and achievements. It documents past performance and demonstrates capacity for future endeavors. Tracking this information over the life of a project will make it easier to complete annual reports to sponsors, respond to information requests, and document achievements in funding applications. If the document is easy to find on the project’s website, stakeholders and other interest parties can easily see how productive (or not) the project has been. For a more dynamic vita, include links to supporting documents, staff biographies, or related web pages; this will allow users to quickly locate items referenced in the vita. For an example of a project vita, see evalu-ate.org/vita. This checklist suggests what to include in a vita and how to organize the information. Projects should tailor their vitae to their specific needs.

File: Click Here
Type: Checklist
Category: General
Author(s): Emma Perk

Checklist: Project Resume

Posted on May 6, 2015 by  in Resources ()

DRAFT: This checklist is designed to help project staff create a project resume. A project resume is a list of all key activities or accomplishments of a project. This document can easily be created in a word processing document, then uploaded to the project’s website. Make the resume easy to find on the project’s website, such as in the “About” section. For a more dynamic resume, include links to supporting documents, staff biographies, or personal Web pages, this will allow users to quickly locate items referenced on the project’s resume. Tracking all activities over the life of a project will make it easier to complete annual reports, apply for future funding, and respond to information requests. For an example of our project resume see (About > EvaluATE’s Resume).

File: Click Here
Type: Checklist
Category: Reporting & Use
Author(s): Emma Perk

Newsletter: What do you do when your evaluator disagrees with a recommendation by your program officer?

Posted on April 1, 2014 by  in Newsletter - ()

Director of Research, The Evaluation Center at Western Michigan University

This was a question submitted anonymously to EvaluATE by an ATE principal investigator (PI), so I do not know the specific nature of the recommendation in question. Therefore, my response isn’t about the substance of whatever this recommendation may have been about, but on the interpersonal and political dynamics of the situation.

Let’s put the various players’ roles into perspective:
As PI, you are ultimately responsible for your project—delivering what you outlined in your grant proposal/negotiations and making decisions about how best to conduct the project based on your experience, expertise, and input from various advisors. You are in the position of authority when it comes to how your project is implemented and what recommendations from what sources to implement in order to ensure the success of your project.

Your NSF program officer (PO) monitors your project, primarily based on information you provide in your annual report, submitted to him or her via research.gov. Your NSF
program officer may provide extremely valuable guidance and advice, but the PO’s role is to comment on your project as described in the report. You are not obligated to accept the advice. However, the PO does approve the report, based on his or her assessment of whether the project is sufficiently meeting the expectations of the grant. If you choose not to accept your program officer’s recommendations—which is completely acceptable—you should be able to provide a clear rationale for your decision in a respectful and diplomatic way by addressing each of the issues raised. Such a response should be documented, such as in your annual report and/or a response to the evaluation report.

Your evaluator is a consultant you hired to provide a service to your project in exchange for compensation. You are not obligated to accept this person’s recommendations, either. Again,
however, you should give your evaluator’s recommendations—especially those based on evidence—careful consideration and express why or why not you believe the recommendations are or are not appropriate for your project. An evaluator should never “ding” your project for not implementing the evaluation recommendations.

If you are really not sure who is right and neither person’s position (the PO’s recommendation or the evaluator’s disagreement with it) especially resonates with you and your understanding of what your project needs, you should seek additional information. If you have an advisory panel, this is exactly the type of tricky situation they can help with. If you don’t, you might consult an experienced person at your institution or another ATE project or center PI. Whichever way you go, you should be able to provide a clear rationale for your position and communicate it to both parties. This is not a popularity contest between your evaluator and your program officer. This is about making the right decisions for your project.

Newsletter: The PI Guide to Working with Evaluators

Posted on January 1, 2014 by  in Newsletter - ()

Principal Research Scientist, Education Development Center, Inc.

(originally published as blog at ltd.edc.org/strong-pievaluator-partnerships-users-guide on January 10, 2013)

Evaluation can be a daunting task for PIs. It can seem like the evaluator speaks another language, and the stakes for the project can seem very high. Evaluators face their own challenges. Often working with a tight budgets and timeframes, expectations are high that they deliver both rigor and relevance, along with evidence of project impact. With all this and more in the mix, it’s no surprise that tension can mount and miscommunication can drive animosity and stress.

As the head of evaluation for the ITEST Learning Resource Center and as a NSF program officer, I saw dysfunctional relationships between PIs and their evaluators contribute to missed deadlines, missed opportunities, and frustration on all sides. As an evaluator, I am deeply invested in building evaluators’ capacity to communicate their work and in helping program staff understand the value of evaluation and what it brings to their programs. I was concerned that these dysfunctional relationships would thwart the potential of evaluation to provide vital information for program staff to make decisions and demonstrate the value of their programs.

To help strengthen PI/evaluator collaborations, I’ve done a lot of what I called “evaluation marriage counseling” for PI/evaluator pairs. Through these “counseling sessions,” I learned that evaluation relationships are not so different from any other relationships. Expectations aren’t always made clear, communication often breaks down, and, more than anything else, all relation-ships need care and feeding.

As a program officer, I had the chance to help shape and create a new resource that supports PIs and evaluators in forming strong working relationships. Rick Bonney of the Cornell Lab of Ornithology and I developed a guide to working with evaluators, written by PIs, for PIs. Although it was designed for the Informal Science Education community, the lessons translate to just about any situation in which program staff are working with evaluators. The Principal Investigator’s Guide: Managing Evaluation in Informal STEM Education Projects is available at bit.ly/1l28nTt.

Newsletter: Data Management Plan (DMP)

Posted on July 1, 2013 by  in Newsletter - ()

Director of Research, The Evaluation Center at Western Michigan University

DMPs are not evaluation plans, but they should address how evaluation data will be handled and possibly shared.

DMPs are required for all NSF proposals, uploaded as a Supplementary Document in the FastLane.gov proposal submission system. They can be up to two pages long and should describe

  • the kind of data you will gather
  • how you’ll format the data and metadata (metadata is documentation about what your primary data are)
  • what policies will govern the access and use of your data by others
  • how you’ll protect privacy, confidentiality, security, and intellectual property
  • who would be interested in accessing your data and in what ways they might use it
  • your plans for archiving the data and preserving access to them

To learn more about data management plans, check out these resources: