Archive: evaluation planning

Video: Introduction to Evaluation for Mentor-Connect Cohort 2017

Posted on April 25, 2017 by  in Videos ()

This video was created for the 2017 Mentor-Connect Cohort, but can be applicable to others interested in learning about ATE Evaluation. Specifically this video provides an overview of What is project evaluation?, Why does NSF require evaluation?, How do you plan for evaluation?, and How can EvaluATE help?

File: Click Here
Type: Video
Category: video
Author(s): Lori Wingate

Blog: Best Practices for Two-Year Colleges to Create Competitive Evaluation Plans

Posted on September 28, 2016 by , in Blog ()
Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

 

Ball
Kelly Ball
Ball
Jeff Grebinoski
Senior Grant Specialist
Northeast Wisconsin Technical College
Institutional Research
Northeast Technical College

Northeast Wisconsin Technical College’s (NWTC) Grants Office works closely with its Institutional Research Office to create ad hoc evaluation teams in order to meet the standards of evidence required in funders’ calls for proposals. Faculty members at two-year colleges often make up the project teams that are responsible for National Science Foundation (NSF) grant project implementation. However, they often need assistance navigating among terms and concepts that are traditionally found in scientific research and social science methodology.

Federal funding agencies are now requiring more evaluative rigor in their grant proposals than simply documenting deliverables. For example, the NSF’s Scholarships in Science, Technology, Engineering, and Mathematics (S-STEM) program saw dramatic changes in 2015: The program solicitation increased the amount of non-scholarship budget from 15% of the scholarship amount to 40% of the total project budget to increase supports for students and to investigate the effectiveness of those supports.

Technical colleges, in particular, face a unique challenge as solicitations change: These colleges traditionally have faculty members from business, health, and trades industries. Continuous improvement is a familiar concept to these professionals; however, they tend to have varying levels of expertise evaluating education interventions.

The following are a few best practices we have developed for assisting project teams in grant proposal development and project implementation at NWTC.

  • Where possible, work with an external evaluator at the planning stage. External evaluators can provide the expertise that principal investigators and project teams might lack as external evaluators are well-versed on current evaluation methods, trends, and techniques.
  • As they develop their projects, teams should meet with their Institutional Research Office to better understand data gathering and research capacity. Some data needed for evaluation plans might be readily available, whereas others might require some advanced planning to develop a system to track information. Conversations about what the data will be used for and what questions the team wants to answer will help ensure that the correct data are able to be gathered.
  • After a grant is awarded, have a conversation early with all internal and external evaluative parties about clarifying data roles and responsibilities. Agreeing to reporting deadlines and identifying who will collect the data and conduct further analysis will help avoid delays.
  • Create a “data dictionary” for more complicated projects and variables to ensure that everyone is on the same page about what terms mean. For example, “student persistence” can be defined term-to-term or year-to-year and all parties need to understand which data will need to be tracked.

With some planning and the right working relationships in place, two-year colleges can maintain their federal funding competitiveness even as agencies increase evaluation requirements.

Webinar: Meeting Requirements, Exceeding Expectations: Understanding the Role of Evaluation in Federal Grants

Posted on March 22, 2016 by , , in Webinars

Presenter(s): Ann Beheler, Leslie Goodyear, Lori Wingate
Date(s): May 25, 2016
Time: 3-4:00 p.m.
Recording: https://www.youtube.com/watch?v=xZdiwSizDUM&feature=youtu.be

External evaluation is a requirement of many federal grant programs. Understanding and addressing these requirements is essential for both successfully seeking grants and achieving the objectives of funded projects. In this webinar, we will review the evaluation language from a variety federal grant programs and translate the specifications into practical steps. Topics will include finding a qualified evaluator, budgeting for evaluation, understanding evaluation design basics, reporting and using evaluation results, and integrating past evaluation results into future grant submissions.

Resources:
Slides
Additional Resource


Doc: HI-TEC 2015- Handout | Evaluation: Don’t Submit Your ATE Proposal Without It

Posted on November 30, 2015 by , in Conferences

A strong evaluation plan that is well integrated into your grant proposal will strengthen your submission and maybe even give you a competitive edge. In this session we’ll provide insights on ways to enhance your proposal and avoid common pitfalls with regard to evaluation. We’ll walk through EvaluATE’s Evaluation Planning Checklist for ATE Proposals, which provides detailed guidance on how to address evaluation throughout a proposal—from the project summary to the budget justification.

File: Click Here
Type: Doc

Author(s): Corey Smith, Emma Perk

Slides: HI-TEC 2015 | Evaluation: Don’t Submit Your ATE Proposal Without It

Posted on November 30, 2015 by , in Conferences ()

A strong evaluation plan that is well integrated into your grant proposal will strengthen your submission and maybe even give you a competitive edge. In this session we’ll provide insights on ways to enhance your proposal and avoid common pitfalls with regard to evaluation. We’ll walk through EvaluATE’s Evaluation Planning Checklist for ATE Proposals, which provides detailed guidance on how to address evaluation throughout a proposal—from the project summary to the budget justification.

File: Click Here
Type: Slides
Category: Proposal Development
Author(s): Corey Smith, Emma Perk

Newsletter: How can you make sure your evaluation meets the needs of multiple stakeholders?

Posted on October 1, 2015 by  in Newsletter () ()

Director of Research, The Evaluation Center at Western Michigan University

We talk a lot about “stakeholders” in evaluation. These are the folks who are involved in, affected by, or simply interested in the evaluation of your project.  But what these stakeholders want or need to know from the evaluation, the time they have available for the evaluation, and their level of interest are probably quite variable.  Here is a generic guide to types of ATE evaluation stakeholders, what they might need, and how to meet those needs.

Stakeholder groups What they might need Tips for meeting those needs
Project leaders (PI, co-PIs)
  • Information that will help you make improvements to the project as it is unfolding
  • Results you can include in your annual reports to NSF to demonstrate accountability and impact
Communicate your needs clearly to your evaluator, including when you need the information in order to make use of it.
Advisory committees or National Visiting Committees
  • Results from the evaluation that show whether the project is on track for meeting its goals, if changes in direction or operations are warranted
  • Summary information about the projects’ strengths and weaknesses
Many advisory committee members donate their time, so they probably aren’t interested in reading lengthy reports.  Provide a brief memo and/or short presentation at meetings with key findings and invite questions about the evaluation. Be forthcoming about strengths and weaknesses.
Participants who provide data for the evaluation
  • Access to reports where their information was used
  • Summaries of what actions were taken based on the information they needed to provide
The most important thing for this group is to demonstrate use of the information they provided.  You can share reports, but a personal message from project leaders along the lines of “we heard you and here is what we’re doing in response” is most valuable.
NSF program officers
  • Evidence that the project is on track for meeting its goals
  • Evidence of impact (not just what was done, but what difference the work is making)
  • Evidence that the project is using evaluation results to make improvements
Focus on Intellectual Merit (the intrinsic quality of the work and potential to advance knowledge) and Broader Impacts (the tangible benefits for individuals and progress toward desired societal outcomes). If you’re not sure about what your program officer needs from your evaluation, ask him or her for clarification.
College administrators (department chairs, deans, executives, etc.)
  • Results that demonstrate impact on students, faculty, institutional culture, infrastructure, and reputation.
Make full reports available upon request, but most busy administrators probably don’t have the time to read technical reports or need the fine-grained data points. Prepare memos or share presentations that focus on the information they’re most interested in.
Partners and collaborators
  • Information that helps them assess the return on the investment of their time or other resources
See above – like with college administrators, focus on providing the information most pertinent to this group.

In case you didn’t read between the lines—the underlying message here is to provide stakeholders with the information that is most relevant to their particular “stake” in your project. A good way to not meet their needs is to only send everyone a long, detailed technical report with every data point collected. It’s good to have a full report available for those who request it, but many simply won’t have the time or level of interest needed to consume that quantity of evaluative information about your project. Most importantly, don’t take our word as to what they might need: Ask them!

Not sure what stakeholders to involve in your evaluation or how? Check out our worksheet on Identifying Stakeholders and Their Roles in an Evaluation at (bit.ly/id-stake).

Newsletter: Creating an Evaluation Scope of Work

Posted on October 1, 2015 by  in Newsletter - ()

Director of Research, The Evaluation Center at Western Michigan University

One of the most common requests we get at EvaluATE is for examples of independent contractor agreements and scope of work statements for external evaluators. First, let’s be clear about the difference between these two types of documents.

An independent contractor agreement is typically 90 percent boilerplate language required by your institution. Here at Western Michigan University, contracts are run through one of three offices (Business Services, Research and Sponsored Programs, Grants and Contracts, or Purchasing), depending on the type of contract and the nature of the work/service. We can’t tell you the name of the office at your institution, but there definitely is one and they probably have boilerplate contract forms that you will need to use.

A scope of work statement should be attached to and referenced by the independent contractor agreement (or other type of contract). But unlike the contract, it should not be written in legalese, but in plain language understandable to all parties involved. The key issues to cover in a scope of work statement include the following:

Evaluation questions (or objectives): Including information about the purpose of the evaluation is a good reminder to those involved about why the evaluation is being done. It may serve as a useful reference down the road if the evaluation starts to experience scope creep (or shrinkage).

Main tasks and deliverables (with timelines or deadlines): This information should make clear what services and products the evaluator will provide. Common examples include a detailed evaluation plan (what was included in your proposal probably doesn’t have enough detail), data collection instruments, reports, and presentations.

It’s critical to include timelines (generally when things will occur) and deadlines (when they must be finished) in this statement.

Conditions for payment: You most likely specified a dollar amount for the evaluation in your grant proposal, but you probably do not plan on paying that in a lump sum either at the beginning or end of the evaluation or even yearly. Specify in what increments payments should be made and what conditions must be met for payment. Rather than tying payment(s) to certain dates, consider making payment(s)contingent on the completion of certain tasks or deliverables.

Be sure to come to agreement on these terms in collaboration with your evaluator. This is an opportunity to launch your working relationship from a place of open communication and shared expectations.

Newsletter: Data Collection Planning Matrix

Posted on July 1, 2015 by  in Newsletter - ()

The part of your proposal’s evaluation plan that reviewers will probably scrutinize most closely is the data collection plan. Given that the evaluation section of a proposal is typically just 1-2 pages, you have minimal space to communicate a clear plan for gathering evidence of your project’s quality and impact. An efficient way to convey this information is in a matrix format. To help with this task, we’ve created a Data Collection Planning Matrix, available from (bit.ly/data-matrix).

This tool prompts the user to specify the evaluation questions that will serve as the foundation for the evaluation; what indicators1 will be used to answer each evaluation question; how data for each indicator will be collected, from what sources, by whom, and when; and how the data will be analyzed. (The document includes definitions for each of these components to support shared understandings among members of the proposal development team.) Including details about data collection in your proposal shows reviewers that you have been thoughtful and strategic in determining how you will build a body of evidence about the effectiveness and quality of your NSF-funded work. The value of putting this information in a matrix format is that it ensures you have a clear plan for gathering data that will enable you to fully address all the evaluation questions and, conversely, that all the data you plan to collect will serve a specific purpose.

A good rule of thumb is to develop at least one overarching evaluation question for each main element of a project logic model (i.e., activities, outputs, and short-, mid-, and long-term outcomes). Although not required for ATE program proposals, logic models are an efficient way to convey how your project’s activities and products will lead to intended outcomes. The evaluation’s data collection plan should align clearly with your project’s activities and goals, whether you use a logic model or not. If you are interested in developing a logic model for your project and want to learn more, see our ATE Logic Model Template at (bit.ly/ate-logic).

If you have questions about the data collection planning matrix or logic model template or suggestions for improving it, let us know: email us at info@evalu-ate.org.

1 For more on indicators and how to select ones that will serve your evaluation well, see Goldie MacDonald’s checklist, Criteria for Selection of High-Performing Indicators, available from (bit.ly/indicator-eval).

Blog: Evaluation Procurement: Regulations, Rules and Red Tape… Oh My!

Posted on April 8, 2015 by  in Blog (, )

Grants Specialist, Virginia Western Community College

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

I’m Jacqueline Rearick, and I am a Grants Specialist at Virginia Western Community College where I support our NSF/ATE projects and sub-awards, among other grants. I’m also an evaluation advocate and can get a bit overzealous about logic models, outcomes, surveys, and assessments. Recently, our grants office had to work through the process of procurement to secure evaluation services for our ATE project. Although we referenced an external evaluator in the project design, the policies and procedures of our individual state procurement regulations trumped the grant proposal and became the focus of a steep learning curve for all involved.

Because we have different priorities it may appear that the grants office and the procurement office can be in direct opposition with one another. Grant proposals that require evaluation services, like ATE, work best when the evaluator is part of the process and can assist with developing the plan and then execute the evaluation. Procurement regulations at your individual institution could require a bid process; which may or may not result in securing the evaluator who helped you write the initial evaluation plan.

Hot Tip: Invite the procurement office to the table early

Securing evaluation services for your ATE project is important; so is following internal procurement rules. Touch base with your procurement office early in the evaluation development process. Are there local or state regulations that will require a bid process? If your ATE evaluator assists with the writing of your evaluation section in the proposal, will you be able to use that same evaluator if the grant is funded? Have an honest conversation with your evaluator about the procurement process.

Hot Tip: Levels of procurement, when the rules change

While working through the procurement process, we discovered that state rules change when the procurement of goods or services reach different funding levels. What was a simple evaluation procurement for our first small ATE grant ($200k) turned into much larger scale procurement for our second ATE project grant ($900k), based on our state guidelines. Check with your institution to determine thresholds and the required guidelines for consultant services at various funding levels.

Lesson Learned: All’s well that ends well

The process of securing evaluation services through procurement is designed to be one that allows the PI to review all competitors to determine quality evaluation services at a reasonable price. The evaluator who helped write our evaluation in the proposal was encouraged to bid on the project. What’s even better, this evaluator is now set up as a vendor in our state system and will be available to other colleges in the state as they seek quality ATE evaluation services.