Archive: evaluation procurement

Newsletter: What should I do if my college’s procurement office won’t let me name an evaluator in my proposal?

Posted on July 1, 2015 by  in Newsletter - ()

Director of Research, The Evaluation Center at Western Michigan University

DIY Evaluation Planning

It is generally considered best practice to identify your intended external evaluator by name in an ATE proposal and work with him or her to write the evaluation section. In some cases, college procurement policies may be at odds with this long-standing practice (e.g., see Jacqueline Rearick’s blog post on this topic at http://bit.ly/rearick). If you have to proceed with evaluation planning without the benefit of involvement by an external evaluator, here are some tips for DIY (do-it-yourself) evaluation planning:

Develop a project logic model that specifies your project’s activities, outputs (products), and outcomes. Yes, you can do this! The task of logic model development often falls to an evaluator, although it’s really just project planning. But it provides a great foundation for framing your evaluation plan. Try out our ATE Logic Model Template (http://bit.ly/ate-logic).

Specify the focus of the evaluation by formulating evaluation questions. These should be clearly tied to what is in the logic model. Here are some generic evaluation questions: How well did the project reach and engage its intended audience? How satisfied are participants with the project’s activities and products? To what extent did the project bring about changes in participants’ knowledge, skills, attitudes, and/or behaviors? How well did the project meet the needs it was designed to address? How sustainable is the project? Ask questions about both the project’s implementation and outcomes and avoid questions that can be answered with a yes/no or single number.

Describe the data collection plan. Identify the data and data sources that will be used to answer each of the evaluation questions. Keep in mind most evaluation questions will need multiple sources of evidence in order to answer adequately. Utilizing both qualitative and quantitative data will strengthen your evidence base. Use our Data Collection Planning Matrix to work out the details of your plan (see p. 3- Data Collection Planning Matrix).

Describe the analytical and interpretive procedures to be used for making sense of the evaluation data. For DIY evaluation plans, keep it simple. In fact, most project evaluations (not including research projects) rely mainly on basic descriptive statistics (e.g., percentages, means, aggregate numbers) for analysis. As appropriate, compare data over time, by site, by audience type, and/or against performance targets to aid in interpretation.

Identify the main evaluation deliverables. These are the things the evaluation effort specifically (not the overall project) will produce. Typical deliverables include a detailed evaluation plan (i.e., an expanded version of the plan included in the proposal that is developed after the project is funded), data collection instruments, and evaluation reports. NSF also wants to see how the project will use the evaluation findings, conclusions, and recommendations to inform and improve ongoing project work.

Include references to the evaluation literature. At minimum, consult and reference the NSF User Friendly Handbook for Project Evaluation (http://bit.ly/nsf-evalguide) and the Program Evaluation Standards (http://bit.ly/jc-pes).

Include a line item in your budget for evaluation. The average allocation among ATE projects for evaluation is 7 percent (see Survey Says on p. 1).

Finally, if you’re including a DIY evaluation plan in your proposal, specify the policy prohibiting you from identifying and working with a particular evaluator at the proposal stage. Make it absolutely clear to reviewers why you have not engaged an external evaluator and what steps you will take to procure one once an award is made.

Blog: Evaluation Procurement: Regulations, Rules and Red Tape… Oh My!

Posted on April 8, 2015 by  in Blog (, )

Grants Specialist, Virginia Western Community College

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

I’m Jacqueline Rearick, and I am a Grants Specialist at Virginia Western Community College where I support our NSF/ATE projects and sub-awards, among other grants. I’m also an evaluation advocate and can get a bit overzealous about logic models, outcomes, surveys, and assessments. Recently, our grants office had to work through the process of procurement to secure evaluation services for our ATE project. Although we referenced an external evaluator in the project design, the policies and procedures of our individual state procurement regulations trumped the grant proposal and became the focus of a steep learning curve for all involved.

Because we have different priorities it may appear that the grants office and the procurement office can be in direct opposition with one another. Grant proposals that require evaluation services, like ATE, work best when the evaluator is part of the process and can assist with developing the plan and then execute the evaluation. Procurement regulations at your individual institution could require a bid process; which may or may not result in securing the evaluator who helped you write the initial evaluation plan.

Hot Tip: Invite the procurement office to the table early

Securing evaluation services for your ATE project is important; so is following internal procurement rules. Touch base with your procurement office early in the evaluation development process. Are there local or state regulations that will require a bid process? If your ATE evaluator assists with the writing of your evaluation section in the proposal, will you be able to use that same evaluator if the grant is funded? Have an honest conversation with your evaluator about the procurement process.

Hot Tip: Levels of procurement, when the rules change

While working through the procurement process, we discovered that state rules change when the procurement of goods or services reach different funding levels. What was a simple evaluation procurement for our first small ATE grant ($200k) turned into much larger scale procurement for our second ATE project grant ($900k), based on our state guidelines. Check with your institution to determine thresholds and the required guidelines for consultant services at various funding levels.

Lesson Learned: All’s well that ends well

The process of securing evaluation services through procurement is designed to be one that allows the PI to review all competitors to determine quality evaluation services at a reasonable price. The evaluator who helped write our evaluation in the proposal was encouraged to bid on the project. What’s even better, this evaluator is now set up as a vendor in our state system and will be available to other colleges in the state as they seek quality ATE evaluation services.