Finding and Selecting an Evaluator for Advanced Technological Education (ATE) Proposals
ATE PROPOSERS SHOULD CAREFULLY READ THE ATE PROGRAM SOLICITATION
All ATE proposals are required to request “funds to support an evaluator independent of the project.” Ideally, this external evaluator should be identified in the project proposal. The information in this guide is for individuals who are able to select and work with an external evaluator at the proposal stage. However, some institutions prohibit selecting an evaluator on a noncompetitive basis in advance of an award being made. Advice for individuals in that situation is provided in an EvaluATE blog (bit.ly/rearick) and newsletter article (bit.ly/no-eval).
This guide includes advice on how to locate and select an external evaluator. It is not intended as a guide for developing an evaluation plan or contracting with an evaluator.
1. What is an external evaluator?
An external evaluator is the person who will lead the design and implementation of the evaluation of your ATE project. The evaluation will include systematic collection and analysis of evidence related to the quality, effectiveness, and impact of the project. To be external, the evaluator must be independent of the project (see Question 3).
2. When should I start working with an evaluator?
Proposal developers should contact an evaluator at least one month in advance of the proposal’s due date—earlier if possible. A good evaluation plan should be closely aligned with the project’s goals and activities. To achieve good alignment, the evaluator needs time to review a draft of the proposal, ask questions, and develop a sound evaluation plan. With short notice, some evaluators may offer to provide a generic evaluation plan. However, seasoned proposal reviewers will give your proposal a more favorable review if it has a well-integrated, tailored evaluation plan.
3. Where should I look for an evaluator?
There is no list of vetted or approved evaluators for NSF projects. It is up to the proposal developer (which is usually the principal investigator) to locate an evaluator and determine if they are qualified and right for a project.
Here are three sources for locating a potential evaluator:
- Ask colleagues for recommendations: If you know someone with a grant that has an evaluation component, ask for the evaluator’s name and contact information.
- Use the American Evaluation Association’s evaluator directory (bit.ly/aea-dir): It’s searchable by state and keyword.
- Use ATE Central’s evaluator map (atecentral.net/evaluators): This interactive map can be used to identify evaluators by location and the types of ATE projects they evaluate.
- Check out the list of current ATE evaluators.
Most ATE projects employ evaluators based outside of their home institutions. However, program rules do allow grant recipients to contract with an evaluator who is employed by the project’s home institution, as long as the evaluator is independent of the project. That is, the evaluator should not work in the same unit where the project is housed. However, neither the evaluator nor any of the project’s personnel should have supervisory responsibilities in relation to the other party.
4. How do I determine if an evaluator is qualified and right for my project?
At minimum, an evaluator for an ATE project should have basic social science or education research skills, and academic preparation or extensive practical experience in evaluation. Ideally, ATE project evaluators will also have experience with community colleges and knowledge of the project’s disciplinary area.
Keep in mind that there is no certification or credential for evaluators in the United States. Do not assume that just because a person calls themselves an “evaluator” or has evaluated a grant project in the past that they are qualified to evaluate your project. If possible, assess a potential evaluator’s qualifications before contacting them. Sometimes you can learn a lot about an evaluator’s credentials and experience by searching the web. For example, if the evaluator has a website, review it for evidence of their experience and expertise related to evaluation in general and your type of project in particular. Look for examples of reports, academic papers, presentations, and blogs.
If you find someone who looks promising, contact them to learn more. Here’s an example of what to say:
I am developing a proposal for the National Science Foundation’s Advanced Technological Education program and I’m looking for an evaluator who will help us with the evaluation plan. The project is about [insert super short description of what your project is about]. If you think you might be interested, may I [call or email] you with a few questions?
In that follow-up dialogue, here are examples of questions you may want to ask:
- What experience have you had evaluating STEM education or similar types of projects?
- What is your experience with community colleges?
- Do you have experience evaluating [insert discipline/content area] projects?
- Tell me about how you work with your clients.
- Who are some of your past clients?
- Pay attention not only to how they answer your questions, but the degree of rapport you feel in interacting with them. Successful client-evaluator relationships are grounded in open communication and respect. If this is missing from the start, there are likely to be problems down the road.
If it’s not possible to find someone with expertise in both your content area and evaluation, prioritize evaluation knowledge. All evaluators—regardless of their content area knowledge—should take time to learn about the specific contexts of the projects they work with. Evaluation expertise is needed throughout the evaluation process, while content area expertise is needed more intermittently. Without a strong background in evaluation, subject matter experts may be prone to making methodological errors that compromise evaluative findings. Evaluation conclusions should be based on systematically collected data more than the evaluator’s experience and opinion. If needed, evaluators may consult with content area experts to compensate for gaps in knowledge.
To learn more about what professional evaluators should know and be able to do, see the following resources:
- The Program Evaluation Standards: bit.ly/jc-pes
- American Evaluation Association Guiding Principles for Evaluators: bit.ly/aea-gp
- Competencies for Canadian Evaluation Practice (U.S. evaluation competencies are being drafted): bit.ly/10v3dc3
5. What should I do after I find the evaluator I want to work with?
First, confirm they want to proceed in working with you on the proposal. Then, ask what they need from you. Most likely, this will include the draft proposal, a timeline for completing the evaluation plan, and a ballpark figure for the evaluation budget (see Question 6). Allow time for one or two conversations with the evaluator, to make sure that you share a common understanding of the proposed project and what responsibilities each party will have for the evaluation. IMPORTANT! Provide the evaluator with a link to the ATE Program Solicitation (bit.ly/2017ATE) and the ATE Proposal Evaluation Planning Checklist (bit.ly/checklist-evalplan). The latter document includes details about the evaluation-related information needed for the proposal.
6. How much should I budget for the external evaluation?
A prospective evaluator will probably ask you how much your evaluation budget is. A rule of thumb is to dedicate 10 percent of a project’s direct costs to evaluation. Among ATE grant recipients, the average is 7 percent.
7. How do I compensate the evaluator for their assistance with the proposal?
Many evaluators are willing to help develop a proposal evaluation plan at no charge with the understanding that they will get the evaluation contract if the proposal is funded. Make this agreement explicit. If you do not get the grant, there will be no financial benefit to them, which is the nature of grant funding. Try to avoid making numerous demands for information and assistance (particularly if it is not specifically about evaluation), given that there is a cost to the evaluator (time) with uncertain benefits. Whether the proposal is funded or not, share the reviewers’ feedback with the evaluator. This will be valuable information for the evaluator’s professional development and is a type of compensation in and of itself.
8. The award notification has arrived – what happens next?
If your proposal is accepted, contact the evaluator right away. Begin the contracting process as soon as possible, since it will almost certainly take longer than you expect. Defer to your institution’s established contracting process and boilerplate contracts. Work with the evaluator to prepare a statement of work to append to the formal contract. The statement of work should specify the evaluation activities, deliverables, and timeline, elaborating on what was stated in the grant proposal. Once the contract is fully executed, the document will serve as the basis for developing a detailed and actionable evaluation plan.
We are grateful to Sharon Gusky, Mike Rudibaugh, and Brad Watts for their feedback on a draft version of this document.