Finding and Selecting an Evaluator for ATE Proposals

An external evaluator is the person who will lead the design and implementation of the evaluation of your ATE project. The evaluation will include systematic collection and analysis of evidence related to the quality, effectiveness, and impact of the project. To be external, the evaluator must be independent of the project.

Proposal developers should contact an evaluator at least one month in advance of the proposal’s due date—earlier if possible. A good evaluation plan should be closely aligned with the project’s goals and activities. To achieve good alignment, the evaluator needs time to review a draft of the proposal, ask questions, and develop a sound evaluation plan. With short notice, some evaluators may offer to provide a generic evaluation plan. However, seasoned proposal reviewers will give your proposal a more favorable review if it has a well-integrated, tailored evaluation plan.

There is no list of vetted or approved evaluators for NSF projects. It is up to the proposal developer (who is usually the principal investigator) to locate an evaluator and determine if they are qualified and right for a project.

Here are three ways to locate a potential evaluator:

• Ask colleagues for recommendations: If you know someone with a grant that has an evaluation component, ask for the evaluator’s name and contact information.
• Use the American Evaluation Association’s evaluator directory: It’s searchable by state and keyword.
• Use ATE Central’s evaluator map: This interactive map can be used to identify evaluators by location and the types of ATE projects they evaluate.

Most ATE projects employ evaluators based outside of their home institutions. However, program rules do allow grant recipients to contract with an evaluator who is employed by the project’s home institution, as long as the evaluator is independent of the project. That is, the evaluator should not work in the same unit where the project is housed. However, neither the evaluator nor any of the project’s personnel should have supervisory responsibilities in relation to the other party.

At minimum, an evaluator for an ATE project should have basic social science or education research skills, and academic preparation or extensive practical experience in evaluation. Ideally, ATE project evaluators will also have experience with community colleges and knowledge of the project’s disciplinary area.

Keep in mind that there is no certification or credential for evaluators in the United States. Do not assume that just because a person uses the title “evaluator” or has evaluated a grant project in the past they are qualified to evaluate your project. If possible, assess a potential evaluator’s qualifications before contacting them. Sometimes you can learn a lot about an evaluator’s credentials and experience by searching the web. For example, if the evaluator has a website, review it for evidence of their experience and expertise related to evaluation in general and your type of project in particular. Look for examples of reports, academic papers, presentations, and blogs.

If you find someone who looks promising, contact them to learn more. Here’s an example of what to say:

I am developing a proposal for the National Science Foundation’s Advanced Technological Education program, and I’m looking for an evaluator who will help us with the evaluation plan. The project is about [insert super short description of what your project is about]. If you think you might be interested, may I [call or email] you with a few questions?

In that follow-up dialogue, here are examples of questions you may want to ask:

• What experience have you had evaluating STEM education or similar types of projects?
• What is your experience with community colleges?
• Do you have experience evaluating [insert discipline/content area] projects?
• Tell me about how you work with your clients.
• Who are some of your past clients?
• Pay attention not only to how they answer your questions, but the degree of rapport you feel in interacting with them. Successful client-evaluator relationships are grounded in open communication and respect. If this is missing from the start, there are likely to be problems down the road.

If it’s not possible to find someone with expertise evaluation and your content area, prioritize evaluation knowledge. All evaluators—regardless of their content area knowledge—should take time to learn about the specific contexts of the projects they work with. Evaluation expertise is needed throughout the evaluation process, while content area expertise is needed more intermittently. Without a strong background in evaluation, subject matter experts may be prone to making methodological errors that compromise evaluative findings. Evaluation conclusions should be based on systematically collected data more than the evaluator’s experience and opinion. If needed, evaluators may consult with content area experts to compensate for gaps in knowledge.

To learn more about what professional evaluators should know and be able to do, see the following resources:

• American Evaluation Association Evaluation Competencies
• The Program Evaluation Standards
• American Evaluation Association Guiding Principles for Evaluators
• Competencies for Canadian Evaluation Practice

First, confirm they want to proceed in working with you on the proposal. Then, ask what they need from you. Most likely, this will include the draft proposal, a timeline for completing the evaluation plan, and a ballpark figure for the evaluation budget. Allow time for one or two conversations with the evaluator, to make sure that you share a common understanding of the proposed project and what responsibilities each party will have for the evaluation. IMPORTANT! Provide the evaluator with a link to the ATE program solicitation and the ATE evaluation plan checklist. The latter document includes details about the evaluation-related information needed for the proposal.

A prospective evaluator will probably ask you how much your evaluation budget is. The cost of an evaluation should be consistent with the scope of the evaluation effort. ATE evaluations are generally between 4 and 10% of a project’s direct costs.

Many evaluators are willing to help develop a proposal evaluation plan at no charge, with the understanding that they will get the evaluation contract if the proposal is funded. Make this agreement explicit. If you do not get the grant, there will be no financial benefit to the evaluator; this is the nature of grant funding. Try to avoid making numerous requests for information and assistance (particularly if your questions are not specifically about evaluation), given the evaluator is investing time in your project with uncertain benefits. Whether the proposal is funded or not, share the reviewers’ feedback with the evaluator. This will be valuable information for the evaluator’s professional development and is a type of compensation in and of itself.

If your proposal is accepted, contact the evaluator right away. Begin the contracting process as soon as possible, since it will almost certainly take longer than you expect. Defer to your institution’s established contracting process and boilerplate contracts. Work with the evaluator to prepare a statement of work to append to the formal contract. The statement of work should specify the evaluation activities, deliverables, and timeline, elaborating on what was stated in the grant proposal. Once the contract is fully executed, the document will serve as the basis for developing a detailed and actionable evaluation plan.