ATE Evaluation Primer

Posted on April 6, 2020 by  in

Updated September 2013.

Introduction

Evaluation is the systematic determination of the merit, worth, or significance of something. (The “something” of interest here is an ATE project or center but, for simplicity, we’ll just use the term “project” in the rest of this document). “Merit” is the inherent quality of a project, basically how “good” it is. “Worth” refers to how well the project meets a need in its present context in relation to its cost and value, basically how “worthwhile” it is. “Significance” refers to how important or groundbreaking a project is. Systematic evaluations are guided by questions and involve the collection of data and analysis of these data to reach conclusions about a project’s performance on one or more of these dimensions.

ATE evaluations should be keyed to grantees’ information needs and NSF accountability requirements. For maximum utility, ATE evaluations should describe and assess project processes (content and implementation) and outcomes (what happened as a result of the project) in such a way that project personnel can use the information to improve their work and be accountable for their grant funding.

What qualifies someone to conduct an ATE evaluation?

Evaluators have diverse academic and professional backgrounds. Few evaluators actually have a degree in “evaluation” per se. Although not a precise measure, an evaluator’s resume typically includes the terms “evaluation” or “assessment” in describing academic preparation or work history. Where such indicators are absent, more investigation is needed to ensure the person has the requisite knowledge and skills to be a competent evaluator.

Previous evaluation work should be readily evident in an experienced evaluator’s resume. Subject matter experts and research methods experts have valuable knowledge and skills that may complement an evaluator’s competencies. However, if they do not also have practical evaluation experience, their evaluation skills may be short of what is required for a comprehensive, sound, and useful project evaluation.

It is always a good idea to ask for references and work samples before contracting with an evaluator to ensure there is a good match between the needs of the project and what the evaluator can deliver.

Where can ATE grantees find qualified evaluators?

A good place to start is the American Evaluation Association’s Directory of Evaluators, available at www.eval.org. It is searchable by keyword and geographic location. AEA also has a list of graduate programs in evaluation, which also is a means for locating university faculty and graduate students with evaluation expertise. Grantees can check with local universities to find out if they have research centers or institutes that engage in evaluation. Some university websites have listings of areas of expertise for individual faculty members.

EvaluATE maintains a directory of evaluators with experience in STEM education and community college contexts, available from evalu-ate.org/community/evaluator_directory/. EvaluATE does not recommend specific individuals or firms for evaluation work.

Is it OK for grantees to use internal evaluators?

Yes (with caveats), to supplement but not replace your external evaluator. The project must allocate some funds to support an external evaluation (as noted in the ATE program solicitation). Internal evaluation and external evaluation may be considered as existing on a continuum. At the internal extreme, evaluative activities are carried out by project staff who draw their salaries from the personnel portion of the grant budget and are supervised by other project personnel. At the other extreme, someone outside the host institution with no prior affiliation with the funded project is hired to conduct the evaluation under a separate subcontract or contract. However, there is much “gray area” between these extremes, such as when someone is hired to conduct an evaluation from within the same host institution, but who isn’t on the project’s staff. Whether an evaluator is considered internal or external, the involved parties should address potential conflicts of interest directly and take steps to minimize their influence on the evaluation.

That said, there are ways to use internal evaluation to maximize the use of evaluation resources while maintaining the credibility of the evaluation. For example, an external evaluation  consultant could be hired to guide staff in the development of the initial evaluation design, with regular check-ins to assess how the evaluation is going and give advice to the internal evaluator as needed. This evaluation coach role can be especially helpful in designing data collection strategies and instruments, developing criteria for assessing project success, and ensuring that data are property analyzed and reported. This arrangement is appropriate for projects that have personnel available to spend time on evaluation tasks, but who have minimal experience with evaluation.

Another option is to hire an external metaevaluator—someone who evaluates the internal evaluation. A person in this type of role would play a less direct role in the design of the evaluation, but would provide feedback on the quality of the evaluation’s design, implementation, instruments, and reports. This arrangement may be appropriate for projects that have strong internal capacity for evaluation. If an evaluation is to be carried out mostly internally, it would be especially important to get this metaevaluator’s “stamp of approval” on the plan before it is implemented.

Because of perceptions that conflicts of interest are inherent with internal evaluation, internal
evaluation activities should be especially transparent—especially with regard to how data were
collected, analyzed, and interpreted—to enhance the credibility of the evaluation.

How much of a grant’s budget should be devoted to evaluation?

The general guideline is that 7 to 10 percent of a project’s direct costs should be allocated for
evaluation. Current expenditures on evaluation among ATE grantees average around 8 percent.
Prospective evaluation clients often want to know simply how much an evaluation costs or how much an evaluator should be paid for his or her time. However, these costs depend on the scale of the evaluation and the experience and expertise of the evaluator. The costs should be tied to specific deliverables and activities appropriate to the scope and goals of your project.

What do ATE grantees and NSF program officers expect to see in grant-level evaluation reports?

At a minimum, an evaluation report should describe what need the project is addressing and how the project is addressing that need, what the project accomplished, how “good” the results are, and how the conclusions were reached (what and how data were collected, how they were analyzed, what criteria were used to interpret the results). Evaluation clients often appreciate the inclusion of actionable recommendations, but recommendations should be based on the evaluation results, not only on the evaluator’s expert opinion. Grantees should expect more than a detailed description of their project activities and a regurgitation of data they routinely collect. Evaluations should yield information that adds to the project staff’s understanding of what they are doing, how well they are doing it, and what could be done to improve the project’s effectiveness.

Why do grantees have to complete the annual survey conducted by Western Michigan University in addition to submitting annual reports and having grant-level evaluations?

The annual survey data is used by NSF in its reports to Congress and other federal agencies to justify the program and its continuation/expansion. Although there is some overlap between the information required for the annual survey and ATE annual reports, the survey is much more specific about ATE activities and outcomes. Moreover, there is no way to aggregate information from annual reports submitted via Research.gov into a report about the overall ATE program. The annual survey takes place each year in February and March so that grantees can use the information they submit to the survey in their annual reports, which are due in April. The survey questions also may be used to guide data collection for the project evaluation.

What other evaluation tasks are required for NSF ATE grants?

All ATE grantees are expected to complete an annual survey conducted by EvaluATE, which takes place February-March. All grantees must submit an annual project report to NSF via Research.gov. ATE grantees are asked to upload their annual evaluation reports with their annual report submitted via Research.gov. For ATE centers, an additional evaluative report is developed annually by the center’s National Visiting Committee (NVC) to the NSF program officer. The NVC report provides an industry focused perspective on the success of a project and often makes recommendations on how to accomplish its goals.

Where can I learn more about evaluation?

The following resources are especially helpful for orienting evaluation clients and consumers to what they can and should expect from professional evaluation services; they provide a practical, nontechnical orientation to evaluation and matters related to the professional conduct of  evaluators:

NSF 2010 User-Friendly Handbook for Project Evaluation
American Evaluation Association’s Guiding Principles
Program Evaluation Standards
Competencies for Canadian Evaluation Practice

Download

Evaluation Primer