This brief focuses on project/center evaluation and is divided into 4 sections. This section, Section 1, provides an overview of ATE expectations for evaluation and principal investigators’ responses that describe how they meet those requirements—who conducts the evaluations, how much money is spent on evaluations, and the extent to which these evaluations vary by characteristics such as the type of grant and type of evaluator conducting the evaluation. Section 2 describes PI perceptions of the utility of their evaluations and the extent to which PI perceptions of utility are related to the evaluation characteristics described in Section 1. Section 3 focuses on the activities of external evaluators — PI satisfaction with these evaluators, the relationship between PI ratings and standards for sound program evaluations, whether the PIs view their evaluations as meeting ATE intellectual merit requirements, and PIs’ characterizations of the attributes of their external evaluators. Section 4 draws together findings reported in Sections 1 to 3 to identify strengths and weaknesses of project-level evaluations and to suggest changes that appear likely to improve on current evaluation practices.
This study analyzes project-level evaluation practices occurring in the Advanced Technological Education program of the National Science Foundation. Of special interest in this study were factors thought to affect the quality and utility of evaluations such as the cost of evaluations, who engaged in evaluation planning, and the use of external evaluators. The ATE program requires project-level evaluations and provides guidelines regarding what evaluations can and should do. The report closes with a discussion of discrepancies between expectations and project level actions and the apparent strengths and weaknesses of project evaluations. Suggestions are offered on how to improve these evaluation practices.
This brief examines the major challenges and resolutions associated with ATE project implementation, as reported by project and center principal investigators (PIs). Ten challenges were identified. Of these, “difficulty recruiting students,” “changes in industry served,” and “lack of institutional administrative support/interest,” were identified as the most important. Examining projects and centers as two separate entities, “difficulty recruiting students” retained its number one ranking, although differences emerge d in rankings between projects and centers in the remaining issues. There were
also distinctions between ratings of challenges identified by PIs of 2- and 4-year institutions, with 2-year institutions ranking “difficulty recruiting students” as most important and 4-year institutions ranking “project/center staff/personnel turnover” as their greatest challenge. Overall, these findings suggest that a large majority of possible challenges to ATE project implementation were either not identified by PIs as important or had been at least partially resolved. Although the challenges discussed in this brief are important at the level of individual ATE projects and centers, none are substantial enough to raise concern on a programmatic level.
File: Click Here