This brief focuses on project/center evaluation and is divided into 4 sections. This section, Section 1, provides an overview of ATE expectations for evaluation and principal investigators’ responses that describe how they meet those requirements—who conducts the evaluations, how much money is spent on evaluations, and the extent to which these evaluations vary by characteristics such as the type of grant and type of evaluator conducting the evaluation. Section 2 describes PI perceptions of the utility of their evaluations and the extent to which PI perceptions of utility are related to the evaluation characteristics described in Section 1. Section 3 focuses on the activities of external evaluators — PI satisfaction with these evaluators, the relationship between PI ratings and standards for sound program evaluations, whether the PIs view their evaluations as meeting ATE intellectual merit requirements, and PIs’ characterizations of the attributes of their external evaluators. Section 4 draws together findings reported in Sections 1 to 3 to identify strengths and weaknesses of project-level evaluations and to suggest changes that appear likely to improve on current evaluation practices.
This study analyzes project-level evaluation practices occurring in the Advanced Technological Education program of the National Science Foundation. Of special interest in this study were factors thought to affect the quality and utility of evaluations such as the cost of evaluations, who engaged in evaluation planning, and the use of external evaluators. The ATE program requires project-level evaluations and provides guidelines regarding what evaluations can and should do. The report closes with a discussion of discrepancies between expectations and project level actions and the apparent strengths and weaknesses of project evaluations. Suggestions are offered on how to improve these evaluation practices.
This report, Volume 3 of the 2004 Annual Survey Report, focuses on the following fundamental elements of the ATE program:
1) What is the size and scope of work for ATE projects?
2) To what degree do ATE projects apply rigorous internal practices in their operations?
3) How extensive are ATE project collaborations?
4) How productive are ATE projects in terms of the primary ATE work categories?
5) What impact are ATE projects having on students?
This report, Volume 2 of the 2004 Annual Survey Report, specifically addresses the following fundamental elements of the ATE centers:
1) What are the size and the scope of work for ATE centers?
2) To what degree do ATE centers apply rigorous internal practices in their operations?
3) How extensive are ATE center collaborations?
4) How productive are ATE centers in terms of the primary ATE work categories?
5) What impact are ATE centers having on students?
This technical report presents the processes and procedures employed by The Evaluation Center in its annual survey of National Science Foundation ATE Program grantees; using the sixth annual survey (2005) data.
This brief examines the major challenges and resolutions associated with ATE project implementation, as reported by project and center principal investigators (PIs). Ten challenges were identified. Of these, “difficulty recruiting students,” “changes in industry served,” and “lack of institutional administrative support/interest,” were identified as the most important. Examining projects and centers as two separate entities, “difficulty recruiting students” retained its number one ranking, although differences emerge d in rankings between projects and centers in the remaining issues. There were
also distinctions between ratings of challenges identified by PIs of 2- and 4-year institutions, with 2-year institutions ranking “difficulty recruiting students” as most important and 4-year institutions ranking “project/center staff/personnel turnover” as their greatest challenge. Overall, these findings suggest that a large majority of possible challenges to ATE project implementation were either not identified by PIs as important or had been at least partially resolved. Although the challenges discussed in this brief are important at the level of individual ATE projects and centers, none are substantial enough to raise concern on a programmatic level.
File: Click Here