Archive: common guidelines

Blog: Evidence and Evaluation in STEM Education: The Federal Perspective

Posted on August 12, 2015 by  in Blog ()

Evaluation Manager, NASA Office of Education

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

If you have been awarded federal grants over many years, you probably have seen the increasing emphasis on evaluation and evidence. As a federal evaluator working at NASA, I have seen firsthand the government-wide initiative to increase use of evidence to improve social programs. Federal agencies have been strongly encouraged by the administration to better integrate evidence and rigorous evaluation into their budget, management, operational, and policy decisions by:

(1) making better use of already-collected data within government agencies; (2) promoting the use of high-quality, low-cost evaluations and rapid, iterative experimentation; (3) adopting more evidence-based structures for grant programs; and (4) building agency evaluation capacity and developing tools to better communicate what works. (https://www.whitehouse.gov/omb/evidence)

Federal STEM education programs have also been affected by this increasing focus on evidence and evaluation. Read, for example, the Federal Science, Technology, Engineering and Mathematics (STEM) Education 5-Year Strategic Plan (2013),1 which was prepared by the Committee on STEM Education of the National Science and Technology Council.2 This strategic plan provides an overview of the importance of STEM education to American society and describes the current state of federal STEM education efforts. Five priority STEM education investment areas are discussed where a coordinated federal strategy is currently under development. The plan also presents methods to build and share evidence. Finally, the plan lays out several strategic objectives for improving the exploration and sharing of evidence-based practices, including supporting syntheses of existing research that can inform federal investments in the STEM education priority areas, improving and aligning evaluation and research expertise and strategies across federal agencies, and streamlining processes for interagency collaboration (e.g., Memoranda of Understanding, Interagency Agreements).

Another key federal document that is influencing evaluation in STEM agencies is the Common Guidelines for Education Research and Development (2013),3 jointly prepared by the U.S. Department of Education’s Institute of Education Sciences and the National Science Foundation. This document describes the two agencies’ shared understandings of the roles of various types of research in generating evidence about strategies and interventions for increasing student learning. These research types range from studies that generate fundamental understandings related to education and learning to research (“Foundational Research”) to studies that assesses the impact of an intervention on an education-related outcome, including efficacy research, effectiveness research, and scale-up research. The Common Guidelines provide the two agencies and the broader education research community with a common vocabulary to describe the critical features of these study types.

Both documents have shaped, and will continue to shape, federal STEM programs and their evaluations. Reading them will help federal grantees gain a richer understanding of the larger federal context that is influencing reporting and evaluation requirements for grant awards.

1 A copy of the Federal Science, Technology, Engineering and Mathematics (STEM) Education 5-Year Strategic Plan can be obtained here: https://www.whitehouse.gov/sites/default/files/microsites/ostp/stem_stratplan_2013.pdf

2 For more information on the Committee on Science, Technology, Engineering, and Math Education, visit https://www.whitehouse.gov/administration/eop/ostp/nstc/committees/costem.

3 A copy of the Common Guidelines for Education Research and Development can be obtained here: http://ies.ed.gov/pdf/CommonGuidelines.pdf

Checklist: The Common Guidelines for Education Research and Development

Posted on January 28, 2015 by  in Resources ()

This document includes a series of six checklists—one for each of the six types of research outlined in the Common Guidelines for Education Research and Development. The Guidelines, developed by the Institute of Education Sciences at the U.S. Department of Education and the National Science Foundation, explains those agencies’ shared expectations for education research and development. The checklists, created by EvaluATE, are distillations of key points from the Guidelines. The checklists are intended to support use of the Guidelines, enabling users to quickly reference a type of research and determine whether they are following guideline’s expectations. As such, they provide an overview and orientation to the Guidelines. They do not replace that report nor do they expand or elaborate on the report’s content. The checklists’ content has been extracted (usually verbatim) from the full report. All checklist users are strongly encouraged to read the complete Guidelines, available from http://bit.ly/nsf-ies_guide.

A graphic overview of the Common Guidelines is available here.

You may download the entire guidelines checklist or go directly to the checklist for each type of research by clicking on the links below:

1. Foundational Research to advance the frontiers of education and learning; develop and refine theory and methodology; and provide fundamental knowledge about teaching and/or learning

2. Early-Stage or Exploratory Research to investigate approaches to education problems to establish the basis for design and development of new interventions or strategies and/or to provide evidence for whether an established intervention or strategy is ready to be tested in an efficacy study

3. Design and Development Research to develop new or improved interventions or strategies to achieve well-specified learning goals or objectives, including making refinements on the basis of small-scale testing

4. Efficacy Research to determine whether an intervention or strategy can improve outcomes under “ideal” conditions (e.g., with more implementation support, highly trained personnel, and/or more homogenous participants than is typical)

5. Effectiveness Research to estimate the impacts of an intervention or strategy when implemented under conditions of routine practice (i.e., conditions similar to what would occur if a study were not being conducted)

6. Scale-Up Research to estimate the impacts of an intervention or strategy under conditions of routine practice and across a broad spectrum of populations and settings, sufficiently diverse to broadly generalize findings

Checklist: Common Guidelines: Type 1 – Foundational Research

Posted on December 10, 2014 by  in Resources ()

This checklist is a distillation of key points from the Common Guidelines for Education Research and Development regarding Foundational Research. The Guidelines, developed by the Institute of Education Sciences at the U.S. Department of Education and the National Science Foundation, explains those agencies’ shared expectations for education research and development. This checklist, created by EvaluATE, is intended to support use of the Guidelines, enabling users to quickly reference those that specifically relate to Foundational Research. As such, it provides an overview and orientation to the Guidelines. It does not replace the Guidelines nor does it expand or elaborate on that report’s content. The checklist’s content has been extracted (usually verbatim) from the full report. All checklist users are strongly encouraged to read the complete Guidelines.

The complete guide and checklists for the other five types of research outlined in the Guidelines are available here.

A graphic overview of the Common Guidelines is available here.

TYPE 1: FOUNDATIONAL RESEARCH to advance the frontiers of education and learning; develop and refine theory and methodology; and provide fundamental knowledge about teaching and/or learning.

File: Click Here
Type: Checklist
Category: Resources
Author(s): EvaluATE

Checklist: Common Guidelines: Type 2 – Early Stage or Exploratory Research

Posted on December 10, 2014 by  in Resources ()

This checklist is a distillation of key points from the Common Guidelines for Education Research and Development regarding Early-Stage or Exploratory Research. The Guidelines, developed by the Institute of Education Sciences at the U.S. Department of Education and the National Science Foundation, explains those agencies’ shared expectations for education research and development. This checklist, created by EvaluATE, is intended to support use of the Guidelines, enabling users to quickly reference those that specifically relate to Foundational Research. As such, it provides an overview and orientation to the Guidelines. It does not replace the Guidelines nor does it expand or elaborate on that report’s content. The checklist’s content has been extracted (usually verbatim) from the full report. All checklist users are strongly encouraged to read the complete Guidelines.

The complete guide and checklists for the other five types of research outlined in the Guidelines are available here.

A graphic overview of the Common Guidelines is available here.

TYPE 2: EARLY-STAGE OR EXPLORATORY RESEARCH to investigate approaches to education problems to establish the basis for design and development of new interventions or strategies and/or to provide evidence for whether an established intervention or strategy is ready to be tested in an efficacy study.

File: Click Here
Type: Checklist
Category: Resources
Author(s): EvaluATE

Checklist: Common Guidelines: Type 3 – Design and Development Research

Posted on December 10, 2014 by  in Resources ()

This checklist is a distillation of key points from the Common Guidelines for Education Research and Development regarding Design and Development Research. The Guidelines, developed by the Institute of Education Sciences at the U.S. Department of Education and the National Science Foundation, explains those agencies’ shared expectations for education research and development. This checklist, created by EvaluATE, is intended to support use of the Guidelines, enabling users to quickly reference those that specifically relate to Foundational Research. As such, it provides an overview and orientation to the Guidelines. It does not replace the Guidelines nor does it expand or elaborate on that report’s content. The checklist’s content has been extracted (usually verbatim) from the full report. All checklist users are strongly encouraged to read the complete Guidelines.

The complete guide and checklists for the other five types of research outlined in the Guidelines are available here.

A graphic overview of the Common Guidelines is available here.

TYPE 3: DESIGN AND DEVELOPMENT RESEARCH to develop new or improved interventions or strategies to achieve well-specified learning goals or objectives, including making refinements on the basis of small-scale testing

File: Click Here
Type: Checklist
Category: Resources
Author(s): EvaluATE

Checklist: Common Guidelines: Type 4 – Efficacy Research

Posted on December 10, 2014 by  in Resources ()

This checklist is a distillation of key points from the Common Guidelines for Education Research and Development regarding Efficacy Research. The Guidelines, developed by the Institute of Education Sciences at the U.S. Department of Education and the National Science Foundation, explains those agencies’ shared expectations for education research and development. This checklist, created by EvaluATE, is intended to support use of the Guidelines, enabling users to quickly reference those that specifically relate to Foundational Research. As such, it provides an overview and orientation to the Guidelines. It does not replace the Guidelines nor does it expand or elaborate on that report’s content. The checklist’s content has been extracted (usually verbatim) from the full report. All checklist users are strongly encouraged to read the complete Guidelines.

The complete guide and checklists for the other five types of research outlined in the Guidelines are available here.

A graphic overview of the Common Guidelines is available here.

TYPE 4: EFFICACY RESEARCH to determine whether an intervention or strategy can improve outcomes under “ideal” conditions (e.g., with more implementation support, highly trained personnel, and/or more homogenous participants than is typical).

File: Click Here
Type: Checklist
Category: Resources
Author(s): EvaluATE

Checklist: Common Guidelines: Type 5 – Effectiveness Research

Posted on December 10, 2014 by  in Resources ()

This checklist is a distillation of key points from the Common Guidelines for Education Research and Development regarding <strong>Effectiveness Research</strong>. The Guidelines, developed by the Institute of Education Sciences at the U.S. Department of Education and the National Science Foundation, explains those agencies’ shared expectations for education research and development. This checklist, created by EvaluATE, is intended to support use of the Guidelines, enabling users to quickly reference those that specifically relate to Foundational Research. As such, it provides an overview and orientation to the Guidelines. It does not replace the Guidelines nor does it expand or elaborate on that report’s content. The checklist’s content has been extracted (usually verbatim) from the full report. All checklist users are strongly encouraged to read the complete Guidelines.

The complete guide and checklists for the other five types of research outlined in the Guidelines are available here.

A graphic overview of the Common Guidelines is available here.

TYPE 5: EFFECTIVENESS RESEARCH to estimate the impacts of an intervention or strategy when implemented under conditions of routine practice (i.e., conditions similar to what would occur if a study were not being conducted).

File: Click Here
Type: Checklist
Category: Resources
Author(s): EvaluATE

Checklist: Common Guidelines Type 6 – Scale-Up Research

Posted on December 10, 2014 by  in Resources ()

This checklist is a distillation of key points from the Common Guidelines for Education Research and Development regarding Scale-Up Research. The Guidelines, developed by the Institute of Education Sciences at the U.S. Department of Education and the National Science Foundation, explains those agencies’ shared expectations for education research and development. This checklist, created by EvaluATE, is intended to support use of the Guidelines, enabling users to quickly reference those that specifically relate to Foundational Research. As such, it provides an overview and orientation to the Guidelines. It does not replace the Guidelines nor does it expand or elaborate on that report’s content. The checklist’s content has been extracted (usually verbatim) from the full report. All checklist users are strongly encouraged to read the complete Guidelines.

The complete guide and checklists for the other five types of research outlined in the Guidelines are available here.

A graphic overview of the Common Guidelines is available here.

TYPE 6: SCALE-UP RESEARCH to estimate the impacts of an intervention or strategy under conditions of routine practice and across a broad spectrum of populations and settings, sufficiently diverse to broadly generalize findings

File: Click Here
Type: Checklist
Category: Resources
Author(s): EvaluATE