Archive: efficacy research

Blog: Evaluating Impact: How I Moved From Pipeline to Interstate

Posted on December 10, 2014 by  in Blog ()

CKO of Sener Knowledge, LLC

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

John Sener of Sener Knowledge LLC, an external evaluator for over a dozen ATE and other NSF grants over the past eight years, discusses how to move beyond the “pipeline” metaphor for describing technician education and adopt the more useful “interstate” metaphor instead.

Evaluate an NSF ATE center or project long enough, and eventually you’ll hear the word pipeline—as in, “closing gaps in the education pipeline,” or “increasing the cybersecurity workforce pipeline.” The pipeline metaphor describes the ATE technician education process pretty well to some extent:

A) Students proceed through an NSF-funded ATE program “pipeline” in well-defined cohorts on a relatively uniform timetable.

B) The pipeline feeds program graduates to employers, often in a relatively limited number of local or regional employers in the related field.

C) Evaluators aim to document that B is the result of A.

However, I have long been dissatisfied with the pipeline metaphor because it traps its users into limited thinking, making it easy to overlook important sources of project impact. As noted in my book, The Seven Futures of American Education, many students exhibit characteristics that reflect a greater degree of choice than the pipeline metaphor implies: “stopouts” who drop in and out of school, “swirlers” who attend multiple institutions, “stay-longers” who exceed the prescribed time period for program completion. I’ve found that an interstate highway is a much more useful metaphor to understand learners who:

  • Choose among multiple entry and exit points rather than following a prescribed path;
  • Travel at different speeds and on different schedules through their education programs;
  • Sometimes seek alternate routes and multiple destinations during their educational journeys.

Here are some ways I use the interstate metaphor to find sources of project impact.

Pipelines move their cargo from Point A to Point B, while interstates support two-way traffic; look in both directions for indications of project impact, for instance:

  • Career changers with bachelor’s or graduate degrees returning to community colleges for additional training or certification
  • Four-year students mentoring two-year students to prepare for student competitions
  • Community college students mentoring high school students or serving as judges for high school student competitions
  • Business and industry practitioners getting involved on community college curriculum advisory boards or forming three-way partnerships with faculty and students to enhance the learning/assessment experience or create knowledge collaboratively

Expand the realm of acceptable outcomes. ATE projects have significant impact beyond program completion or employment placement. Students sometimes find jobs before completing a program. Alternatively, intermediate outcomes—such as certificate or multiple course completion—may also indicate progress, especially for students returning to college to enhance their existing professional prospects.

Look in other places for impact. Extracurricular activities, such as student competitions, clubs, or organizations, are one good place to look. I sometimes think of such places as “toll booths”—activities where impact can be measured more easily as students pass through them.

Checklist: Common Guidelines: Type 4 – Efficacy Research

Posted on December 10, 2014 by  in Checklist, Resources ()

This checklist is a distillation of key points from the Common Guidelines for Education Research and Development regarding Efficacy Research. The Guidelines, developed by the Institute of Education Sciences at the U.S. Department of Education and the National Science Foundation, explains those agencies’ shared expectations for education research and development. This checklist, created by EvaluATE, is intended to support use of the Guidelines, enabling users to quickly reference those that specifically relate to Foundational Research. As such, it provides an overview and orientation to the Guidelines. It does not replace the Guidelines nor does it expand or elaborate on that report’s content. The checklist’s content has been extracted (usually verbatim) from the full report. All checklist users are strongly encouraged to read the complete Guidelines.

The complete guide and checklists for the other five types of research outlined in the Guidelines are available here.

A graphic overview of the Common Guidelines is available here.

TYPE 4: EFFICACY RESEARCH to determine whether an intervention or strategy can improve outcomes under “ideal” conditions (e.g., with more implementation support, highly trained personnel, and/or more homogenous participants than is typical).

File: Click Here
Type: Checklist
Category: Resources
Author(s): EvaluATE