Archive: knowledge

Blog: Addressing Challenges in Evaluating ATE Projects Targeting Outcomes for Educators

Posted on November 21, 2017 by  in Blog ()

CEO, Hezel Associates

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Kirk Knestis—CEO of Hezel Associates and ex-career and technology educator and professional development provider—here to share some strategies addressing challenges unique to evaluating Advanced Technological Education (ATE) projects that target outcomes for teachers and college faculty.

In addition to funding projects that directly train future technicians, the National Science Foundation (NSF) ATE program funds initiatives to improve abilities of grade 7-12 teachers and college faculty—the expectation being that improving their practice will directly benefit technical education. ATE tracks focusing on professional development (PD), capacity building for faculty, and technological education teacher preparation all count implicitly on theories of action (typically illustrated by a logic model) that presume outcomes for educators will translate into outcomes for student technicians. This assumption can present challenges to evaluators trying to understand how such efforts are working. Reference this generic logic model for discussion purposes:

Setting aside project activities acting directly on students, any strategy aimed at educators (e.g., PD workshops, faculty mentoring, or preservice teacher training) must leave them fully equipped with dispositions, knowledge, and skills necessary to implement effective instruction with students. Educators must then turn those outcomes into actions to realize similar types of outcomes for their learners. Students’ action outcomes (e.g., entering, persisting in, and completing training programs) depend, in turn, on them having the dispositions, knowledge, and skills educators are charged with furthering. If educators fail to learn what they should, or do not activate those abilities, students are less likely to succeed. So what are the implications—challenges and possible solutions—of this for NSF ATE evaluations?

  • EDUCATOR OUTCOMES ARE OFTEN NOT WELL EXPLICATED. Work with program designers to force them to define the new dispositions, understandings, and abilities that technical educators require to be effective. Facilitate discussion about all three outcome categories to lessen the chance of missing something. Press until outcomes are defined in terms of persistent changes educators will take away from project activities, not what they will do during them.
  • EDUCATORS ARE DIFFICULT TO TEST. To truly understand if an ATE project is making a difference in instruction, it is necessary to assess if precursor outcomes for them are realized. Dispositions (attitudes) are easy to assess with self-report questionnaires, but measuring real knowledge and skills requires proper assessments—ideally, performance assessments. Work with project staff to “bake” assessments into project strategies, to be more authentic and less intrusive. Strive for more than self-report measures of increased abilities.
  • INSTRUCTIONAL PRACTICES ARE DIFFICULT AND EXPENSIVE TO ASSESS. The only way to truly evaluate instruction is to see it, assessing pedagogy, content, and quality with rubrics or checklists. Consider replacing expensive on-site visits with the collection of digital videos or real-time, web-based telepresence.

With clear definitions of outcomes and collaboration with ATE project designers, evaluators can assess whether technician training educators are gaining the necessary dispositions, knowledge, and skills, and if they are implementing those practices with students. Assessing students is the next challenge, but until we can determine if educator outcomes are being achieved, we cannot honestly say that educator-improvement efforts made any difference.

Newsletter: How can PIs demonstrate that their projects have “advanced knowledge”?

Posted on January 1, 2016 by  in Newsletter - () ()

Director of Research, The Evaluation Center at Western Michigan University

NSF’s Intellectual Merit criterion is about advancing knowledge and understanding within a given field or across fields. Publication in peer-reviewed journals provides strong evidence of the Intellectual Merit of completed work. It is an indication that the information generated by a project is important and novel. The peer review process ensures that articles meet a journal’s standard of quality, as determined by a panel of reviewers who are subject matter experts.

In addition, publishing in an academic journal is the best way of ensuring that the new knowledge you have generated is available to others, becomes part of a shared scientific knowledge base, and is sustained over time. Websites and digital libraries tend to come and go with staff and funding changes. Journals are archived by libraries worldwide and, importantly, indexed to enable searches using standard search terms and logic. Even if a journal is discontinued, its articles remain available through libraries. Conference presentations are important dissemination vehicles, but don’t have the staying power of publishing. Some conferences publish presented papers in conference proceedings documents, which helps with long-term accessibility of information presented at these events.

The peer review process that journals employ to determine if they should publish a given manuscript is essentially an evaluative process. A small group of reviewers assesses the manuscript against criteria established for the journal. If the manuscript is accepted for publication, it met the specified quality threshold. Therefore, it is not necessary for the quality of published articles produced by ATE projects to be separately evaluated as part of the project’s external evaluation. However, it may be worthwhile to investigate the influence of published works, such as through citation analysis (i.e., determination of the impact of a published article based on the number of times it has been cited—to learn more, see

Journals focused on two-year colleges and technical education are good outlets for ATE-related publications. Examples include Community College Enterprise, Community College Research Journal, Community College Review, Journal of Applied Research in the Community College, New Directions for Community Colleges, Career and Technical Education Research, Journal of Career and Technical Education, and Journal of Education and Work. (For more options, see the list of journals maintained by the Center of Education and Work (CEW) at the University of Wisconsin at

NSF’s Intellectual Merit criterion is about contributing to collective knowledge. For example, if a project develops embedded math modules for inclusion in an electrical engineering e-book, students may improve their understanding of math concepts and how they relate to a technical task—and that is certainly important given the goals of the ATE program. However, if the project does not share what was learned about developing, implementing, and evaluating such modules and present evidence of their effectiveness so that others may learn from and build on those advances, the project hasn’t advanced disciplinary knowledge and understanding.

If you are interested in preparing a journal manuscript to disseminate knowledge generated by your project, first look at the type of articles that are being published in your field (check out CEW’s list of journals referenced above). You will get an idea of what is involved and how the articles are typically structured. Publishing can become an important part of a PI’s professional development, as well as a project’s overall effort to disseminate results and advance knowledge.