Newsletter - Winter 2016

Newsletter: Revisiting Intellectual Merit and Broader Impact

Posted on January 1, 2016 by  in Newsletter - () ()

Director of Research, The Evaluation Center at Western Michigan University

If you have ever written a proposal to the National Science Foundation (NSF) or participated in a proposal review panel for NSF, you probably instantly recognize the terms Intellectual Merit and Broader Impacts as NSF’s merit review criteria. Proposals are rated and funding decisions are made based on how well they address these criteria. Therefore, proposers must describe the potential of their proposed work to advance knowledge and understanding (Intellectual Merit) and benefit society (Broader Impacts).

Like cramming for an exam and then forgetting 90 percent of what you memorized, it’s all too easy for principal investigators to lose sight of Intellectual Merit and Broader Impacts after proposal submission. But there are two important reasons to maintain focus on Intellectual Merit and Broader Impacts after an award is made and throughout project implementation.

First, the goals and activities expressed in a proposal are commitments about how a particular project will advance knowledge (Intellectual Merit) and bring tangible benefits to individuals, institutions, communities, and/or our nation (Broader Impacts). Simply put, PIs have an ethical obligation to follow through on these commitments to the best of their abilities.

Second, when funded PIs seek subsequent grants from NSF, they must describe the results of their prior NSF funding in terms of Intellectual Merit and Broader Impacts. In other words, proposers must explain how they used their NSF funding to actually advance knowledge and understanding and benefit society. PIs who have evidence of their accomplishments in these areas and can convey it succinctly will be well-positioned to seek additional funding. To ensure evidence of both Intellectual Merit and Broader Impacts are being captured, PIs should revisit project evaluation plans with their evaluators, crosschecking the proposal’s claims about potential Intellectual Merit and Broader Impacts in relation to the evaluation questions and data collection plan to make sure compelling evidence is captured.

Last October, I conducted a workshop on this topic at the ATE Principal Investigators Conference with colleague Kirk Knestis, an evaluator from Hezel Associates. Dr. Celeste Carter, ATE program co-lead, spoke about how to frame results of prior NSF support in proposals. She noted that a common misstep that she has seen in proposals is when proposers speak to results from prior support by simply reiterating what they said they were going to do in their funded proposals, rather than describing the actual outcomes of the grant. Project summaries (one-page descriptions that address a proposed project’s Intellectual Merit and Broader Impacts that are required as part of all NSF proposals) are necessarily written in a prospective, future-oriented manner because the work hasn’t been initiated yet. In contrast, the Results of Prior NSF Support sections are about completed work and therefore are written in past tense and should include evidence of accomplishments. Describing achievements and presenting evidence of the quality and impact of those achievements shows reviewers that the proposer is a responsible steward of federal funds, can deliver on promises, and is building on prior success.

Take time now, well before it is time to submit a new proposal or a Project Outcomes Report, to make sure you haven’t lost sight of the Intellectual Merit and Broader Impact aspects of your grant and how you promised to contribute to these national priorities.

Newsletter: Survey Says Winter 2016

Posted on January 1, 2016 by , in Newsletter - ()

On the 2015 ATE survey, 65 of 230 principal investigators (28%) reported spending some portion of their annual budgets on research. Six of these projects were funded as targeted research. Among the other 59 projects, expenditures on research ranged from 1% to 65% with a median of 14%. With just six targeted research projects and less than a third of all ATE grantees engaging in research, there is immense opportunity within the ATE program to expand research on technician education.



The full report of 2015 ATE survey findings, along with data snapshots and downloadable graphics, is available from

Newsletter: Transformative

Posted on January 1, 2016 by  in Newsletter - ()

Director of Research, The Evaluation Center at Western Michigan University

NSF identifies five questions that proposal reviewers should consider in relation to the NSF merit criteria of Intellectual Merit and Broader Impacts.1 One of these questions is, “To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts?”

NSF defines transformative research as involving “ideas, discoveries, or tools that radically change our understanding of an important existing scientific or engineering concept or educational practice or leads to the creation of a new paradigm or field of science, engineering, or education. Such research challenges current understanding or provides pathways to new frontiers.”2

The Transformative Research section of the NSF website ( offers additional insights on this topic. It explains that transformative research, “challenges conventional wisdom; leads to unexpected insights that enable new techniques or methodologies; or redefines the boundaries of science, engineering, or education.”

Understanding what NSF means by “transformative” is important so that proposers and grantees use the term appropriately and do not accidentally overstate their project’s potential or actual achievements. While some projects may bring about important institutional transformation, that type of transformation is of a smaller scale than the “radical changes” in scientific understanding and practices associated with NSF’s definition. Claims related to “transformation” should be reserved for the truly extraordinary, revolutionary, and ground-breaking changes in understanding or practice.


Newsletter: How can PIs demonstrate that their projects have “advanced knowledge”?

Posted on January 1, 2016 by  in Newsletter - () ()

Director of Research, The Evaluation Center at Western Michigan University

NSF’s Intellectual Merit criterion is about advancing knowledge and understanding within a given field or across fields. Publication in peer-reviewed journals provides strong evidence of the Intellectual Merit of completed work. It is an indication that the information generated by a project is important and novel. The peer review process ensures that articles meet a journal’s standard of quality, as determined by a panel of reviewers who are subject matter experts.

In addition, publishing in an academic journal is the best way of ensuring that the new knowledge you have generated is available to others, becomes part of a shared scientific knowledge base, and is sustained over time. Websites and digital libraries tend to come and go with staff and funding changes. Journals are archived by libraries worldwide and, importantly, indexed to enable searches using standard search terms and logic. Even if a journal is discontinued, its articles remain available through libraries. Conference presentations are important dissemination vehicles, but don’t have the staying power of publishing. Some conferences publish presented papers in conference proceedings documents, which helps with long-term accessibility of information presented at these events.

The peer review process that journals employ to determine if they should publish a given manuscript is essentially an evaluative process. A small group of reviewers assesses the manuscript against criteria established for the journal. If the manuscript is accepted for publication, it met the specified quality threshold. Therefore, it is not necessary for the quality of published articles produced by ATE projects to be separately evaluated as part of the project’s external evaluation. However, it may be worthwhile to investigate the influence of published works, such as through citation analysis (i.e., determination of the impact of a published article based on the number of times it has been cited—to learn more, see

Journals focused on two-year colleges and technical education are good outlets for ATE-related publications. Examples include Community College Enterprise, Community College Research Journal, Community College Review, Journal of Applied Research in the Community College, New Directions for Community Colleges, Career and Technical Education Research, Journal of Career and Technical Education, and Journal of Education and Work. (For more options, see the list of journals maintained by the Center of Education and Work (CEW) at the University of Wisconsin at

NSF’s Intellectual Merit criterion is about contributing to collective knowledge. For example, if a project develops embedded math modules for inclusion in an electrical engineering e-book, students may improve their understanding of math concepts and how they relate to a technical task—and that is certainly important given the goals of the ATE program. However, if the project does not share what was learned about developing, implementing, and evaluating such modules and present evidence of their effectiveness so that others may learn from and build on those advances, the project hasn’t advanced disciplinary knowledge and understanding.

If you are interested in preparing a journal manuscript to disseminate knowledge generated by your project, first look at the type of articles that are being published in your field (check out CEW’s list of journals referenced above). You will get an idea of what is involved and how the articles are typically structured. Publishing can become an important part of a PI’s professional development, as well as a project’s overall effort to disseminate results and advance knowledge.

Newsletter: Communicating Results from Prior NSF Support

Posted on January 1, 2016 by  in Newsletter - () ()

Director of Research, The Evaluation Center at Western Michigan University

ATE proposal season is many months away in early October, but if you are submitting for new funding this year, now is the time to reflect on your project’s achievements and make sure you will be able to write a compelling account of your current or past project’s results as they relate to the NSF review criteria of Intellectual Merit and Broader Impacts. A section titled Results from Prior NSF Support is required whenever a proposal PI or co-PI has received previous grants from NSF in the past five years. A proposal may be returned without review if it does not use the specific headings of “Intellectual Merit” and “Broader Impacts” when presenting results from prior support.

Given that these specific headings are required, you should have something to say about your project’s achievements in these distinct areas. It is OK for some projects to emphasize one area over another (Intellectual Merit or Broader Impacts), but grantees should be able to demonstrate value in both areas. Descriptions of achievements should be supported with evidence. Bold statements about a proposed project’s potential broader impacts, for example, will be more convincing to reviewers if the proposer can describe tangible benefits of previously funded work.

To help with this aspect of proposal development, EvaluATE has created a Results from Prior NSF Support Checklist (see This one-page checklist lists the NSF requirements for this section of a proposal, as well as our additional suggestions for what to include and how.

Two EvaluATE blogs include additional guidance in this area: Amy Germuth ( offers specific guidance regarding wording and structure, and Lori Wingate ( shares tips for assessing the quality and quantity of evidence of a project’s Intellectual Merit and Broader Impacts, with links to helpful resources.

The task of identifying and collecting evidence of results from prior support should not wait until proposal writing time. It should be embedded in a project’s ongoing evaluation.

Newsletter: Project Spotlight: PATHTECH Successful Academic & Employment Pathways in Advanced Technologies

Posted on January 1, 2016 by  in Newsletter - () ()

Associate Professor of Sociology, University of South Florida

Will Tyson is PI for Path Tech, an ATE targeted research project. He is an associate professor of sociology at the University of South Florida. Learn more about his project at

Q: What advice do you have for PIs who want to pursue targeted research in technician education?

The Targeted Research on Technician Education strand of ATE is an ideal avenue for current ATE PIs looking to fund small projects to learn more about student outcomes resulting from prior activities. The best advice I have is to seek out scholars with backgrounds in social science and education, preferably with NSF experience, to partner with on a targeted research submission.

Q: You’ve published numerous articles on your research. What is your sense of what journal editors and reviewers are looking for when it comes to research on technician education?

I’m not sure journal editors and reviewers are actually looking for research on technician education. This is both a challenge and an opportunity. Most STEM education research generally ignores the “T” and focuses on traditional pathways to science, engineering, and mathematics degrees and careers. I think people know “good tech jobs” exist, but generally lack knowledge about the educational pathways to those jobs and the rich life stories of community college students in technician education programs.

Q: How do you see ATE research fitting within the NSF-IES Common Guidelines for Education Research and Development?

I think there are some challenges to fitting ATE research into the Common Guidelines. There are several research types and ATE researchers have to be careful to make sure the type they choose is the best fit for their research questions. The Guidelines are a good start for new investigators, but senior investigators should continue to build upon their work and use prior research to justify their new research ideas.

Q: Based on your experience as an NSF proposer and reviewer, what are some common mistakes when it comes to targeted research proposals?

Everyone should pay close attention to the goals of the Targeted Research on Technician Education track as outlined in the ATE program solicitation, which are to simulate and support research on technician education and build the partnership capacity between 2- and 4-year institutions to design and conduct research and development projects. All projects should focus on studying education through partnerships between 2- and 4-year institutions. In my experience, targeted research proposals tend to be led by 2-year college faculty or scholars from 4-year institutions or private research institutes. The 2-year personnel tend to lack the capacity to conduct targeted research due to lack of experience or personnel, as evidenced by their biosketches. On the other hand, 4-year personnel tend to lack familiarity with 2-year colleges and seek to use students as “guinea pigs.” Proposals often do not show that the scholar will be able to recruit student participants. Targeted research proposals should show clear evidence that 2- and 4-year institutions or private research institutes are going to work collaboratively.