Archive: goals

Blog: Getting Ready to Reapply – Highlighting Results of Prior Support

Posted on December 2, 2015 by  in Blog ()

Founder and President, EvalWorks, LLC

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Hello. My name is Amy A. Germuth and I own EvalWorks, LLC, an education evaluation firm in Durham, NC, which has a strong focus on evaluating STEM projects. Having conducted evaluations of ATE and multiple other NSF STEM projects since the early 2000s, I have worked with PIs to help them better respond to NSF solicitations.

For every ATE solicitation, NSF has required that proposers identify the “Results of Prior Support.” NSF requests that proposers provide the following information:

  1. The NSF award number, amount and period of support
  2. The title of the project
  3. A summary of the results of the completed work
  4. A list of publications resulting from the NSF award
  5. A brief description of available data, samples, physical collections, and other related research products not described elsewhere
  6. If the proposal is for renewal of a grant, a description of the relation of the completed work to the proposed work

This is an excellent opportunity for proposers who have been funded previously by NSF to highlight how their prior funds were used to support a positive change among the targeted group or individuals. For point 3, rather than simply stating the number of persons served, proposers should do the following:

  • State briefly the main goal(s) of the project.
  • Identify who was served, how many were served, and in what capacity.
  • Explain the impact on these persons that resulted from their participation in this project.
  • Provide what evidence was used to make the above inference.

An example may read something like this:

“As part of this project, our goal was to increase the number of women who successfully earned an associate’s degree in welding. To this end, we began a targeted recruiting campaign focusing on women who were about to complete or had recently completed other related programs such as pipefitting and construction and developed a brochure for new students that included positive images of women in welding. We used funding to develop the Women in Welding program and support team building and outreach efforts by them. Institutional data reveal that since this project was started, the number of women in the welding program has almost tripled from 12 (2006 – 2010), of which only 8 graduated to 34 (2011 – 2016), of which 17 have already graduated and 5 have only one semester left. Even if the remaining 17 were not to graduate, the 17 who already have is double the number of female students who graduated from the program between 2006 – 2010.”

To summarize, if you have received prior support from NSF, use this opportunity to show how the funding supported project activities that made a difference and how they inform your current proposal (if applicable). Reviewers look to this section as a way to ascertain the degree to which you have been a good steward of the funding that you received and what impacts it had. Attention to this section will provide one more measure by which reviewers will judge the ability of your proposed project to be successful.

Blog: Developing a Theory of Change for Your Innovation Project

Posted on September 23, 2015 by  in Blog ()

Senior Educational Researcher, SRI International

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Have you ever had to express a theory of change to convince funders that there’s a logical argument behind your innovation? Funders expect that data will be collected to evaluate whether innovations are meeting their goals. The goals should be tied to the innovation’s theory of change.

This type of thinking is very abstract, so the more you apply the abstractions to concrete situations, the clearer it will be. A useful exercise is to complete some sentences about your STEM education innovation. But first, try the same exercise for something that’s less intellectually challenging. In the table below, Columns C and D show examples of both. The sentence prompts are in Column B.

Zalles Chart.emf

Of course, theories of change and the evaluation strategies that arise from them can get very complicated quickly. For example, let’s say you’re writing a “proposal” to your best friend, who happens to like your kids and is ready to pay for their ice cream if he’s convinced that it will make them happy. Yet, your friend is the skeptical type. He argues, “Why just analyze what they say twice? Wouldn’t it help if you also did it the day after to see if they revert back to their usual remarks? Also, what if the kids simply decide to say fewer things after they eat the ice cream? How would you interpret fewer statements rather than nicer statements? Wouldn’t it be better to simply ask them to report how happy they are on a scale or tell them that they have to make at least four remarks or they’ll only get one scoop next time?”

In the second case provided in the table, let’s say your friend expresses concern about the random assignment. “What if the students won’t like being randomly assigned?” he says. Are there other ways you could get legitimate comparison data from which to draw conclusions? How about getting a participant group by asking for volunteers, then using a design task pretest to see how they match up with those who didn’t volunteer? Then, when analyzing the post task, you could limit your comparisons to groups of participants and nonparticipants to those whose designs on the pre-task were about the same quality.

In closing, it’s helpful to think of theory of change generation as an exercise in reflecting on and expressing the logic behind your innovation’s value. But first, do it around something ordinary, like getting more exercise, and be prepared for those tough questions!