Archive: managing your evaluator

Resource: Finding and Selecting an Evaluator for Advanced Technological Education (ATE) Proposals

Posted on July 13, 2017 by  in Resources ()

All ATE proposals are required to request “funds to support an evaluator independent of the project.” Ideally, this external evaluator should be identified in the project proposal. The information in this guide is for individuals who are able to select and work with an external evaluator at the proposal stage. However, some institutions prohibit selecting an evaluator on a noncompetitive basis in advance of an award being made. Advice for individuals in that situation is provided in an EvaluATE blog and newsletter article.

This guide includes advice on how to locate and select an external evaluator. It is not intended as a guide for developing an evaluation plan or contracting with an evaluator.

File: Click Here
Type: Doc
Category: Resources
Author(s): Lori Wingate

Template: Evaluator Biographical Sketch

Posted on July 13, 2017 by  in Resources ()

This template was created by EvaluATE. It is based on the National Science Foundation’s guidelines for preparing biographical sketches for senior project personnel. The information about what evaluators should include in Products and Synergistic Activities sections are EvaluATE’s suggestions, not NSF requirements. The biosketch must not exceed two pages.

File: Click Here
Type: Worksheet
Category: Resources
Author(s): Lori Wingate

Blog: Mistakes Made and Lessons Learned, Part I – Working with Your Evaluator

Posted on March 18, 2015 by  in Blog ()

Director, Experiential Learning Center, Truckee Meadows Community College

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

When I assumed the PI-ship of the Scenario-Based Learning Project in 2006, I had worked closely with the prior PI as the project’s instructional designer, knew I enjoyed the ATE community, shared their vision of an innovative 21st century technician workforce, and had management and teaching experience. What more was there to know? A lot, as it turned out. It quickly became apparent to me that the world of ATE projects and centers was a different place than any I had worked in before.

When I took over as PI on the second grant proposal, the former PI suggested we use the evaluators from a previous non-ATE grant she had led. Big mistake. Those evaluators reported to my National Visiting Committee (NVC) during our initial committee meeting that they didn’t have any results to report because they did not plan to collect data until the end of the project year. My NVC was not happy. I was not happy. My stakeholders were not happy.

How did this happen? In my naivety, I didn’t even discuss the evaluation with the evaluators beyond an initial outline of a plan involving questions to be answered by the evaluation—I thought the evaluators knew what they were doing because they were evaluators. I didn’t understand the complex nature of the profession of evaluation. Since then I have joined the American Evaluation Association, attended their annual conference, and regularly attend EvaluATE’s webinars. I made a mistake, learned from it, and the project improved.

I quickly learned that some evaluators and funders are all about the summative report. The project said they would do A, here is the data to show they did or did not do A. End of report. In contrast, the ATE program is interested in how we are doing as we progress through our work. Formative reports from the evaluator serve as a check-in on where you are in your work plan and outcomes. Your evaluator needs to be a critical friend—an advisor who keeps a distance and is critical where needed yet still supportive with ideas, solutions, and contacts.

Choose an evaluator with ATE experience and expertise in collecting and analyzing the kind of data you will need. Confirm who will collect the data, how it will be collected, from whom, and when early in your discussions with your evaluator and project team. Confirm that your evaluator is willing to provide mentoring to your data collection team as needed if you decide to collect the data yourselves and have the evaluator do the analysis (saves money but requires time).

You might need interim reports/summaries from your evaluator for meetings with stakeholders, your NVC, advisory boards, the ATE annual survey, and your annual and final reports. It is a good idea to align your data collection with your reporting needs to best use your resources.

Learn more about the process of evaluation every chance you get. Choose an evaluator with the expertise appropriate to your project or center. Think of your evaluation as a resource and your evaluator as an ally to help you and your team to create the best project or center possible.

Blog: Managing Your Evaluator

Posted on November 12, 2014 by  in Blog () ()

Director, South Carolina Advanced Technological Education Center of Excellence, Florence-Darlington Technical College

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

I am Elaine Craft, Director of the SC ATE Center of Excellence since 1995 and President/CEO of SCATE Inc. since 2005. My dual roles mean that I am both a grantee and an evaluator. I’ve seen the ups, the downs, the good, and the bad on both sides of evaluation.

Managing your evaluator begins even before you contract for this service, as the contract sets parameters for the work ahead. It is your responsibility to see that your evaluator and the evaluation are serving your project well. Keep in mind that you will need to include much of the information the evaluator will be generating in the “Results of Prior Support” section of your next NSF ATE proposal!

It is helpful if your evaluator not only knows the essentials of project evaluation, but also understands the NSF ATE program and the two-year college environment. If you have an evaluator who hasn’t “walked a mile in your community college moccasins,” you will need to devote time to helping him or her understand your environment and the students you serve. There may also be terminology that is specific to community colleges, your institution, or your discipline that needs to be explained.

Everyone is busy, so scheduling should be a top priority. Share a copy of your institution’s calendar and discuss good times and bad times for certain activities. For example, the timing of student surveys is particularly sensitive to the academic calendar. Also, your evaluator may want to attend special project events such as advisory board meetings, professional development events, or summer camps. These dates should be scheduled with your evaluator as early as possible, as the evaluator is likely to have other clients and commitments that must be taken into consideration.

Make sure that you have a clear understanding with your evaluator about when reports are due. You should ask to receive your annual evaluation report before your annual report to the NSF is due. You will want to have time to review the report and work with your evaluator to correct any errors of fact before it is finalized and presented to the NSF or others.

Don’t settle for fewer evaluation services than you have contracted for, but also avoid adding things that were not in the original contract. The evaluator may be amenable to some modifications in the scope of work, but keeping your project and evaluation aligned with the original plan will help avoid mission creep for both the project and the evaluator.

Last, speak up! Your evaluator can’t adjust to better meet your expectations if you don’t articulate areas that are especially great and/or areas of concern. If both grantee and evaluator are on the same page and communicate often around the topics above, evaluation becomes a win-win for both. If your evaluator is not proactive in contacting you, you need to be proactive to keep communications flowing.

Tip: “Good advice is not often served in our favorite flavor.” Tim Fargo

An evaluator’s role is to see you better than you can see yourself. Let your evaluator know that you appreciate both accolades and guidance for improvement.