Archive: external evaluator

Blog: What is the best way to coordinate internal and external evaluation?

Posted on December 11, 2020 by  in Blog

Executive Director, The Evaluation Center at Western Michigan University

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

All ATE projects are required to allocate funds for external evaluation services. So, when it comes to internal and external evaluation, the one certain thing is that you must have an external evaluator. (In rare cases, alternative arrangements may be approved by a program officer.)

There are two types of external evaluators:

Type 1: Evaluators who are completely external to the institution

Type 2 Evaluators who are external to the project, but internal to the institution (such as an institutional researcher or a faculty member from a different department from where the project is located).

Both types are considered external, as long as the Type 2 external evaluator is truly independent from the project.

An internal evaluator is a member of the project staff, who is directly funded by the project, such as a project manager. More commonly, internal evaluation is a shared responsibility among team members. There are many options for coordinating internal and external evaluation functions. Over the years, I have noted four basic approaches:

(1) External Evaluator as Coach: The external evaluator provides guidance and feedback to the internal project team throughout the life of the grant. This is a good approach when there is already some evaluation competence among team members. The external evaluator’s involvement enhances the credibility of the evaluation and helps the team continue to build their evaluation knowledge and skills.

(2) External Evaluator as Heavy-Lifter: The external evaluator takes the lead in planning the evaluation, designing instruments, analyzing results, and writing reports. The internal team mainly gathers data and provides it to the external evaluator for processing. In this approach, the external evaluator should provide clear-cut data collection protocols to ensure systematic collection and handling of data by the internal team before they turn the information over to the external evaluator.

(3) External Evaluator as Architect: The external evaluator designs the overall evaluation and develops data collection instruments. The project team executes the plan, with technical assistance from the external evaluator as needed—particularly at critical junctures in the evaluation such as analysis and reporting. With this approach, it is important to front-load the evaluation budget in the first year of the project to allow for intensive involvement by the external evaluator.

(4) Divide-and-Conquer: The internal team is responsible for evaluating project implementation and immediate results. The external evaluator handles the evaluation of longer-term outcomes. This is the approach that EvaluATE uses. We carefully track and analyze data about how well we are reaching and engaging our audience. We are responsible for assessing immediate outcomes of our webinars and workshops (i.e., participants’ satisfaction, self-reported learning, and intent to use content). Our external evaluator is responsible for determining and assessing EvaluATE’s impact on the evaluation practices of our users.

A note of caution: Taking on part of an evaluation internally is often seen as a means of conserving project resources, and it can have that effect. But do not make the mistake of thinking internal evaluation is cost-free. At minimum, it takes time, which is sometimes a rarer commodity than money. In short, there is no one best way to coordinate internal and external evaluation. Your approach should make sense for your project in light of available resources (including staff time and expertise) and what you need your evaluation to do for your project.

What to know more? Check out this 5-minute video in which I explain what counts as independent, how to find an external evaluator, and how to divide responsibilities between integrate internal and external evaluators: Evaluation Basics Part 4: Who Can Do It?

Blog: Evaluation Feedback Is a Gift

Posted on July 3, 2018 by  in Blog ()

Chemistry Faculty, Anoka-Ramsey Community College

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

I’m Christopher Lutz, chemistry faculty at Anoka-Ramsey Community College. When our project was initially awarded, I was a first-time National Science Foundation (NSF) principal investigator. I understood external evaluation was required for grants but saw it as an administrative hurdle in the grant process. I viewed evaluation as proof for the NSF that we did the project and as a metric for outcomes. While both of these aspects are important, I learned evaluation is also an opportunity to monitor and improve your process and grant. Working with our excellent external evaluators, we built a stronger program in our grant project. You can too, if you are open to evaluation feedback.

Our evaluation team was composed of an excellent evaluator and a technical expert. I started working with both about halfway through the proposal development process (a few months before submission) to ensure they could contribute to the project. I recommend contacting evaluators during the initial stages of proposal development and checking in several times before submission. This gives adequate time for your evaluators to develop a quality evaluation plan and gives you time to understand how to incorporate your evaluator’s advice. Our funded project yielded great successes, but we could have saved time and achieved more if we had involved our evaluators earlier in the process.

After receiving funding, we convened grant personnel and evaluators for a face-to-face meeting to avoid wasted effort at the project start. Meeting in person allowed us to quickly collaborate on a deep level. For example, our project evaluator made real-time adjustments to the evaluation plan as our academic team and technical evaluator worked to plan our project videos and training tools. Include evaluator travel funds in your budget and possibly select an evaluator who is close by. We did not designate travel funds for our Kansas-based evaluator, but his ties to Minnesota and understanding of the value of face-to-face collaboration led him to use some of his evaluation salary to travel and meet with our team.

Here are three ways we used evaluation feedback to strengthen our project:

Example 1: The first-year evaluation report showed a perceived deficiency in the project’s provision of hands-on experience with MALDI-MS instrumentation. In response, we had students make small quantities of liquid solution instead of giving pre-mixed solutions, and let them analyze more lab samples. This change required minimal time but led students to regard the project’s hands-on nature as a strength in the second-year evaluation.

Example 2: Another area for improvement was students’ lack of confidence in analyzing data. In response to this feedback, project staff create Excel data analysis tools and a new training activity for students to practice with literature data prior to analyzing their own. The subsequent year’s evaluation report indicated increased student confidence.

Example 3: Input from our technical evaluator allowed us to create videos that have been used in academic institutions in at least three US states, the UK’s Open University system, and Iceland.

Provided here are some overall tips:

  1. Work with your evaluator(s) early in the proposal process to avoid wasted effort.
  2. Build in at least one face-to-face meeting with your evaluator(s).

Review evaluation data and reports with the goal of improving your project in the next year.

Consider external evaluators as critical friends who are there to help improve your project. This will help move your project forward and help you have a greater impact for all.