Archive: evaluative thinking

Blog: Making the Most of Virtual Conferences: An Exercise in Evaluative Thinking

Posted on September 2, 2020 by  in Blog ()

Research Associate, Western Michigan University

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

We at EvaluATE affectionately call the fall “conference season.” Both the ATE PI Conference and the American Evaluation Association’s annual conference usually take place between October and November every year. This year, both conferences will be virtual events. Planning how our project will engage in this new virtual venue got me thinking: What makes a virtual conference successful for attendees? What would make a virtual conference successful for me?

I started by considering what makes an in-person conference successful, and I quickly realized that this was an exercise in evaluative thinking. The concept of evaluative thinking has been defined in a variety of ways—as a “type of reflective practice” (Baker & Bruner, 2012, p. 1), a combination of “critical thinking, creative thinking, inferential thinking, and practical thinking” (Patton, 2018, p. 21), and a “problem-solving approach” (Vo, 2013, p. 105). In this case, I challenged myself to consider what my personal evaluation criteria would be for a successful conference and what my ideal outcomes would look like.

In my reflection process, I came up with a list of key outcomes for attending a conference. Specifically, at conferences, I hope to:

  • build new relationships with peers;
  • grow relationships with existing partners;
  • learn about new trends in research and practice;
  • learn about future research opportunities (places I might be able to fill in the gaps); and
  • feel part of a community and re-energized about my work.

I realized that many of these outcomes are typically achieved through happenstance. For example, at previous conferences, most of my new relationships with peers occurred because of a hallway chat or because I sat next to someone in a session and we struck up a conversation and exchanged information. It’s unlikely these situations would occur organically in a virtual conference setting. I would need to be intentional about how I participated in a virtual conference to achieve the same outcomes.

I began to work backwards to determine what actions I could take to ensure I achieved these outcomes in a virtual conference format. In true evaluator fashion, I constructed a logic model for my virtual conference experience (shown in Figure 1). I realized I needed to identify specific activities—agreements with myself—to get the most out of the experience and have a successful virtual conference.

For example, one of my favorite parts of a conference is feeling like I am part of a larger community and becoming re-energized about my work. Being at home, it can be easy to become distracted and not fully engage with the virtual platform, potentially threatening these important outcomes. To address this, I have committed to blocking off time on my schedule during both conferences to authentically engage with the content and attendees.

How do you define a successful conference? What outcomes do you want to achieve in upcoming conferences that have gone virtual? While you don’t have to make a logic model out of your thoughts, I would challenge you to think evaluatively about upcoming conferences, asking yourself what you hope to achieve and how can you ensure that it happens.

Figure 1. Lyssa’s Logic Model to Achieve a Successful Virtual Conference

Figure 1. Lyssa’s Logic Model to Achieve a Successful Virtual Conference

Blog: An Evaluative Approach to Proposal Development*

Posted on June 27, 2019 by  in Blog - ()

Executive Director, The Evaluation Center at Western Michigan University

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

A student came into my office to ask me a question. Soon after she launched into her query, I stopped her and said I wasn’t the right person to help because she was asking about a statistical method that I wasn’t up-to-date on. She said, “Oh, you’re a qualitative person?” And I answered, “Not really.” She left looking puzzled. The exchange left me pondering the vexing question, “What am I?” (Now imagine these words echoing off my office walls in a spooky voice for a couple of minutes.) After a few uncomfortable moments, I proudly concluded, “I am a critical thinker!”  

Yes, evaluators are trained specialists with an arsenal of tools, strategies, and approaches for data collection, analysis, and reporting. But critical thinking—evaluative thinking—is really what drives good evaluation. In fact, the very definition of critical thinking—“the mental process of actively and skillfully conceptualizing, applying, analyzing, synthesizing, and evaluating information to reach an answer or conclusion”2—describes the evaluation process to a T. Applying your critical, evaluative thinking skills in developing your funding proposal will go a long way toward ensuring your submission is competitive.

Make sure all the pieces of your proposal fit together like a snug puzzle. Your proposal needs both a clear statement of the need for your project and a description of the intended outcomes—make sure these match up. If you struggle with the outcome measurement aspect of your evaluation plan, go back to the rationale for your project. If you can observe a need or problem in your context, you should be able to observe the improvements as well.

Be logical. Develop a logic model to portray how your project will translate its resources into outcomes that address a need in your context. Sometimes simply putting things in a graphic format can reveal shortcomings in a project’s logical foundation (like when important outcomes can’t be tracked back to planned activities). The narrative description of your project’s goals, objectives, deliverables, and activities should match the logic model.

Be skeptical. Project planning and logic model development typically happen from an optimistic point of view. (“If we build it, they will come.”) When creating your work plan, step back from time to time and ask yourself and your colleagues, What obstacles might we face? What could really mess things up? Where are the opportunities for failure? And perhaps most important, ask, Is this really the best solution to the need we’re trying to address? Identify your plan’s weaknesses and build in safeguards against those threats. I’m all for an optimistic outlook, but proposal reviewers won’t be wearing rose-colored glasses when they critique your proposal and compare it with others written by smart people with great ideas, just like you. Be your own worst critic and your proposal will be stronger for it.

Evaluative thinking doesn’t replace specialized training in evaluation. But even the best evaluator and most rigorous evaluation plan cannot compensate for a disheveled, poorly crafted project plan. Give your proposal a competitive edge by applying your critical thinking skills and infusing an evaluative perspective throughout your project description.

* This blog is a reprint of an article from an EvaluATE newsletter published in summer 2015.

2 dictionary.com

Newsletter: An Evaluative Approach to Proposal Development

Posted on July 1, 2015 by  in Newsletter - () ()

Executive Director, The Evaluation Center at Western Michigan University

A student came into my office to ask me a question. Soon after she launched into her query, I stopped her and said I wasn’t the right person to help because she was asking about a statistical method that I wasn’t up-to-date on. She said, “Oh, you’re a qualitative person?” And I answered, “Not really.” She left looking puzzled. The exchange left me pondering the vexing question, “What am I?” (Now imagine these words echoing off my office walls in a spooky voice for a couple of minutes.) After a few uncomfortable moments, I proudly concluded, “I am a critical thinker!”

Yes, evaluators are trained specialists with an arsenal of tools, strategies, and approaches for data collection, analysis, and reporting. But critical thinking—evaluative thinking—is really what drives good evaluation. In fact, the very definition of critical thinking as “the mental process of actively and skillfully conceptualizing, applying, analyzing, synthesizing, and evaluating information to reach an answer or conclusion”1 describes the evaluation process to a T. Applying your critical, evaluative thinking skills in developing your funding proposal will go a long way toward ensuring your submission is competitive.

Make sure all the pieces of your proposal fit together like a snug puzzle. Your proposal needs both a clear statement of the need for your project and a description of the intended outcomes—make sure these match up. If you struggle with the outcome measurement aspect of your evaluation plan, go back to the rationale for your project. If you can observe a need or problem in your context, you should be able to observe the improvements as well. Show linkages between the need you intend to address, your activities and products, and expected outcomes.
Be logical. Develop a logic model to portray how your project will translate its resources into outcomes that address a need in your context. Sometimes simply putting things in a graphic format can reveal shortcomings in a project’s logical foundation (like when important outcomes can’t be tracked back to activities). The narrative description of your project’s goals, objectives, deliverables, and activities should match the logic model.

Be skeptical. Project planning and logic model development typically happen from an optimistic point of view. (“If we build it, they will come.”) While crafting your work plan, step back from time to time and ask yourself and your colleagues, what obstacles might we face? What could really mess things up? Where are the opportunities for failure? And perhaps most importantly, is this really the best solution to the need we’re trying to address? Identify your plan’s weaknesses and build in safeguards against those threats. I’m all for an optimistic outlook, but proposal reviewers won’t be wearing rose-colored glasses when they critique your proposal and compare it with others written by smart people with great ideas, just like you. Be your own worst critic and your proposal will be stronger for it.

Evaluative thinking doesn’t replace specialized training in evaluation. But even the best evaluator and most rigorous evaluation plan cannot compensate for a disheveled, poorly crafted project plan. Give your proposal a competitive edge by applying your critical thinking skills and infusing an evaluative perspective throughout your project description.

1 dictionary.com