Archive: logic models

2016 High Impact Technology Exchange Conference (HI-TEC)

Posted on July 15, 2016 by , in Conferences ()

2016 High Impact Technology Exchange Conference (HI-TEC)
Pittsburgh, PA
July 25-28, 2016

Workshop

Logic Models: The Swiss Army Knife of Project Planning and Evaluation
Kelly Robertson
Lyssa Wilson

July 27, 2016 | 3:45-4:30 p.m.

A logic model is a graphic depiction of how a project translates its resources and activities into outcomes. Logic models are useful tools for succinctly communicating a project’s goals and activities, but they have many other applications. They provide a foundation for a project evaluation plan (and subsequent reporting) and can be used to organize the content of a grant proposal.  In this session, participants will learn the basics of how to create a logic model and we will demonstrate its use for planning a project evaluation and organizing a grant proposal.  Participants will receive the Evaluation Planning Checklist for ATE Proposals and ATE Project Logic Model Template.

Participants will receive the Evaluation Planning Checklist for ATE Proposals and ATE Project Logic Model Template.

For more information about the conference, and for conference registration, please visit http://www.highimpact-tec.org/

Resources:
Slides
Handout

Webinar: Logic Models: Getting Them Right and Using Them Well

Posted on July 13, 2016 by , in Webinars

Presenter(s): Lori Wingate, Miranda Lee
Date(s): August 17, 2016
Time: 1-2:00 p.m.
Recording: https://www.youtube.com/watch?v=z4PY1KH9R0w

A logic model is a succinct graphic depiction of how a project translates its resources and activities into outcomes. A good logic model efficiently communicates the overall logic and purpose of a project and serves as a foundation for evaluation planning. A logic model can be a powerful addition to your funding proposal, but only if it is well aligned to your project’s narrative description. In this session, participants will learn (a) how to create a project logic model, while avoiding common mistakes (like confusing outcomes with activities or outputs and providing too many or too few details), and (b) how to effectively integrate a logic model into a funding proposal.

Resources:
Slides
Handout
Logic Model Template

Blog: Articulating Intended Outcomes Using Logic Models: The Roles Evaluators Play

Posted on July 6, 2016 by , in Blog ()
Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Wilkerson Peery
Stephanie B. Wilkerson Elizabeth Peery

Articulating project outcomes is easier said than done. A well-articulated outcome is one that is feasible to achieve within the project period, measurable, appropriate for the phase of project development, and in alignment with the project’s theory of change. A project’s theory of change represents causal relationships – IF we do these activities, THEN these intended outcomes will result. Understandably, project staff often frame outcomes as what they intend to do, develop, or provide, rather than what will happen as a result of those project activities. Using logic models to situate intended outcomes within a project’s theory of change helps to illustrate how project activities will result in intended outcomes.

Since 2008, my team and I have served as the external evaluator for two ATE project cycles with the same client. As the project has evolved over time, so too have its intended outcomes. Our experience using logic models for program planning and evaluation has illuminated four critical roles we as evaluators have played in partnership with project staff:

  1. Educator. Once funded, we spent time educating the project partners on the purpose and development of a theory of change and intended outcomes using logic models. In this role, our goal was to build understanding of and buy-in for the need to have logic models with well-articulated outcomes to guide project implementation.
  1. Facilitator. Next, we facilitated the development of an overarching project logic model with project partners. The process of defining the project’s theory of change and intended outcomes was important in creating a shared agreement and vision for project implementation and evaluation. Even if the team includes a logic model in the proposal, refining it during project launch is still an important process for engaging project partners. We then collaborated with individual project partners to build a “family” of logic models to capture the unique and complementary contributions of each partner while ensuring that the work of all partners was aligned with the project’s intended outcomes. We repeated this process during the second project cycle.
  1. Methodologist. The family of logic models became the key source for refining the evaluation questions and developing data collection methods that aligned with intended outcomes. The logic model thus became an organizing framework for the evaluation. Therefore, the data collection instruments, analyses, and reporting yielded relevant evaluation information related to intended outcomes.
  1. Critical Friend. As evaluators, our role as a critical friend is to make evidence-based recommendations for improving project activities to achieve intended outcomes. Sometimes evaluation findings don’t support the project’s theory of change, and as critical friends, we play an important role in challenging project staff to identify any assumptions they might have made about project activities leading to intended outcomes. This process helped to inform the development of tenable and appropriate outcomes for the next funding cycle.

Resources:

There are several resources for articulating outcomes using logic models. Some of the most widely known include the following:

Worksheet: Logic Model Template for ATE Projects & Centers: http://www.evalu-ate.org/resources/lm-template/

Education Logic Model (ELM) Application Tool for Developing Logic Models: http://relpacific.mcrel.org/resources/elm-app/

University of Wisconsin-Extension’s Logic Model Resources: http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

W.K. Kellogg Foundation Logic Model Development Guide: https://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide

Newsletter: What’s the Difference Between Outputs, Outcomes, and Impacts?

Posted on July 1, 2016 by  in Newsletter - () ()

Director of Research, The Evaluation Center at Western Michigan University

LMGraphic

A common source of confusion among individuals who are learning about logic models is the difference between outputs, outcomes, and impacts. While most people generally understand that project activities are the things that a project does, the other terms may be less straightforward.

Outputs are the tangible products of project activities. I think of outputs as things whose existence can be observed directly, such as websites, videos, curricula, labs, tools, software, training materials, journal articles, and books. They tend to be the things that remain after a project ends or goes away.

Outcomes are the changes brought about through project activities and outputs/products.  Outcomes may include changes in individual knowledge, skills, attitudes, awareness, or behaviors; organizational practices; and broader social/economic conditions.  In her blog post “Outputs are for programs, outcomes are for people” (http://bit.ly/srob0314), Sheila Robinson offers this guidance: “OUTCOMES are changes in program participants or recipients (aka the target population). They can be identified by answering the question:  How will program participants change as a result of their participation in the program?” This is a great way to check to see if your logic model elements are located in the right place.  If the outcomes in your logic model include things that don’t sound like an appropriate answer to that question, then you may need to move things around.

The term impact is usually used to refer to outcomes that are especially large in scope or the ultimate outcomes a project is seeking to bring about. Sometimes the terms impacts and long-term outcomes are used interchangeably.

For example, one of EvaluATE’s main activities are webinars. Outputs of these webinars include resource materials, presentation slides, and recordings. Short-term outcomes for webinar participants are expected to include increased knowledge of evaluation. Mid-term outcomes include modifications or changes in their evaluation practice. Long-term outcomes are improved quality and utility of ATE project evaluations. The ultimate intended impact is for ATE projects to achieve better outcomes through strategic use of high-quality evaluations.

Keep in mind that not all logic models use these specific terms, and not everyone adheres to these particular definitions. That’s OK! The important thing to remember when developing a logic model is to understand what YOU mean in using these terms and to use and apply them consistently in your model and elsewhere.  And regardless of how you define them, each column in your model should present new information, not a reiteration of something already communicated.

Newsletter: Project Spotlight: Geospatial Technician Education – Unmanned Aircraft Systems & Expanding Geospatial Technician Education through Virginia’s Community Colleges

Posted on July 1, 2016 by  in Newsletter - () ()

Deputy Director, Virginia Space Grant Consortium

Chris Carter is the Deputy Director of the Virginia Space Grant Consortium, where he leads two ATE projects.

How do you use logic models in your ATE projects?

Our team recently received our fourth ATE award, which will support the development of academic pathways and faculty training in unmanned aircraft systems (UAS). UAS, when combined with geospatial technologies, will revolutionize spatial data collection and analysis.

Visualizing desired impacts and outcomes is an important first step to effective project management. Logic models are wonderful tools for creating a roadmap of key project components. As a principal investigator on two ATE projects, I have used logic models to conceptualize project outcomes and the change that our team desires to create. Logic models are also effective tools for articulating the inputs and resources that are leveraged to offer the activities that bring about this change.

With facilitation and guidance from our partner and external evaluator, our team developed several project logic models. We developed one overarching project logic model to conceptualize the intended outcomes and desired change of the regional project. Each community college partner also developed a logic model to capture its unique goals and theory of change while also articulating how it contributes to the larger effort. These complementary logic models allowed the team members to visualize and understand their contributions while ensuring everyone was on the same path.

Faculty partners used these logic models to inform their administrations, business partners, and employers about their work. They are great tools for sharing the vision of change and building consensus among key stakeholders.

Our ATE projects are focused on creating career pathways and building faculty competencies to prepare technicians. The geospatial and UAS workforce is a very dynamic employment sector that is constantly evolving. We find logic models helpful tools for keeping the team and partners focused on the desired outputs and outcomes. The models remind us of our goals and help us understand how the components fit together. It is crucial to identify the project inputs and understand that as these evolve, project activities also need to evolve. Constantly updating a logic model and understanding the relationships between the various sections are key pieces of project management.

I encourage all ATE project leaders to work closely with their project evaluators and integrate logic models. Our external evaluator was instrumental in influencing our team to adopt these models. Project evaluators must be viewed as team members and partners from the beginning. I cannot imagine effectively managing a project without the aid of this project blueprint.

Blog: Logic Models and Evaluation Planning – Working Together!

Posted on January 20, 2016 by  in Blog ()

Research Scientist, Education Development Center

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

As an evaluator, I am often asked to work on evaluation plans for National Science Foundation proposals. I believe it is important for evaluators and clients to work together, so I start the conversation by asking about project goals and outcomes and then suggest that we work together to develop a project logic model. Developing a logic model helps create a unified vision for the project and promotes common understanding. There are many types and formats of logic models and while no model is “best,” a logic model usually has the following key elements: inputs, activities, outputs, outcomes, and impacts.

    Reasons to develop logic models:

  • A logic model is a visual “elevator speech” that can be helpful when reviewing the proposal as it provides a quick overview of the project.
  • It is logical! It aligns the resources, activities, deliverables (outputs), and outcomes (short, and medium) with impacts (long-term outcomes). I have often been told that it has helped my clients organize their proposals.

Focus: I love logic models because they help me, the evaluator, focus my work on critical program elements. When a logic model is developed collaboratively by the project team (client) and the evaluator, there is a shared understanding of how the project will work and what it is designed to achieve.

Frame the evaluation plan: Now comes the bonus! A logic model helps form the basis of an outcomes-based evaluation plan. I start the plan by developing indicators with my client for each of the outcomes on the logic model. Indicators are the criteria used for measuring the extent to which projected outcomes are being achieved. Effective indicators align directly to outcomes and are clear and measurable. And while measurable, indicators do not always need to be quantifiable. They can be qualitative and descriptive such as “Youth will describe that they ….” Note that in this example, it is stated how you will determine whether an outcome has been met (youth state that… self-report). It is likely you will have more than one indicator for each outcome. An indicator answers questions like these: How will you know it when you see it? What does it look like when an outcome is met? What is the evidence?

Guide the evaluation questions: After the indicators are developed we decide on the guiding evaluation questions (what we will be evaluating), and I get to work on the rest of the evaluation plan. I figure out an overall design and then add methods, measures, sampling, analysis, reporting, and dissemination (potential topics for future blog posts). Once the project is funded, we refine the evaluation plan, develop a project/evaluation timeline, and determine the ongoing evaluation management and communication – then we are ready for action.

Resources:
1. W.K. Kellogg Foundation Logic Model Development Guide
2. W.K. Kellogg Foundation Evaluation Handbook (also available in Spanish)
3. EvaluATE’s Logic Model Template for ATE Projects and Centers

Blog: Figures at Your Fingertips

Posted on October 28, 2014 by  in Blog ()

Co-Principal Investigator, Op-Tec Center

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

In formative evaluation, programs or projects are typically assessed during their development or early implementation to provide information about how best to revise and modify for improvement. (www.austinisd.org)

 I’m Gordon Snyder and I’m currently the principal investigator of the National Center for Information and Communications Technologies (ICT Center). My experience as new ATE center PI back in July of 2001 did not get off to a very smooth start. With the retirement of the founding PI after three years, I was moving from a co-PI position and faced with making a number of decisions and changes in a short period of time.  We were right in the middle of the “dot com bust” and the information and communications technology field was in free fall. I knew our decisions needed to be data-driven, smart, focused, quick, and correct if we were going to continue to be a resource for faculty and students.

As a center co-PI during the boom times between 1998 and 2000, my role was focused on curriculum development and helping faculty learn and teach new technology in their classrooms and labs. I honestly did not understand nor pay much attention to the work our evaluator was doing – that was something the PI liked to handle, and I was perfectly fine with that.

In my new role as a PI, things changed. One of the first things I did was read the evaluation reports for the past two years. I found a lot of flowery complimentary language with little else in those reports – I recall using the term “pile of fluff” along with a few others that I won’t repeat here. I found nothing substantial that was going to help me making any decisions.

In August of 2001, I received our year 3 annual evaluation report and this one was even more “fluffy.” Lesson learned: Within a month I dismissed that evaluator, replacing that individual with someone more in tune with what we needed. Things were much better with the new evaluator, but I still found it difficult making intelligent data-based decisions.  I did not have the information I needed. There had to be a better way.

Fast forward to today: ATE PIs need even more access to valid, reliable, useful, evaluative data for decision making. This data needs to be available in real time, or close to real time throughout the funding cycle, more frequently than the typical annual evaluation reports. However, most PIs still simply do not have the time, resources, and expertise required to systematize the collection and use of this kind of information.

Logic models are one method that’s catching on to keep track of and use information to make formative data-driven decisions. I’ve been working on the FAS4ATE project (Formative Assessment for ATE) with Western Michigan University that will ultimately develop some logic model-based online tools to streamline data collection and more effectively scope and plan evaluation activities to include formative and summative processes. We’re early in the development process. Here’s a short video demonstrating our prototype of one of the tools.

Logic models are a great way to keep up with project work and more quickly and confidently make data-based decisions. If you’d like to learn more about this formative assessment project, contact me at gordonfsnyder@gmail.com