Jane Ostrander

Director, Experiential Learning Center, Truckee Meadows Community College

Dr. Jane Ostrander is Principal Investigator for the National Science Foundation Advanced Technological Education (NSF ATE) project Destination: Problem-Based Learning (PBL) (DUE#1161352), and Director of the Experiential Learning Center at Truckee Meadows Community College, Reno, NV. Ostrander serves on the Community College Liaison Panel for the ATE EvaluATE Center and as a Mentor for the Mentor-Connect for Leadership Development and Outreach Project. Her research interests include PBL, faculty professional development, online knowledge sharing in communities of practice, and social psychological interventions for transformative change. Prior to her project work Ostrander taught computer literacy, web site design, project management, and business.


Blog: Mistakes Made and Lessons Learned, Part I – Working with Your Evaluator

Posted on March 18, 2015 by  in Blog ()

Director, Experiential Learning Center, Truckee Meadows Community College

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

When I assumed the PI-ship of the Scenario-Based Learning Project in 2006, I had worked closely with the prior PI as the project’s instructional designer, knew I enjoyed the ATE community, shared their vision of an innovative 21st century technician workforce, and had management and teaching experience. What more was there to know? A lot, as it turned out. It quickly became apparent to me that the world of ATE projects and centers was a different place than any I had worked in before.

When I took over as PI on the second grant proposal, the former PI suggested we use the evaluators from a previous non-ATE grant she had led. Big mistake. Those evaluators reported to my National Visiting Committee (NVC) during our initial committee meeting that they didn’t have any results to report because they did not plan to collect data until the end of the project year. My NVC was not happy. I was not happy. My stakeholders were not happy.

How did this happen? In my naivety, I didn’t even discuss the evaluation with the evaluators beyond an initial outline of a plan involving questions to be answered by the evaluation—I thought the evaluators knew what they were doing because they were evaluators. I didn’t understand the complex nature of the profession of evaluation. Since then I have joined the American Evaluation Association, attended their annual conference, and regularly attend EvaluATE’s webinars. I made a mistake, learned from it, and the project improved.

I quickly learned that some evaluators and funders are all about the summative report. The project said they would do A, here is the data to show they did or did not do A. End of report. In contrast, the ATE program is interested in how we are doing as we progress through our work. Formative reports from the evaluator serve as a check-in on where you are in your work plan and outcomes. Your evaluator needs to be a critical friend—an advisor who keeps a distance and is critical where needed yet still supportive with ideas, solutions, and contacts.

Choose an evaluator with ATE experience and expertise in collecting and analyzing the kind of data you will need. Confirm who will collect the data, how it will be collected, from whom, and when early in your discussions with your evaluator and project team. Confirm that your evaluator is willing to provide mentoring to your data collection team as needed if you decide to collect the data yourselves and have the evaluator do the analysis (saves money but requires time).

You might need interim reports/summaries from your evaluator for meetings with stakeholders, your NVC, advisory boards, the ATE annual survey, and your annual and final reports. It is a good idea to align your data collection with your reporting needs to best use your resources.

Learn more about the process of evaluation every chance you get. Choose an evaluator with the expertise appropriate to your project or center. Think of your evaluation as a resource and your evaluator as an ally to help you and your team to create the best project or center possible.

Newsletter: Meet EvaluATE’s Community College Liaison Panel

Posted on January 1, 2014 by , , , in Newsletter - ()

The ATE program is community college-based, and as such EvaluATE places a priority on meeting the needs of this constituency. To help ensure the relevancy and utility of its resources, EvaluATE has convened a Community College Liaison Panel (CCLP). CCLP members Michael Lesiecki, Marilyn Barger, Jane Ostrander, and Gordon Snyder are tasked with keeping the EvaluATE team tuned into the needs and concerns of 2-year college stakeholders and engaging the ATE community in the review and pilot testing of EvaluATE-produced materials.

These resources distill relevant elements of evaluation theory, principles, and best practices so that a user can quickly understand and apply them for a specific evaluation-related task. They are intended to support members of the ATE community to enhance the quality of their evaluations.

The CCLP’s role is to coordinate a three-phase review process. CCLP members conduct a first-level review of an EvaluATE resource. The EvaluATE team revises it based on the CCLP’s feedback, then each of the four CCLP members reaches out to diverse members of the ATE community—PIs, grant developers, evaluators, and others—to review the material and provide confidential, structured feedback and suggestions. After another round of revisions, the CCLP engages another set of ATE stakeholders to actually try out the resource to ensure it “works” as intended in the real world. Following this pilot testing, EvaluATE finalizes the resource for wide dissemination.

The CCLP has shepherded two resources through the entire review process: the ATE Evaluation Primer and ATE Evaluation Planning Checklist. In the hopper for review in the next few months are the ATE Logic Model Template and Evaluation Planning Matrix, Evaluation Questions Checklist, ATE Evaluation Reporting Checklist, and Professional Development Feedback Survey Template. In addition, CCLP members are leading the development of a Guide to ATE Evaluation Management—by PIs for PIs.

The CCLP invites anyone interested in ATE evaluation to participate in the review process. For a few hours of your time, you’ll get a first look at and tryout of new resources. And your inputs will help shape and strengthen the ATE evaluation community. We also welcome recommendations of tools and materials that others have developed that would be of interest to the ATE community.

To get involved, email CCLP Director Mike Lesiecki at mlesiecki@gmail.com. Tell him you would like to help make EvaluATE be the go-to evaluation resource for people like yourself.

Webinar: Ready, Set, Evaluate!

Posted on September 21, 2011 by , , , , , in Webinars ()

Presenter(s): Eileen Lewis, Jane Ostrander, Jason Burkhardt, John Kmiec, Lori Wingate, Stephanie Evergreen
Date(s): September 21, 2011
Time: 1:00 PM
Recording: https://vimeo.com/29749424

The focus of this webinar is on helping new ATE grantees and evaluators understand the role of evaluation in ATE grants, particularly on administrative/managerial aspects of evaluation. At the start of a new grant, it’s important for grant staff and evaluators to be clear about their responsibilities, deliverables, timelines, procedures, and information needs with respect to evaluation. This webinar will be a live Q&A session with an panel of experts who have extensive experience with and diverse perspectives on ATE evaluation, including Eileen Lewis (University of California – Berkeley, Cañada College, and former co-Lead of the ATE program), Jane Ostrander (PI, Problem-Based Learning), John Kmiec (Evaluator, Pearl River Community College ATE grant), and EvaluATE staff. These ATE veterans will answer participants’ questions and share tips and tools to facilitate the implementation and management of ATE project and center evaluations.

Further Reading:

Analyzing an Evaluation Plan (Eileen Lewis)
Possible Evaluation Components (Eileen Lewis)
EDTTS (ATE project) Evaluation Report (John Kmiec/Scott Alsobrooks)

Resources:
Handout PDF
Teacher Professional Development Email Survey (J Ostrander)
Evolution of an Evaluation Plan (J Ostrander)
Classroom Observation Tool (P Saflund/J Ostrander)