Archive: research

Blog: Evaluator, Researcher, Both?

Posted on June 21, 2017 by  in Blog ()

Professor, College of William & Mary

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Having served as a project evaluator and as a project researcher, it is apparent to me how critical it is to have conversations about roles at the onset of funded projects.  Early and open conversations can help avoid confusion, help eliminate missed timing to collect critical data, and highlight where differences exist for each project team role. The blurring of lines over time regarding strict differences between evaluator and researcher requires project teams, evaluators, and researchers to create new definitions for project roles, to understand scope of responsibility for each role, and to build data systems that allow for sharing information across roles.

Evaluation serves a central role in funded research projects. The lines between the role of the evaluator and that of the researcher can blur, however, because many researchers also conduct evaluations. Scriven (2003/2004) saw the role of evaluation as a means to determine “the merit, worth, or value of things” (para. #1), whereas social science research instead is “restricted to empirical (rather than evaluative) research, and bases its conclusion only on factual results—that is, observed, measured, or calculated data” (para. #2).  Consider too, how Powell (2006) posited “Evaluation research can be defined as a type of study that uses standard social research methods for evaluative purposes” (p. 102).  It is easy to see how confusion arises.

Taking a step back can shed light on the differences in these roles and ways they are now being redefined. The role of researcher shows a different project perspective, as a goal of research is the production of knowledge, whereas the role of the external evaluator is to provide an “independent” assessment of the project and its outcomes. Typically, an evaluator is seen as a judge of a project’s merits, which assumes a perspective that a “right” outcome exists. Yet inherent in the role of evaluation are the values held by the evaluator, the project team, and the stakeholders as context influences the process and who makes decisions on where to focus attention, why, and how feedback is used (Skolits, Morrow, & Burr, 2009).  Knowing more about how the project team intends to use evaluation results to help improve project outcomes requires a shared understanding of the role of the evaluator (Langfeldt & Kyvik, 2011).

Evaluators seek to understand what information is important to collect and review and how to best use the findings to relate outcomes to stakeholders (Levin-Rozalis, 2003).  Researchers instead focus on diving deep into investigating a particular issue or topic with a goal of producing new ways of understanding in these areas. In a perfect world, the roles of evaluators and researchers are distinct and separate. But, given requirements for funded projects to produce outcomes that inform the field, new knowledge is also discovered by evaluators. The swirl of roles results in evaluators publishing results of projects that informs the field, researchers leveraging their evaluator roles to publish scholarly work, and both evaluators and researchers borrowing strategies from each other to conduct their work.

The blurring of roles requires project leaders to provide clarity about evaluator and researcher team functions. The following questions can help in this process:

  • How will the evaluator and researcher share data?
  • What are the expectations for publication from the project?
  • What kinds of formative evaluation might occur that ultimately changes the project trajectory? How do these changes influence the research portion of the project?
  • How does shared meaning of terms, role, scope of work, and authority for the project team occur?

Knowing how the evaluator and researcher will work together provides an opportunity to leverage expertise in ways that move beyond the simple additive effect of both roles.  Opportunities to share information is only possible when roles are coordinated, which requires advanced planning. It is important to move beyond siloed roles and towards more collaborative models of evaluation and research within projects. Collaboration requires more time and attention to sharing information and defining roles, but the time spent on coordinating these joint efforts is worth it given the contributions to both the project and to the field.


References

Levin-Rozalis, M. (2003). Evaluation and research: Differences and similarities. The Canadian Journal of Program Evaluation, 18(2):1-31.

Powell, R. R. (2006).  Evaluation research:  An overview.  Library Trends, 55(1), 102-120.

Scriven, M. (2003/2004).  Michael Scriven on the differences between evaluation and social science research.  The Evaluation Exchange, 9(4).

Newsletter: Survey Says Winter 2016

Posted on January 1, 2016 by , in Newsletter - ()

On the 2015 ATE survey, 65 of 230 principal investigators (28%) reported spending some portion of their annual budgets on research. Six of these projects were funded as targeted research. Among the other 59 projects, expenditures on research ranged from 1% to 65% with a median of 14%. With just six targeted research projects and less than a third of all ATE grantees engaging in research, there is immense opportunity within the ATE program to expand research on technician education.

 

Survey-Says

The full report of 2015 ATE survey findings, along with data snapshots and downloadable graphics, is available from www.evalu-ate.org/annual_survey/.

Blog: Building Effective Partnerships to Conduct Targeted Research on Student Pathways

Posted on March 4, 2015 by  in Blog ()

Associate Professor of Sociology, University of South Florida

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

My name is Will Tyson, associate professor of sociology at the University of South Florida. I am also principal investigator of PathTech (“Successful Academic and Employment Pathways in Advanced Technologies” [NSF #1104214]), an NSF ATE targeted research project aimed at better understanding pathways into technician education and into the workforce. In this post, I describe effective models through which ATE projects and centers can develop targeted research partnerships with STEM education researchers.

Personnel from 2-year and 4-year institutions bring different expertise to the table, but there is great potential for mutually beneficial partnerships built around the desire to learn more about student pathways and student outcomes. Within ATE, centers and projects are typically led by educators and practitioners with expertise in program development, curricular development, and professional development within their areas of technical expertise and technician education. Targeted research in technician education projects are led by STEM education researchers with backgrounds in social science and education interested in learning more about student pathways and outcomes while placing their experiences in a broader social context. What we do is very different, but our goals are the same.

When I discuss my research with ATE grantees and other stakeholders in K-12 education, community colleges, and local industry I get the same revealing responses: “NSF always wants to know about student outcomes, but we don’t really know how to do the research” and “We didn’t know there were people like you out there who did this research.” On the other hand, experienced NSF grantees who conduct research in K-12 education and/or four-year universities often know little about the “T” in STEM in community colleges and work being done through ATE Centers and Projects. Developing ways to bridge knowledge gaps between practitioners and researchers is necessary to increase our understanding of the processes of technician education.

PathTech is a partnership between social science and education researchers at the University of South Florida and the Florida Advanced Technological Education Center (FLATE), an NSF-ATE regional center of excellence. Such a partnership is both mandated by the ATE program solicitation and necessary to conduct high-impact research that can effectively be put into practice. This collaboration is an essential element of the PathTech research model, along with the proactive and enthusiastic participation of our community college, high school, and industry partners.

Through this multifaceted, interdisciplinary collaboration, we have been able to create a regional scale model that allows for the organic development of research objectives driven by the experiences and needs of college personnel as well as theory and scholarship. This is the foundation whereby knowledge is constructed and produced through interface and interaction with those experiencing technician educational and occupational pathways as administrators, teachers, students, employers, and policymakers. Most importantly, this collaboration also allows us to develop a mechanism for real-time dissemination of emerging findings and developing knowledge, thus allowing all parties to benefit from the research.