Archive: EvaluATE

Blog: A Rose Isn’t as Sweet by Any Other Name: Lessons on Subject Lines for Web Surveys

Posted on February 25, 2015 by  in Blog ()

Principal Consultant, The Rucks Group

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Survey developers typically spend a great deal of time on the content of questionnaires. We struggle with what items to include, how to ask the question, whether an item should be closed-ended or open-ended; the list of considerations goes on.  After all that effort, we generally spend less time on a smaller aspect that is incredibly important to web surveys: the subject line.

I have come to appreciate the extent to which the subject line acts as a “frame” for a survey. In simplistic terms, a frame is how a concept is categorized. Framing is the difference between calling an unwanted situation a challenge versus a problem. There is a significant literature that suggests that the nature of a frame will produce particular types of behaviors. For instance, my firm recently disseminated a questionnaire to gain feedback on the services that EvaluATE provides. As shown in the chart below, initially we received about 100 responses. With that questionnaire invitation, we used the subject line EvaluATE Services Survey. Based on past experience, we would have expected the next dissemination to garner about 50 responses, but we got closer to 90. So what happened? We had started playing with the subject line.

Rucks_Chart1

 

EvaluATE’s Director, Lori Wingate, sent out a reminder email with the subject line, What do you think of EvaluATE? When we sent out the actual questionnaire, we used the subject line, Tell us what you think. For the next two iterations of dissemination, we had slightly higher than expected response rates.

For the third dissemination, Lori conducted an experiment. She sent out reminder notices but manipulated the subject lines. There were seven different subject lines in total, each sent to about 100 different individuals. The actual questionnaire disseminated had a constant subject line of Would you share your thoughts today? As you see below, the greatest response rate occurred when the subject line of the reminder was How is EvaluATE doing?, while the lowest response rate was when Just a few days was used.

Rucks_Chart2

 

These results aren’t completely surprising. In the 2012 presidential election, the Obama campaign devoted much effort to identifying subject lines that produced the highest response rates. They found that a “gap in information” was the most effective. Using this explanation, the question may emerge as to why the subject line Just a few days would garner the lowest response rate, because it presents a gap in information. The reason this occurred is unclear. One possibility is that incongruity between the sense of urgency implied by the subject line and the importance of the topic of the email to respondents made them feel tricked and they opted not to complete the survey.

Taking all of these findings together tells us that a “rose by any other name would not smell as sweet” and that what something is called does make a difference. So when you are designing your next web survey, make sure crafting the subject line is part of the design process.

Blog: Reflections on the 2014 ATE PI Conference

Posted on November 6, 2014 by , , in Blog
Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Well, the 2014 ATE PI conference has come and gone. First, let us say thank you to AACC for again hosting a wonderful conference. It was great to meet and share ideas with such an amazing group of professionals. We truly look forward to this conference every year. This blog entry contains the EvaluATE team’s reflections on our personal highlights from this year’s conference and hopes for next year.

Jason

Robots! Guitars! Technology in action! My biggest highlight was getting to see the variety of work that is being done within the ATE program. It can be easy in our day-to-day work to forget how important the work of the ATE program is, but we are truly at the forefront of technological education in the United States. I also enjoy getting to see people that I know from within the program. I hope that next year continues to see even more new amazing tech bits!

Lori

For me, the highlight of the conference was when ATE Program Co-Lead David Campbell quoted EvaluATE’s Emma Perk to a room of 100+ people: “The most important purpose of evaluation is not to prove, but to improve.” She shared this quotation from Daniel Stufflebeam in her portion of the Getting Started workshop the day before. (View Emma and Jason’s Getting Started slides). At next year’s conference, I hope there will be more presentations in the research and evaluation conference track. Participants in the preconference workshop on evaluation appreciated hearing about real-world evaluations and practical tips from seasoned ATE evaluators.  We need more of this at every ATE PI conference! (Check out the workshop slides by Candiya Mann, Amy Nisselle, and Bruce Nash).

Emma

This year was my first time attending the ATE PI conference. The showcase sessions were the highlight of the conference for me. I really enjoyed interacting with the different PIs and staff from all the projects and centers. It was great to learn more about the ATE community and how we can expand on what we offer to them as a resource center. EvaluATE’s showcase booth was situated between ATE Central and Mentor-Connect, so we were able to reinforce our great relationship with them and refer people to their useful resources. My hope for next year is to do an evaluation session or roundtable, focusing on identifying the needs of the ATE community.

Corey

Unfortunately I was unable to be at the ATE conference this year. I missed the opportunity to put faces to names. As the annual survey coordinator, I communicate with many of you over the course of the year, so it’s nice to meet some of you face-to-face at the conference. I enjoy being able to talk in person with individuals about the ATE annual survey, to hear concerns, listen to suggestions, and talk data. If you didn’t see the latest reports based on the 2014 survey—like our data snapshots on the representation of women and underrepresented minorities in ATE—check them out here: http://www.evalu-ate.org/annual_survey/

 

We look forward to seeing you all at the conference next year! For more highlights from this year’s conference, including pictures please visit our