How we designed a survey that gives Americans a voice

Posted in Design by Jenny

The American Voice is a large-scale survey that gathers feedback from the American people and uses an all-American human-powered algorithm to analyze the results. We conducted surveys with a nationwide sample of 1,045 adults ages 18 and older, from August 21 to September 6, 2011. We used the Saygent platform to ask each respondent pre-recorded, open-ended questions. We used our human-powered algorithm to automatically analyze the opinions and emotions in each response.

What’s this all about?
How do Americans really feel about their current job situation? We know more than 9% of the population is unemployed, but we don’t know the stories behind the statistics. Do people feel angry, worried, or hopeful, and how do people’s feelings correlate with whether they approve or disapprove of the President?

These are questions we want to allow you to explore using our interactive infographic The American Voice: Jobs Edition. The infographic shows the results of our nationwide survey of 1,045 Americans. For the first time, politicians and voters can not only see how the nation feels about President Obama and unemployment, they can also listen to each individual response.

The American Voice focuses on a hot topic and showcases our ability at Saygent to design surveys that encourage people to give in-depth responses. In this post, I’ll briefly share the process we went through coming up with these questions, as well as the methodology behind our survey.

Designing the survey
A great infographic needs good data, and to get the data, we needed to design a rock-solid survey. This was done using an iterative process. First, we used team brainstorming activities to come up with a large pool of hundreds of possible questions. I then edited this batch down to just over 40 possible questions that I tested out with people. Some of the questions we didn’t end up using included “What would your dream job look like?”, “What advice would you give to someone looking for a job?”, and even “What super hero would you hire to fix unemployment in the U.S.?” After a few weeks of testing, we narrowed our initial set down to the final four questions that we felt resulted in the most insightful, open, and thoughtful responses.

We took precautions to protect respondents’ privacy while at the same time encouraging people to give honest answers. Although we collected demographic data like race, age, and voting preferences, we didn’t include any information with people’s answers that could be used to identify them personally. We told survey-takers that their answers would be recorded and published on a public website, but we also reassured them that their answers were anonymous. We believe that this process helped encourage respondents to speak freely.

The Methodology
After finalizing the questions, we began rolling out the survey. We conducted surveys with a nationwide sample of 1,045 adults ages 18 and older, from August 21 to September 6, 2011. We used the Saygent platform to ask each respondent pre-recorded, open-ended questions. Each survey response was automatically analyzed for opinions and emotion using our human-powered algorithm. We chose to focus on four emotions in particular: whether people feel angry, neutral, worried, or happy.

One question that is frequently asked about surveys is whether it is representative of the United States as a nation. With our survey, our respondents are younger in general and have a higher percentage of women than the United States as a whole, when compared to the results of the 2010 U.S. Census. When asked what political party they plan to vote for in 2012, our participants were distributed nearly equally between planning to vote Democrat, Republican, and being undecided.

Summing up
So what did we learn from creating and running this nationwide survey? Three things in particular stand out:

• - Crafting open-ended questions that will get in-depth, honest answers takes time and several rounds of testing and iterations.
• - When survey respondents feel comfortable and know that their privacy is protected, they share personal and emotional answers. We were amazed at the level of trust and openness that our participants displayed and we feel that the answers we received were heartfelt.
• - Finally, Americans have many nuanced opinions about President Obama, but these often stem out of their own personal experiences with unemployment. We hope that our infographic will help you understand the what and the why behind people’s opinions, by letting you listen to each individual voice.