Statistical Evidence – Survey and Opinion Polling
Posted by Keren Wang, 2024
This lesson covers using statistical evidence, specially surveys and opinion polls surveys in argumentation.
Surveys, also known as opinion polls, are designed to represent the views of a population by posing a series of questions to a sample group and then extrapolating broader trends or conclusions in quantitative terms. [1] [2] For example, in an election year, polling agencies might survey a diverse group of registered voters to gauge public support for different candidates, often reporting findings in percentages to indicate the overall popularity of each option. These results help forecast likely outcomes and provide insights into public sentiment on key issues. However, like all statistical evidence, opinion poll results are susceptible to inaccuracies and distortions, which can be misused or misinterpreted.
In this lesson, we will examine various types of surveys, their advantages and limitations, and their appropriate applications.
There are two basic types of surveys – descriptive and analytic:
Survey Types
Survey Type | Descriptive Survey | Analytic Survey |
---|---|---|
Purpose | Document and describe the characteristics, behaviors, or attitudes of a specific population at a given time. | Goes beyond description to investigate why certain patterns or behaviors occur. |
Focus | Covers demographic data, behavior frequency, or opinions and preferences. | Tests hypotheses by examining relationships between variables. |
Use in Communication Research | Useful for establishing a snapshot of audience preferences or public opinion. | Used to analyze how communication factors affect audience attitudes and behaviors. |
Example 1 | Survey examining social media platform usage among college students. | Survey studying the relationship between crisis communication and trust in government. |
Example 2 | Survey by a PR firm to gauge public perceptions of a corporate brand. | Survey exploring whether exposure to diverse news sources affects political polarization. |
Advantages and Limitations of Surveys
Advantages
- Anonymity: Anonymous surveys can increase comfort in sharing personal or sensitive information. Respondents may feel more at ease providing honest answers when their identities are not disclosed, leading to more accurate data. [3]
- Broad Reach: Surveys / opinion polls allow researchers to reach large, diverse populations. For instance, online surveys can be distributed globally, allowing for data collection from a wide audience. [4]
- Cost-Effective: Surveys are accessible across industries for large-scale data collection. Online surveys, in particular, are less expensive compared to other methods, such as face-to-face or telephone interviews. [5]
- Quantifiable Data: Surveys allow for measurable insights into the target population. By using structured questions, researchers can collect data that is easy to analyze statistically, facilitating the identification of patterns and trends. When designed and applied correctly, surveys can increase consistency and reduce researcher bias. By administering the same set of questions to all respondents, surveys can bring uniformity in opinion collection and analysis.
Limitations
- Ineffective or Misleading Questions: Survey questions are sometimes written in vague, overly broad, or ambiguous ways that can lead to misinterpretation or inaccurate responses. Closed-ended questions, such as yes/no or rating scale questions, may lack depth. While these types of questions facilitate quantitative analysis, they often fail to capture the full complexity of respondents’ thoughts and feelings. [6]
- Example: A closed-ended question like “Do you support increased sales taxes to improve education and healthcare? YES or NO” can be misleading. It combines two issues—education and healthcare—forcing respondents to address both at once, even if they feel differently about each. It also presents “increased taxes” as the only solution, ignoring other funding options. Imagine if 70% of respondents to this question answered “NO,” a news headline claiming “Poll Shows 70% of Voters Don’t Want to Improve Education & Healthcare” would also be misleading. Many may have chosen “NO” not because they oppose improvements, but because they doubt the effectiveness using higher sales taxes to fund them.
- Low Response Rates: Surveys / opinion polls can lead to unrepresentative samples. A low response rate may result in a sample that does not accurately reflect the target population, potentially skewing the results. [7]
- Example: In the 2016 U.S. presidential election, many opinion polls predicted a victory for Hillary Clinton. However, these polls often had low response rates, leading to unrepresentative samples that underestimated support for Donald Trump. This discrepancy highlighted how low participation can skew survey results. [8]
- Sampling Challenges: Surveys risk sampling bias if the sample lacks diversity. If certain groups are underrepresented, the findings may not be generalizable to the entire population.
- Example: A 1936 Literary Digest poll for U.S. presidential election predicted Alf Landon’s victory over Franklin D. Roosevelt by sampling its readers, car owners, and telephone users—groups not representative of the general population during the Great Depression. This sampling bias led to an incorrect prediction. [9]
- Response Bias: Surveys may not always provide truthful answers. Respondents might give socially desirable responses or may not recall information accurately, leading to biased data. [10] Surveys may also lead to other self-reporting inaccuracies due to memory lapses, misinterpreting questions, or self-censorship. [11]
- Example: Surveys on sensitive topics, such as dietary habits or illicit drug use, often face response bias, with respondents under-reporting due to social desirability bias. This can result in data that grossly misrepresents actual behaviors.
Writing Effective Survey Questions
Clarity and Simplicity
Use straightforward language and avoid technical jargon or complex wording.
Conciseness and Single-Focus
Keep questions short to reduce respondent fatigue and prevent confusion. Ask only one piece of information per question to avoid ambiguity.
Examples:
Neutral Language
Avoid leading questions that imply a preferred answer. Use neutral phrasing to encourage honesty.
Relevance and Sensitivity
Ensure questions are relevant and appropriate for the audience. Avoid overly personal questions unless essential.
Choosing Between Open-Ended and Close-Ended Questions
Researchers tend to use closed-ended questions when they need quantifiable data that is easily comparable, such as demographic details or satisfaction ratings. Open-ended questions are used when in-depth feedback is needed, to explore new topics, or to understand respondents’ reasoning. There are pros and cons to both question types.
Close-Ended Questions
Close-ended questions provide respondents with predefined options, such as multiple-choice, yes/no, or rating scales.
- Ease of Analysis: Quantifiable data is easy to code and analyze. Ensures uniform responses across participants.
- Efficiency: Quick for respondents to answer, which improves response rates.
- Limited Insight: Restricts responses to preset options, missing depth.
- Risk of Bias: Poor answer choices may bias responses. See “Limitations” section above.
Yes/No: “Do you feel our product meets your expectations?”
Likert Scale: “How satisfied are you with our service?” Options: 1. Very Dissatisfied, 2. Dissatisfied, 3. Neutral, 4. Satisfied, 5. Very Satisfied
Open-Ended Questions
These questions allow respondents to answer freely in their own words, offering more detailed insights.
- Nuance: Allows for detailed responses, offering deeper insights.
- Flexibility: Enables responses that may reveal unexpected insights.
- Time-Consuming: Longer response times and more complex analysis.
- Complex Analysis: Open-ended responses need qualitative coding.
Example 2: “Describe your experience with our customer service.”
Example 3: “What motivates you to choose one news source over another?”
*Click here to continue to our lesson on Sampling & Evaluating Survey Accuracy.
❖ Further Reading ❖
- Fink, A. (2009). How to conduct surveys: A step-by-step guide (4th ed.). Thousand Oaks, CA: Sage.
- Fowler Jr, F. J. (2013). Survey research methods. Sage publications.
- Mehrabi, N., et al. (2021). A survey on bias and fairness in machine learning. ACM computing surveys.
- Nardi, P. M. (2018). Doing survey research: A guide to quantitative methods. Routledge.
- Rea, L. M., & Parker, R. A. (2005). Designing and conducting survey research. San Francisco: Jossey-Bass.