4.6 Histograms

The researchers later discovered the Wave 1 data was suspiciously correlated with an existing survey: 2012 CCAP.

  • They believe the researcher likely used CCAP for Wave 1 - used survey responses from real humans that took a real survey – but not the humans that the researcher claimed to interview in the experiment.
  • Then the researcher generated the Wave 2 data by adding random noise to the Wave 1 data
  • Part of why they believe this has to do with a histogram plot they generated to compare Waves 1 and 2

A histogram is a useful plot for summarizing the distribution of a single variable.

  • It shows the frequency of observations (e.g., the number of survey respondents) who give an answer within a particular interval of numeric values

Because a histogram is a single variable summary, we just supply R with the numeric variable we want to summarize.

  • The new argument here breaks tells R how many of the individual rectangles we want. You can play around with that number to see how the plot changes.
hist(x=controlonly$therm1, breaks=50,
     main = "W1 Histogram", ylim = c(0,1000))

hist(x=controlonly$therm2,breaks=50,
     main = "W2 Histogram", ylim = c(0,1000))

The researchers noticed that the heaping patterns were different between Wave 1 and Wave 2.

  • When real humans answer these types of feeling thermometer questions, we often see heaping (tall spikes) at values of 0, 50, and 100. Humans tend to gravitate toward those nice round numbers to anchor their responses. In addition, often researchers might recode people with missing responses (people who skip a question), as having a score of 50, increasing the number at that point.
    • Wave 1 has a lot of this heaping– look at the higher bars around 0, 50, and 100, suggesting a lot of survey respondents gave those answers.
    • However, Wave 2 has less heaping, particularly at 50. This suggested to the researchers that the Wave 2 data were likely generated by a computer and not real humans

4.6.1 Happy research ending

While the original article was retracted

  • Researchers who found irregularities received funding to conduct similar studies with real data this time
  • Multiple publications suggest the canvassing approach was effective:
    • Broockman and Kalla. 2016. “Durably reducing transphobia: A field experiment on door-to-door canvassing” Science 352 no. 6282.
    • Broockman and Kalla. 2020. “Reducing exclusionary attitudes through interpersonal conversation: evidence from three field experiments.” American Political Science Review
    • Kalla and Broockman. 2021. “Which narrative strategies durably reduce prejudice? Evidence from field and survey experiments supporting the efficacy of perspective-getting.” American Journal of Political Science. Forthcoming.