Here's a fun video to test your reflexes: Go watch this video of a few students passing a basketball to each other and count how many passes there are. It's harder than it looks. How many passes did you count?
The researchers in this paper asked: does the same phenomenon occur when we are analyzing a dataset?
So they made up a dataset and asked students to analyze it. The dataset contained made-up data about men's and women's body mass index (BMI) and the number of steps they took.
Students were split into two groups. The first group was asked to consider specific questions about the data, such as is there a statistically significant difference in the average number of steps taken by men and women? They were also asked if there was anything else important they could conclude about the data.
The second group was just given the dataset and asked what they could conclude – without any additional prompts.
There was one catch. When the data were plotted together, it looked like this.
The first group, the "hypothesis-focused" group that was given specific questions to answer, found the gorilla much less frequently than the second group.
One lesson I took from the experiment is to keep an open mind. The paper says it even better, "Not all who wander are lost."
For those interested, the original paper (2020). An interesting response from Andrew Gelman, professor of statistics and political science at Columbia University.