Experimenting on skeptics

At SkeptiCamp Sydney on May 26 I conducted a small experiment to show that even people motivated to attend a skeptical conference can have their actions influenced by things they read. It was not intended to be a scientific experiment so no statistical analysis of the results has been or will be attempted. It was merely designed to demonstrate principles. It was done as part of a talk titled "We're so skeptical we can't be fooled".

Each person in the audience was given a piece of paper. On one side was an instruction about turning the page over before being asked to do so. Half the forms had the instruction "Don't turn over until asked to do so" and the other half said "Please don't turn over until asked to do so". The hypothesis was that the addition of the word "Please" would reduce the probability that the subjects would ignore the instruction. About a dozen people admitted to turning over before being asked to do so, but none of these people had the "Please" version. We can conclude from this that skeptics are more likely to do what you want them to do if you ask politely. The principle being tested here is known in skeptic circles as "Don't be a dick" and there is no reason to suggest that it doesn't apply to the wider population at large.

The main part of the experiment tested the psychological principle known as "Anchoring bias". This states that when people are asked to estimate something their guess will be influenced by information provided at the time. Told that a number is incorrect, the average estimate will be higher in a group shown a high number than in a group getting a lower hint. These were the four statements, together with the numbers shown to the two groups:

	High	Low
The number of people receiving Newstart unemployment benefits in Australia is	3,475,321	41,293
The population of West Wyalong is	73,283	803
There are X kilometres of track in the Sydney CityRail system	11,345	253
The average consumption of marijuana by high school students in South Australia is X grams per year	4,286	2

Here are the results, with the average estimate for each group shown together with the best estimate currently available for the correct value.

Experimental group	High	Low	Actual
The number of people receiving Newstart unemployment benefits in Australia is	713,252	351,568	329,500
The population of West Wyalong is	32,919	6,827	2,946
There are X kilometres of track in the Sydney CityRail system	24,512	2,020	1,595
The average consumption of marijuana by high school students in South Australia is X grams per year	791	349

There was no useful estimate available for marijuana consumption in SA high schools, although figures do show a significant reduction in marijuana use in the age group over the last few years. Five subjects in the "High" group estimated use at 10 kilograms per year. As this would represent heavy individual usage and would be almost inconceivable as an average these were exclude from the results. Exclusion of these outliers did not affect the direction of the overall results with the "High" group still having a higher average estimate than that for subjects shown a low figure.

So on all four statements the people shown a higher target provided, on average, higher estimates with the opposite happening for the group shown lower clues. If this proves anything at all it is that committed skeptics are little different to non-skeptics in the way that they are influenced by information provided to them in the absence of corroborating data. I would hope that the difference is that skeptics would not take single, unsupported numbers at face value and would ask for additional evidence to back up what they are told. Isn't this what makes us skeptics, after all?

References:

Anchoring bias in decision-making
Jobseekers receiving Newstart Allowance
West Wyalong Local Stats
CityRail Facts and Stats
Statistics on Illicit Drug Use in South Australia