Following the experiment I described in a previous post, one side question that was raised concerns the relevance of the confidence level as an indicator of accuracy. That is, we wanted to know if it’s a good strategy to imitate people expressing a high level of confidence.
To answer this question, we have asked participants in the lab a series of 32 factual questions, such as “To you opinion, what is the distance between Berlin and London ?”. Additionally, participants were instructed to evaluate their level of confidence on a scale ranging from 1 (very unsure), to 6 (very sure).
Now the question is: How strong is the relation between the confidence level of the participants and their accuracy? The following figure gives some first answers.
So in the above figure, the colors indicate the confidence level, from 1 to 6, and the height of the bars indicate the probability to get a very good, a good, a bad, or a very bad answer. The maximum confidence level 6 is indeed a good indicator of the quality of the answer. It leads to a good or very good estimate about 80% of the time. However, lower confidence levels are less reliable. For example, the second highest confidence value C=5 has already 40% chance to be a bad or very bad estimate – not far from a random chance! Moreover, the confidence level C=4 – which is still above the average – actually has a greater chance to be a bad or very bad answer (53%) than good or very good (47%). Finally, the lowest confidence values C=1 and C=2 do not even differ from each other.
Therefore, the confidence level is a bad indicator of quality. Only people who are absolutely certain of what they say are reliable, not the others.
(and by the way…. Berlin is 910 km away from London).
Reference:
M. Moussaïd, et al.
Social influence and the collective dynamics of opinion formation
PLoS ONE, 2013