Base rate fallacy


The base rate fallacy, also called base rate neglect or base rate bias, is a fallacy. If presented with related base rate information and specific information, the mind tends to ignore the former and focus on the latter.
Base rate neglect is a specific form of the more general extension neglect.

False positive paradox

An example of the base rate fallacy is how surprised people are by the false positive paradox, situations where there are more false positive test results than true positives. For example, it might be that of 1,000 people tested for AIDS, 50 of them test positive for having it, but that is due to 10 truly having it and 40 mistaken test results, because only 10 people of those tested actually have AIDS but the test sometimes gives false results. The probability of a positive test result is determined not only by the accuracy of the test but by the characteristics of the sampled population. When the prevalence, the proportion of those who have a given condition, is lower than the test's false positive rate, even tests that have a very low chance of giving a false positive in an individual case will give more false than true positives overall. The paradox surprises most people.
It is especially counter-intuitive when interpreting a positive result in a test on a low-prevalence population after having dealt with positive results drawn from a high-prevalence population. If the false positive rate of the test is higher than the proportion of the new population with the condition, then a test administrator whose experience has been drawn from testing in a high-prevalence population may conclude from experience that a positive test result usually indicates a positive subject, when in fact a false positive is far more likely to have occurred.

Examples

Example 1: Disease

High-incidence population

Imagine running an infectious disease test on a population A of 1000 persons, in which 40% are infected. The test has a false positive rate of 5% and no false negative rate. The expected outcome of the 1000 tests on population A would be:
So, in population A, a person receiving a positive test could be over 93% confident that it correctly indicates infection.

Low-incidence population

Now consider the same test applied to population B, in which only 2% is infected. The expected outcome of 1000 tests on population B would be:
In population B, only 20 of the 69 total people with a positive test result are actually infected. So, the probability of actually being infected after one is told that one is infected is only 29% for a test that otherwise appears to be "95% accurate".
A tester with experience of group A might find it a paradox that in group B, a result that had usually correctly indicated infection is now usually a false positive. The confusion of the posterior probability of infection with the prior probability of receiving a false positive is a natural error after receiving a health-threatening test result.

Example 2: Drunk drivers

Many would answer as high as 95%, but the correct probability is about 2%.
An explanation for this is as follows: on average, for every 1,000 drivers tested,
Therefore, the probability that one of the drivers among the 1 + 49.95 = 50.95 positive test results really is drunk is.
The validity of this result does, however, hinge on the validity of the initial assumption that the police officer stopped the driver truly at random, and not because of bad driving. If that or another non-arbitrary reason for stopping the driver was present, then the calculation also involves the probability of a drunk driver driving competently and a non-drunk driver driving competently.
More formally, the same probability of roughly 0.02 can be established using Bayes's theorem. The goal is to find the probability that the driver is drunk given that the breathalyzer indicated they are drunk, which can be represented as
where D means that the breathalyzer indicates that the driver is drunk. Bayes's theorem tells us that
We were told the following in the first paragraph:
As you can see from the formula, one needs p for Bayes' theorem, which one can compute from the preceding values using the law of total probability:
which gives
Plugging these numbers into Bayes' theorem, one finds that

Example 3: Terrorist identification

In a city of 1 million inhabitants let there be 100 terrorists and 999,900 non-terrorists. To simplify the example, it is assumed that all people present in the city are inhabitants. Thus, the base rate probability of a randomly selected inhabitant of the city being a terrorist is 0.0001, and the base rate probability of that same inhabitant being a non-terrorist is 0.9999. In an attempt to catch the terrorists, the city installs an alarm system with a surveillance camera and automatic facial recognition software.
The software has two failure rates of 1%:
Suppose now that an inhabitant triggers the alarm. What is the chance that the person is a terrorist? In other words, what is P, the probability that a terrorist has been detected given the ringing of the bell? Someone making the 'base rate fallacy' would infer that there is a 99% chance that the detected person is a terrorist. Although the inference seems to make sense, it is actually bad reasoning, and a calculation below will show that the chances they are a terrorist are actually near 1%, not near 99%.
The fallacy arises from confusing the natures of two different failure rates. The 'number of non-bells per 100 terrorists' and the 'number of non-terrorists per 100 bells' are unrelated quantities. One does not necessarily equal the other, and they don't even have to be almost equal. To show this, consider what happens if an identical alarm system were set up in a second city with no terrorists at all. As in the first city, the alarm sounds for 1 out of every 100 non-terrorist inhabitants detected, but unlike in the first city, the alarm never sounds for a terrorist. Therefore, 100% of all occasions of the alarm sounding are for non-terrorists, but a false negative rate cannot even be calculated. The 'number of non-terrorists per 100 bells' in that city is 100, yet P = 0%. There is zero chance that a terrorist has been detected given the ringing of the bell.
Imagine that the first city's entire population of one million people pass in front of the camera. About 99 of the 100 terrorists will trigger the alarm—and so will about 9,999 of the 999,900 non-terrorists. Therefore, about 10,098 people will trigger the alarm, among which about 99 will be terrorists. So, the probability that a person triggering the alarm actually is a terrorist, is only about 99 in 10,098, which is less than 1%, and very, very far below our initial guess of 99%.
The base rate fallacy is so misleading in this example because there are many more non-terrorists than terrorists, and the number of false positives is so much larger than the true positives.

Findings in psychology

In experiments, people have been found to prefer individuating information over general information when the former is available.
In some experiments, students were asked to estimate the grade point averages of hypothetical students. When given relevant statistics about GPA distribution, students tended to ignore them if given descriptive information about the particular student even if the new descriptive information was obviously of little or no relevance to school performance. This finding has been used to argue that interviews are an unnecessary part of the college admissions process because interviewers are unable to pick successful candidates better than basic statistics.
Psychologists Daniel Kahneman and Amos Tversky attempted to explain this finding in terms of a simple rule or "heuristic" called representativeness. They argued that many judgments relating to likelihood, or to cause and effect, are based on how representative one thing is of another, or of a category. Kahneman considers base rate neglect to be a specific form of extension neglect. Richard Nisbett has argued that some attributional biases like the fundamental attribution error are instances of the base rate fallacy: people do not use the "consensus information" about how others behaved in similar situations and instead prefer simpler dispositional attributions.
There is considerable debate in psychology on the conditions under which people do or do not appreciate base rate information. Researchers in the heuristics-and-biases program have stressed empirical findings showing that people tend to ignore base rates and make inferences that violate certain norms of probabilistic reasoning, such as Bayes' theorem. The conclusion drawn from this line of research was that human probabilistic thinking is fundamentally flawed and error-prone. Other researchers have emphasized the link between cognitive processes and information formats, arguing that such conclusions are not generally warranted.
Consider again Example 2 from above. The required inference is to estimate the probability that a driver is drunk, given that the breathalyzer test is positive. Formally, this probability can be calculated using Bayes' theorem, as shown above. However, there are different ways of presenting the relevant information. Consider the following, formally equivalent variant of the problem:
In this case, the relevant numerical information—p, p, p—is presented in terms of natural frequencies with respect to a certain reference class. Empirical studies show that people's inferences correspond more closely to Bayes' rule when information is presented this way, helping to overcome base-rate neglect in laypeople and experts. As a consequence, organizations like the Cochrane Collaboration recommend using this kind of format for communicating health statistics. Teaching people to translate these kinds of Bayesian reasoning problems into natural frequency formats is more effective than merely teaching them to plug probabilities into Bayes' theorem. It has also been shown that graphical representations of natural frequencies help people to make better inferences.
Why are natural frequency formats helpful? One important reason is that this information format facilitates the required inference because it simplifies the necessary calculations. This can be seen when using an alternative way of computing the required probability p:
where N denotes the number of drivers that are drunk and get a positive breathalyzer result, and N denotes the total number of cases with a positive breathalyzer result. The equivalence of this equation to the above one follows from the axioms of probability theory, according to which N = N × p × p. Importantly, although this equation is formally equivalent to Bayes' rule, it is not psychologically equivalent. Using natural frequencies simplifies the inference because the required mathematical operation can be performed on natural numbers, instead of normalized fractions, because it makes the high number of false positives more transparent, and because natural frequencies exhibit a "nested-set structure".
Not every frequency format facilitates Bayesian reasoning. Natural frequencies refer to frequency information that results from natural sampling, which preserves base rate information. This is different from systematic sampling, in which base rates are fixed a priori. In the latter case it is not possible to infer the posterior probability p from comparing the number of drivers who are drunk and test positive compared to the total number of people who get a positive breathalyzer result, because base rate information is not preserved and must be explicitly re-introduced using Bayes' theorem.