Section: 22 🔖 How can data fool us?
22.1 Introduction
How not to be fooled by data? In the description of this class we promised that data 101 will teach you this. Have we? I hope so. Please use our question roulette to test yourself. In this section we discuss Simpson Paradox, Prosecutorial and Ecological Fallacies. Before we proceed with the paradoxes let us summarize what we have learned so far:
Beware of randomness(Hypothesis testing, p-values, Multiple hypothesis testing)Be an optimist- use Bayesian reasoning. Remember about prior odds first!Beware of extreme results- Apply law of small numbersRemember we process information following availability- we assign high frequency falsely to events which are just talked about very oftenNarrative fallacy - do not find false patterns- “Stocks went down due to concerns about rising cost of living”
Here we will discuss the Simpson paradox as well as Prosecutorial and Ecological fallacies. Let us start with the Simpson paradox. Here is a simple example of two basketball players, Aaron and Barry
The Table 16.1 shows Aaron’s and Barry’s free throw point averages (FTP) over the 2021 and 2022 season respectively. Clearly for both seasons Barry beats Aaron in terms of FTP, in 2021 by 90% to 80% and in 2022 by 70% to 65%. Can Aaron still beat Barry over both seasons - that is get a higher FTP over the sum of two seasons, 2021+2022? First answer which comes to mind is absolutely not. How can Aaaron beat Barry over 2021+2022 when Barry beats him in each of the two seasons?
Table 16.1
| season/player | Aaron | Barry |
| 2021 season | 80% | 90% |
| 2022 season | 65% | 70% |
But table 16.2 explains that it is quite possible that Aaron can beat Barry over 2021+ 2022. Indeed, since we do not know the absolute number of attempts at free throws, we can easily pick any number of attempts for each of them in any of the two seasons.
Indeed - here is the proof that Aaron can still beat Barry. If Barry made 100 attempts in 2021 and 20 attempts in 2022, while Barry made only 20 attempts in 2021 and 100 attempts in 2022, Aaron’s overall FRP for both 2021+2022 will be higher than Barry’s. And this is Simposon’s paradox.
Table 16.2
| season/player | Aaron | Barry |
| 2021 season | 80% out of 100 | 90% out of 20 |
| 2022 season | 65% out of 20 | 70% out of 100 |
Indeed Aaron’s FTP over 2021+2022 is
\[\begin{equation}
\frac{80+13}{120} = \frac{93}{120}
\end{equation}\]
which is larger than Barry’s
\[\begin{equation} \frac{18+20}{120} = \frac{88}{120} \end{equation}\]
More generally, trends in subsets of data may reverse themselves after aggregation.
In fact we can have any number of seasons and have Barry beat Aaron in FTP in each and every season and Aaron still wins with better FTP over all seasons. This is again simply because we do not know how many attempts each player made each season. This applies to many real world situations such as graduate admissions for example (the famous Berkeley admission bias case). There women may have a higher chance to be admitted than men in each single academic department and nevertheless, men beat women in overall acceptance ratio. This is again hard to comprehend at first but it is due to the fact that the absolute number of female and male applicants may be different for each department.
Is such reversal always possible?
Let’s look at the table below:
Table 16.3
| season/player | Aaron | Barry |
| 2021 season | 65% | 90% |
| 2022 season | 60% | 70% |
In this case the Simpson paradox is not possible. Why? Because Aaron’s highest FTP (65% in 2021 season is lower than Barry’s lowest FTP in 2022). You can easily see that no matter what the absolute numbers of attempts in each season, Aaron can never beat Barry for 2021+2022.
Thus the Simpson paradox was possible in this simple case only because Aaron’s highest FTP was higher than Barry’s lowest FTP.
One also has to be careful with the Simpson paradox and not apply it to situations when both groups /individuals have the same absolute number of “attempts”. For example, the Simpson paradox is not possible for students and their individual scores on homework’s and exams. If Barry scores higher than Aaron on each homework and on each exam then Barry will always have a higher score overall than Aaron. There is no Simpsonian trend reversal. Every homework and every exam counts the same for all students. This is as if players always made the same number of free throw attempts.
22.1.1 Ecological Paradox
Ecological paradox is kind of the reverse of the Simpson paradox. Let’s assume that we consider net worth for each member of groups A and B. Even if average net worth of group A is higher than average net worth of group B, it may be possible that random individual member of the group B has higher net worth than random individual member of the group A. Thus the order of aggregates may be reversed when we look at the level of individuals.
For example as table 16.4 illustrates, the average net worth of Group A dominates the average net worth of Group B due to the presence of one wealthy individual. However for 90% of pairs of individuals, group B members are more wealthy than Group A members.
Table 16.3
| Group A | Group B |
| $10,000,000 | $210,000 |
| $100,000 | $290,000 |
| $120,000 | $220,000 |
| $80,000 | $210,000 |
| $60,000 | $270,000 |
| $160,000 | $210,000 |
| $110,000 | $240,000 |
| $100,000 | $210,000 |
| $200,000 | $240,000 |
For example, it is well known that Democrats win the richest states, while (until recently), the richest individuals vote republican. How is it possible? Explanation is simple. Everyone’s vote counts the same and there are few very rich people. Very rich people may contribute more to the average wealth of the state (due to their extreme wealth), but there are just very few of them.
Do not be fooled by aggregates!
Let’s assume that a Democrat wins 70% of the vote and a Republican wins 30% of the vote in some state. Is it possible that, nevertheless, the republican candidate wins all 19 counties out of 20 in the state?
This actually happens a lot when the population is heavily concentrated in a heavily populated urban county which has the vast majority of voters living there.