Section: 10 🔖 Test of Independence
10.1 Introduction
We would like to test if a student’s final grade in Professor Moody’s class depends on the student’s major. The null hypothesis in this case is the hypothesis of independence. Independence means that the distribution of final grades is the same no matter what the major is. The alternative hypothesis is that the distribution of final grades changes from major to major, rejecting the null hypothesis of independence.
Notice that we do not specify how the grades depend on students’ majors. Do CS students get better grades than Psychology majors? Do Economics majors get lower grades than Statistics majors? We do not care about this. We are only testing here whether there is a relationship between Major and the final grade distribution.
We will describe the permutation test which scrambles our data randomly in such a way that any relationship between grades and major (if it ever existed) is broken. We will run a permutation test a large number of times - possibly tens of thousands of times. Then we will determine how likely it is to randomly obtain the observed result. But what is the observed result? It is a bit more complex than the difference of means which were observing the difference of means hypothesis testing.
The observed result in our case is calculated by so called chi-square statistic which is calculated on the contingency table, table(moody$Grade, moody$Major)
To explain it, let us first define two tables: The observed contingency table and the expected contingency table. It is calculated by table() function.
OBSERVED CONTINGENCY TABLE| Grade/Major | CS | Economics | Psychology | Statistics |
| A | 46 | 54 | 69 | 42 |
| B | 46 | 12 | 2 | 35 |
| C | 51 | 33 | 30 | 34 |
| D | 41 | 37 | 29 | 34 |
| F | 108 | 99 | 99 | 99 |
The second table we need is called the expected table. This is the hypothetical table of the relationship between grade and major, assuming grade and major are completely independent. Such a table would be the result of the same distribution of grades for each of the majors. Notice that we 1000 students in the data set the expected table (i.e. table which have grades completely independent from majors) would have the same distribution of grades for each major that over all students - which is shown by the TOTAL column.
EXPECTED CONTINGENCY TABLE| Grade/Major | CS | Economics | Psychology | Statistics | |
| A | 61 | 50 | 48 | 51 | |
| B | 28 | 22 | 22 | 23 | |
| C | 43 | 35 | 34 | 36 | |
| D | 41 | 33 | 32 | 34 | |
| F | 118 | 96 | 93 | 99 | |
| TOTAL | 292 | 236 | 229 | 244 | 1000 |
We kept fractions - although these are number of students - therefore would have to be rounded up to integers
The chi-square formula calculates the “distance” between the observed contingency table and the expected contingency table.
\[\begin{equation} \sum \frac{(O_i - E_i)^2}{E_i}\\ \text{where:}\\ \text{O = observed values}\\ \text{E = expected values}\\ \end{equation}\]
For the two tables above the,
\[\begin{equation} \sum \frac{(O_i - E_i)^2}{E_i} = 60.03\\ \end{equation}\]
To evaluate how far off is this observed result assuming that Grades are independent from Major, we run a permutation test which scrambles Grades and Majors randomly and every time computes the chi-square formula with the new observed table (the expected table is always the same). Then, we see how many times out of, say 10,000 iterations of permutation test we obtain a result larger than the observed result of 60.03? This is the p-value.
Permutation test for independence hypothesis gives us again a better feeling about the impact of randomness and whether the observed chi-square result for “similarity” of grade distributions for different majors can be obtained randomly. In the following snippet we run the chisq test which is based on the so-called chi square distribution. Here we simply show you a function which can calculate p-value, just like the z-test function does. The explanation of the chi-square test is provided in attached link to the excellent Khan Academy video.
Permutation tests in both cases of difference of means and independence hypotheses give a better intuitive sense of how we answer the question - can the observed result be obtained randomly?
Notice that the independence test is looking globally at two vectors and whether one depends on another. If we wanted to be more specific and know if psychology majors are more likely to get an A than CS majors, we can frame this as a difference of means hypothesis. Testing this hypothesis will be using the difference of means of frequencies of A’s among CS majors and psychology majors. This could be done again by permutation test in section 8 or the z-test.