A Psychic Friend Rolling Dice

Suppose your friend tells you they can predict the outcome of a six-sided die roll with 90 percent accuracy because they are psychic. You find this claim difficult to believe, so you set up a hypothesis test using the Bayes factor. As in the Mystic Seer example, you have two hypotheses you want to compare:

H₁: P(correct) = 1/6

H₂: P(correct) = 9/10

The first hypothesis, H₁, represents your belief that the die is fair, and that your friend is not psychic. If the die is fair, there is a 1 in 6 chance of guessing the result correctly. The second hypothesis, H₂, represents your friend’s belief that they can, in fact, predict the outcome of a die roll 90 percent of the time and is therefore given a 9/10 ratio. Next we need some data to start testing their claim. Your friend rolls the die 10 times and correctly guesses the outcome of the roll 9 times.

Comparing Likelihoods

As we often have in previous chapters, we’ll start by looking at the Bayes factor, assuming for now that the prior odds for each hypothesis are equal. We’ll formulate our likelihood ratio as:

Likelihood Ratio = P(D|H₂) ÷ P(D|H₁)

so that our results will tell us how many times better (or worse) your friend’s claim of being psychic explains the data than your hypothesis does.

For this example, we’ll use the variable BF for “Bayes factor” in our equations for brevity. Here is our result, taking into account the fact that your friend correctly predicted 9 out of 10 rolls:

Bayes Factor = P(D₁₀|H₂) ÷ P(D₁₀|H₁)

= [C(10,9) × (9/10)⁹ × (1/10)¹] ÷ [C(10,9) × (1/6)⁹ × (5/6)¹]

= 468,517

Where C(10,9) represents “10 choose 9” combinations

Our likelihood ratio shows that the friend-being-psychic hypothesis explains the data 468,517 times better than the hypothesis that your friend is just lucky. This is a bit concerning. According to the Bayes factor chart we saw in earlier chapters, this means we should be nearly certain that H₂ is true and your friend is psychic. Unless you’re already a deep believer in the possibility of psychic powers, something seems very wrong here.

Incorporating Prior Odds

In most cases where the likelihood alone gives us strange results, we can solve the problem by including our prior probabilities. Clearly, we don’t believe in our friend’s hypothesis nearly as strongly as we believe in our own, so it makes sense to create a strong prior odds in favor of our hypothesis. We can start by simply setting our odds ratio high enough that it cancels out the extreme result of the Bayes factor, and see if this fixes our problem:

Prior Odds O(H₂) = 1/468,517

Now, when we work out our full posterior odds, we find that we are, once again, unconvinced that your friend is psychic:

Posterior Odds = Prior Odds × Likelihood Ratio

= O(H₂) × [P(D₁₀|H₂) ÷ P(D₁₀|H₁)]

= 1

For now, it looks like prior odds have once again saved us from a problem that occurred when we looked only at the Bayes factor.

But suppose your friend rolls the die five more times and successfully predicts all five outcomes. Now we have a new set of data, D₁₅, which represents 15 rolls of a die, 14 of which your friend guessed accurately. Now when we calculate our posterior odds, we see that even our extreme prior is of little help:

With 15 rolls (14 correct):

Posterior Odds = (1/468,517) × [(9/10)¹⁴ × (1/10)¹] ÷ [(1/6)¹⁴ × (5/6)¹]

= 4,592

Using our existing prior, with just five more rolls of the die, we have posterior odds of 4,592—which means we’re back to being nearly certain that your friend is truly psychic!

In most of our previous problems, we’ve corrected nonintuitive posterior results by adding a sane prior. We’ve added a pretty extreme prior against your friend being psychic, but our posterior odds are still strongly in favor of the hypothesis that they’re psychic.

This is a major problem, because Bayesian reasoning should align with our everyday sense of logic. Clearly, 15 rolls of a die with 14 successful guesses is highly unusual, but it’s unlikely to convince many people that the guesser truly possesses psychic powers! However, if we can’t explain what’s going on here with our hypothesis test, it means that we really can’t rely on our test to solve our everyday statistical problems.

Considering Alternative Hypotheses

The issue here is that we don’t want to believe your friend is psychic. If you found yourself in this situation in real life, it’s likely you would quickly come to some alternative conclusion. You might come to believe that your friend is using a loaded die that rolls a certain value about 90 percent of the time, for example. This represents a third hypothesis. Our Bayes factor is looking at only two possible hypotheses: H₁, the hypothesis that the die is fair, and H₂, the hypothesis that your friend is psychic.

Our Bayes factor so far tells us that it’s far more likely that our friend is psychic than that they are guessing the rolls of a fair die correctly. When we think of the conclusion in those terms, it makes more sense: with these results, it’s extremely unlikely that the die is fair. We don’t feel comfortable accepting the H₂ alternative, because our own beliefs about the world don’t support the idea that H₂ is a realistic explanation.

It’s important to understand that a hypothesis test compares only two explanations for an event, but very often there are countless possible explanations. If the winning hypothesis doesn’t convince you, you could always consider a third one.

Let’s look at what happens when we compare H₂, our winning hypothesis, with a new hypothesis, H₃: that the die is rigged so it has a certain outcome 90 percent of the time.

We’ll start with a new prior odds about H₂, which we’ll call O(H₂)′ (the tick mark is a common notation in mathematics meaning “like but not the same as”). This will represent the odds of H₂/H₃. For now, we’ll just say that we believe it’s 1,000 times more likely that your friend is using a loaded die than that your friend is really psychic (though our real prior might be much more extreme). That means the prior odds of your friend being psychic is 1/1,000. If we reexamine our new posterior odds, we get the following interesting result:

Comparing H₂ (psychic) vs H₃ (loaded die):

New Prior Odds O(H₂)′ = 1/1,000 (1,000 times more likely the die is loaded than friend is psychic)

Posterior Odds = O(H₂)′ × [P(D₁₅|H₂) ÷ P(D₁₅|H₃)]

= (1/1,000) × [(9/10)¹⁴ × (1/10)¹] ÷ [(9/10)¹⁴ × (1/10)¹]

= (1/1,000) × 1 = 1/1,000

According to this calculation, our posterior odds are the same as our prior odds, O(H₂)′. This happens because our two likelihoods are the same. In other words, P(D₁₅|H₂) = P(D₁₅|H₃). For both hypotheses, the likelihood of your friend correctly guessing the outcome of the die roll is exactly the same for the loaded die because the probability each assigns to success is the same. This means that our Bayes factor will always be 1.

These results correspond quite well to our everyday intuition; after all, prior odds aside, each hypothesis explains the data we’ve seen equally well. That means that if, before considering the data, we believe one explanation is far more likely than the other, then no amount of new evidence will change our minds. So we no longer have a problem with the data we observed; we’ve simply found a better explanation for it.

In this scenario, no amount of data will change our mind about believing H₃ over H₂ because both explain what we’ve observed equally well, and we already think that H₃ is a far more likely explanation than H₂. What’s interesting here is that we can find ourselves in this situation even if our prior beliefs are entirely irrational. Maybe you’re a strong believer in psychic phenomena and think that your friend is the most honest person on earth. In this case, you might make the prior odds O(H₂)′ = 1,000. If you believed this, no amount of data could convince you that your friend is using a loaded die.

In cases like this, it’s important to realize that if you want to solve a problem, you need to be willing to change your prior beliefs. If you’re unwilling to let go of unjustifiable prior beliefs, then, at the very least, you must acknowledge that you’re no longer reasoning in a Bayesian—or logical—way at all. We all hold irrational beliefs, and that’s perfectly okay, so long as we don’t attempt to use Bayesian reasoning to justify them.

Arguing with Relatives and Conspiracy Theorists

Anyone who has argued with relatives over a holiday dinner about politics, climate change, or their favorite movies has experienced firsthand a situation in which they are comparing two hypotheses that both explain the data equally well (to the person arguing), and only the prior remains. How can we change someone else’s (or our own) beliefs even when more data doesn’t change anything?

We’ve already seen that if you compare the belief that your friend has a loaded die and the belief that they are psychic, more data will do nothing to change your beliefs about your friend’s claim. This is because both your hypothesis and your friend’s hypothesis explain the data equally well. In order for your friend to convince you that they are psychic, they have to alter your prior beliefs. For example, since you’re suspicious that the die might be loaded, your friend could then offer to let you choose the die they roll. If you bought a new die and gave it to your friend, and they continued to accurately predict their rolls, you might start to be convinced. This same logic holds anytime you run into a problem where two hypotheses equally explain the data. In these cases, you must then see if there’s anything you can change in your prior.

Suppose after you purchase the new die for your friend and they continue to succeed, you still don’t believe them; you now claim that they must have a secret way of rolling. In response, your friend lets you roll the die for them, and they continue to successfully predict the rolls—yet you still don’t believe them. In this scenario, something else is happening beyond just a hidden hypothesis. You now have an H₄—that your friend is completely cheating—and you won’t change your mind. This means that for any data Dₙ, P(Dₙ|H₄) = 1. Clearly we’re out of Bayesian territory since you’ve essentially conceded that you won’t change your mind, but let’s see what happens mathematically if your friend persists in trying to convince you.

Let’s look at how these two explanations, H₂ and H₄, compete using our data D₁₀ with 9 correct predictions and 1 missed prediction. The Bayes factor for this is:

For 10 rolls (9 correct, 1 wrong):

Bayes Factor = P(D₁₀|H₂) ÷ P(D₁₀|H₄)

= [(9/10)⁹ × (1/10)¹] ÷ 1

= 1/26

Because you refuse to believe anything other than that your friend is cheating, the probability of what you observe is, and will always be, 1. Even though the data is exactly as we would expect in the case of your friend being psychic, we find our beliefs explain the data 26 times as well. Your friend, deeply determined to change your stubborn mind, persists and rolls 100 times, getting 90 guesses right and 10 wrong. Our Bayes factor shows something very strange that happens:

BF = P(D₁₀₀|H₂) / P(D₁₀₀|H₄) = [(9/10)⁹⁰ × (1-9/10)¹⁰] / 1 = 1/131,272,619,177,803

Even though the data seems to strongly support your friend’s hypothesis, because you refuse to budge in your beliefs, you’re now even more wildly convinced that you’re right! When we don’t allow our minds to be changed at all, more data only further convinces us we are correct.

This pattern may seem familiar to anyone who has argued with a politically radical relative or someone who adamantly believes in a conspiracy theory. In Bayesian reasoning, it is vital that our beliefs are at least falsifiable. In traditional science, falsifiability means that something can be disproved, but in our case it just means there has to be some way to reduce our belief in a hypothesis.

The danger of nonfalsifiable beliefs in Bayesian reasoning isn’t just that they can’t be proved wrong—it’s that they are strengthened even by evidence that seems to contradict them. Rather than persisting in trying to convince you, your friend should have first asked, “What can I show you that would change your mind?” If your reply had been that nothing could change your mind, then your friend would be better off not presenting you with more evidence.

So, the next time you argue with a relative over politics or conspiracy theories, you should ask them: “What evidence would change your mind?” If they have no answer to this, you’re better off not trying to defend your views with more evidence, as it will only increase your relative’s certainty in their beliefs.

Wrapping Up

Hypothesis tests can fail in three key ways. First, the Bayes factor only compares two hypotheses, but there may be other equally valid explanations worth considering. Second, when two hypotheses explain the data equally well (like psychic powers versus a loaded die), only prior beliefs matter, and more data won’t change your mind since neither hypothesis gains an advantage. In these cases, you should examine your prior beliefs. Third, and most dangerously, some hypotheses become unfalsifiable like conspiracy theories—they refuse to change regardless of evidence. When beliefs are unfalsifiable, more data actually makes you more certain of them, which is the opposite of rational reasoning. For proper Bayesian thinking, beliefs must be falsifiable, meaning there must be some possible evidence that could change your mind.