The popcorn Problem: Understanding why you can predict macro events but not the underlying micro events.

It’s easy to predict with almost 99% certainty that at least 80% of the popcorn will pop within 5 minutes of applying a high enough heat.

In fact, after enough observations you may be able to predict when the first popcorn will pop, and how long it will take before 95% of the popcorn is popped.

What you almost certainly can not do is predict which popcorn will pop first, and absolutely can not do is to predict in what order the popcorn will pop.

After all, even if you keep all of the external components consistent, e.g. the heat, pan, oil, etc. each popcorn has some degree of uniqueness which you neither can completely measure nor control for in regards to the amount of moisture in the kernel, its shape, size and hull strength, etc.

Second, the ability to predict specific events is exponentially challenged by the chaotic interplay between the popcorn, as each kernel exploding impacts the popping of their fellow popcorn.

So even if you could accurately measure every contributing aspect of the kernel, it will be absolutely impossible to predict how each explosion will impact the other popcorn nearby with precision, as each pop transfers energy to other kernels and chaotically shuffles the environment.

However, just because you cannot predict which popcorn will pop first, and then in what order the popcorn will pop, does not mean you cannot predict the overall results,i.e. 90% of the popcorn will pop after X time.

I call this the Forecasting’s Popcorn Problem, where it may be very easy to predict overall outcomes with high certainty given long enough time horizons and big enough populations, but almost impossible to predict the specific events that happen along the way.

It’s critical to understand the Popcorn Problem because

unfortunately, people:

1. Want to believe that it’s possible to predict specific events in specific orders with high confidence when it’s just not possible.
2. Use forecasters inability to predict specific events along the way to cast doubt on their larger models/predictions.
3. Falsely overstate their ability to predict specific events with any degree of certainty.

The Popcorn Problem rears its ugly head across just about every field of life, from predicting recessions and pandemics to weather and climate change.

Let’s start with climate change and weather.

We know the atmosphere operates like a greenhouse, letting in and trapping a certain percentage of the sun’s warmth. If you increase greenhouse gases (GHGs), the earth will trap more of the sun’s heat, therefore scientists can conclusively state that increasing GHGs will result in increased atmospheric and ocean temperatures. With enough data, computing power and mathematical models, scientists may even be able to predict how much the temperatures will rise under different conditions and what the overall impacts will be.

But just like we cannot predict exactly which kernel will pop in which order, scientists will not be able to precisely predict the specific weather events that will be triggered by the rising temperatures, given all the interdependencies and unknowns about the environment.

While we can forecast that specific areas will begin to see certain impacts, we will not be able to predict the times and dates of specific weather phenomena.

At the same time that doesn’t mean we cannot predict with great certainty that the earth is rapidly warming and that we should expect weather phenomena like more powerful hurricanes, longer droughts, bigger storms, etc. as a result, even if we cannot predict the quantity, timing or exact size of individual events.

Pandemics, earthquakes and recessions are no different. Even if we cannot predict exactly when they’ll happen and how bad they’ll be, we can confidently state that they will happen, and frequently identify what the common drivers of them are, etc. so that we can try to prepare for them.

So just remember, even just because people cannot predict exactly what will happen tomorrow or next year, doesn’t mean they cannot predict with high degrees of certainty various events happening decades from now.

On the flip side, just because people can predict that yes indeed 90% of the popcorn will pop will pop in the next five minutes, doesn’t mean that they can predict which popcorn will pop first, nor in what order they will pop.