Ebb and flow
In line at the coffee shop
It’s a bright morning. I'm well-rested and woke up before the alarm rang. I get ready to hit my local coffee shop to start the day with a cup of double-shot espresso. As I wait for coffee, I indulge in small talk with a stranger next to me in line. She’s attractive.
We hit it off and the conversation oozes chemistry. Espresso is ready. I take the first sip while Amber taps the digits of her number on my phone. The coffee is not too sour - the right amount of creamy bitterness. As I walk out, Amber reminds me to text her. I know today can’t get any better.
The next morning I am still buzzing from the energy of the day before. My mind is all excited. I want to recreate that smooth experience. I get ready to head out to the cafe, only to realize my favorite spot has been taken. Oh, and the line is extra long today.
Everyone around me is immersed in their own thing and kind of cranky. Even the coffee today tastes overroasted. There’s no refreshing aroma this time. No stranger willing to make eye contact, let alone small talk. The entire experience is so...mediocre. And that’s the story of most of my days.
What happened though? What changed from the day before? The way I trimmed the edges of my beard? The water filter in the coffee machine? I try to come up with plausible answers, but none of them offer enough solid ground. Statistics might illuminate the path here.
Fooled by randomness
An extreme event follows an extreme in the opposite direction. The notion of regression to the mean was first worked out by Sir Francis Galton. In a series of phenomena dependent on many variables, extreme outcomes tend to be followed by more moderate ones. Very good luck is not likely to reoccur a second time.
On average days (or even bad ones) it's normal to think some negative force is conjuring against us. But in truth, there is no such cryptic reason as to why my morning coffee wasn't so great. In fact, reversion to the mean is a statistical tendency.
It happens to everyone and everything, all the time. Whenever a variable whose behavior is accounted for partly by randomness, you're bound to see a reversion to the mean. If your variable doesn't depend on randomness - well, you will register the same result, over and over again. There won't be any variance to begin with. But reality is rarely like that.
In fact, reality is never like that. It's riddled with randomness and factors that are too complicated for us to predict. As a result, we will always see extreme events from time to time in both directions. But over the long run they seem to cancel each other out and the outcomes tend towards a mean.
Predicting ski jumps
Given its universal nature, one would expect to develop an intuition for this phenomenon. You'll be surprised by how often reversion to the mean can be overlooked. One of the more famous examples about reversion to the mean is in Daniel Kahneman’s book “Thinking Fast and Slow”.
Kahneman recalls watching men’s ski jump. In that discipline, the final score is a combination of two separate jumps. He was startled to hear the commentator’s predictions about the second jump. He writes:
"Norway had a great first jump. He will be tense, hoping to protect his lead and will probably do worse.” or “Sweden had a bad first jump and now he knows he has nothing to lose and will be relaxed, which should help him do better".
Kahneman points out that the commentator had noticed the regression to the mean and come up with a story with no causal evidence. This is not to say that his story could not be true. If we measured the heart rates before each jump, we would see that they are more relaxed if the first jump was bad. But that’s not the point. Regression to the mean happens when luck plays a role, as it did in the outcome of the first jump.
You see, someone who performed really well probably had some luck on their side. Of course, these are trained professionals and their abilities have a part to play in what they do. But sometimes luck did not assist you all the way through, despite ability.
The lesson from sports applies to any activity where chance plays a role. We often attach our influence over a particular process to the progress. Or lack of it. So how can we know what interventions work, and which don't?
Smart women "date down"
This at first might all seem confusing and not very intuitive. But the degree of regression to the mean is directly related to the degree of correlation of the variables.
Assume you are at a party and ask why it is that highly intelligent women tend to marry less intelligent men. Most people will quickly jump in with a variety of causal explanations. From avoidance of competition to the fears of loneliness that these women face. A topic of such controversy is likely to stir up a great debate.
Now, what if we asked why the correlation between the intelligence scores of spouses is less than perfect? This question is hardly as interesting and there is little to guess – we all know this to be true. The paradox lies in the fact that the two questions happen to be algebraically equivalent. Kahneman explains:
[…] If the correlation between the intelligence of spouses is less than perfect (and if men and women on average do not differ in intelligence), then it is a mathematical inevitability that highly intelligent women will be married to husbands who are on average less intelligent than they are (and vice versa, of course). The observed regression to the mean cannot be more interesting or more explainable than the imperfect correlation.
The chances of two partners representing the top 1% in terms of any characteristic are extremely low.
Placebos, elections and stock prices
Take medical interventions now. How do we know whether they are effective at all, given that patients with most illnesses tend to feel better with time? How do we know the response to treatment is not regressing to the mean?
That's where control groups and placebos comes in. Once you realize the effect of reversion to the mean, it becomes imperative to have control groups. The intervention needs to be tested to distinguish its effect from a matter of reversion.
What I mean is, if a medicine works better than reversion to the mean, we can be certain about its effect. This has become a crucial part of the medical process and the greater scientific process. Again, reversion to the mean is everywhere, and you have to isolate its impact in your conclusions. The consequences of not doing so can be dangerous.
Reversion to the mean is also seen in the electoral process. More extreme candidates tend to be followed by less extreme candidates. Then there are financial markets, where prices can stall or skyrocket. But generally, they stay in the ballpark of some market average. In fact, moving averages are used by traders around the world every day. Of course, most of us are naive to this.
Reversion to the mean also applies to individual circumstances. If you did well on a test, chances are you won't do so well the next time. Exemplary performance is rarely sustainable. At the same time, if you did poorly the first time around, you're more likely to do better the second time around.
Your ability definitely has a role to play in what you do. But the statistical tendency of ebb and flow remains.
The greatest athlete of all times
You can see it in sports. Athletes who do well in their rookie season rarely live up to expectations. There are many references to this phenomenon as the “Sports Illustrated cover jinx”, or the "commentators curse". Of course, the commentator isn't casting some dark spell over the players. Given the spectacular nature of what a player did, they are not likely to recreate it again.
Michael Jordan was an exceptional basketball player. Needless to say, his talents were justified to the extreme. But regression to the mean is not a natural law. Merely a statistical tendency. And it may take a long time before it happens. If reversion towards the mean is indeed correct, it would predict that his sons were not likely to reach the heights their father did. And that's exactly what happened.
Even though they had inherited the privilege that came with being Michael Jordan's sons, they never made it. They were successful college athletes, sure. But they were no NBA players, let alone one of the greatest athletes to have ever been. Chances are they've worked hard, but genetics is a tad too random. And that makes the effects of reversion to the mean that much more pronounced.
But in some sense, you should thank genetics for being so random. After all, his dad was 175cm, or 5’9’’, and his mom 165cm, or 5’5’’. Jordan’s parents were exactly average height. Michael was 1.98cm, or 6’6’’. And without that seemingly lucky genetical boost in height, who knows? Maybe he would have never become one of the greatest basketball players of all time.
Aboard my ship
Reality tends towards mediocrity. And there's nothing you can do about it. I don't know about you, but I find that hard to accept. To me, the concept of reversion to the mean seems almost boring, and disheartening as well.
All this takes away from the notion that hard work is what gets you success, not chance events. It further casts a shadow on the already poorly lit facade of free will. We're all indeed ebbing and flowing between spectacular and awful, only to end up in mediocre.
Heck, what control do we really have, then? Besides, it seems as though statistics is almost forcing us to be mediocre, to be less than good. And nobody likes to be mediocre.
US Naval Captain Philip Queeg said “Aboard my ship, excellent performance is standard, standard performance is sub-standard, and sub-standard performance is not permitted to exist - that, I warn you.”.
I think a lot of us feel the same way. There's something fundamentally nauseating about mediocrity. Of course, "average" should by definition be a neutral word. But it's hard to disassociate the negative connotation that seems to carry.
That is until you fall sick or until you lose a loved one, or don't have access to food or water, or shelter. Then, well, being average is not too bad, then being healthy seems priceless. Ask someone who has an empty seat at the dining table. A normal family has all they could ask for. Being mediocre doesn't seem so bad, does it?
What we can control is the statistical tendency of the cause of the reversion to the mean. We can gradually push the mean closer to where we want it to be with consistent but ever-so-slight strides. And with hard work, we can all become better averages.
Mean reversion also serves as a reminder that we may not be as responsible for our successes. And as far as free will is concerned, maybe its truest nature lies in the awareness of its limitations. But life is finite. Now let me text Amber and see what she's up to on Friday.