Archive for February, 2009

Survivor Bias in Log Cabins

Michael Graham Richard has a great post about survivor bias of frontier log cabins – essentially all the ones you see are well made because they were well made:

I have to chuckle whenever I read yet another description of American frontier log cabins as having been well crafted or sturdily or beautifully built. The much more likely truth is that 99% of frontier log cabins were horribly built—it’s just that all of those fell down. The few that have survived intact were the ones that were well made. That doesn’t mean all of them were.

He also makes the point that the classical music you hear today is good precisely because it is still around – history buries the mediocre.


Read Full Post »

Spolsky on Survivor Bias

Joel Spolsky (of Joel on Software fame) has a column “How Hard could it be? Startup Static” about survivor bias when trying to emulate successful companies:

The problem is that trying to copy one company’s model is a fool’s errand. It’s hard to figure out which part of the Starbucks formula made the business a smash hit while so many of its rivals failed. Starbucks’s success is the product of a combination of factors that came together in precisely the right way at precisely the right time. It’s nearly impossible to isolate which one was the most important. You would probably have to look at the hundreds of small coffee chains that didn’t make it big before you stood a chance of seeing what really distinguished Starbucks.

The survivorship bias in entrepreneurship was on my mind a few months ago. My company was putting together a conference in Boston, and I invited my friend Jessica Livingston to speak. Jessica is the co-founder of a small angel investment group called Y Combinator. Its model is to give a few thousand dollars to groups of two or three geeks to start tech companies. She has also written a book called Founders at Work, in which she interviews the founders of about 30 successful start-ups. When she asked me what she should speak about, I asked her to consider describing all the different ways a start-up can fail, rather than the usual stuff about lessons learned from people who succeeded.

“That would be boring,” she told me. “They all fail for the same reason: People just stop working on their business.” Um, yeah, well, sure, and most people die because their heart stops beating. But somehow dying in different ways is still interesting enough to support 40 hours a week of prime-time programming.

But the more I thought about it, the more I realized Jessica was onto something. Why do start-ups fail? As she pointed out, it’s usually a collapse of motivation — everyone wanders back to civilian life, and the start-up ends, not with a bang but a whimper.

I wrote on this topic earlier in “Beware advice from the successful” and “Good to Not so great“. Startups fail for lots of reasons, they also succeed for lots of reasons and what worked for Starbucks (even if you could rigorously determine why) won’t really help you much.

People have a strong tendency to attribute to skill everything they did right and luck to everything they did wrong. In reality, it’s best to think of the world as a very large roulette table (or Russian Roulette depending on your predicament) – skill will allow you to place more bets on the table but it’s not a guarantee of success. Couple this bias with the lack of data and an inability to reproduce the experiment – people generally don’t have the ability to rerun their lives to determine what would have happened if they had made some choice differently. This is where Survivor Bias appears – by asking the successful, your ignoring all the people who were unsuccessful.

Read Full Post »

Car Talk recently had a puzzler that involved another example of (literal) survivor bias:

It’s World War II, an RAF airfield north of London. A dimly lit Quonset hut filled with air crews just returned from bombing runs over Germany.

The meeting opens with the chaplain leading the men in prayer for their lost comrades. He is followed by the flight operations chief, who begins the debriefing by asking the airmen, “From what direction were you attacked by the German fighter planes?”

Without hesitation or dissent, the reply was, “From above and behind.”

As in our previous example of this (which also involved world war II pilots…), the problem is that they are only interviewing the survivors.

Read Full Post »