Spolsky on Survivor Bias

Joel Spolsky (of Joel on Software fame) has a column “How Hard could it be? Startup Static” about survivor bias when trying to emulate successful companies:

The problem is that trying to copy one company’s model is a fool’s errand. It’s hard to figure out which part of the Starbucks formula made the business a smash hit while so many of its rivals failed. Starbucks’s success is the product of a combination of factors that came together in precisely the right way at precisely the right time. It’s nearly impossible to isolate which one was the most important. You would probably have to look at the hundreds of small coffee chains that didn’t make it big before you stood a chance of seeing what really distinguished Starbucks.

…

The survivorship bias in entrepreneurship was on my mind a few months ago. My company was putting together a conference in Boston, and I invited my friend Jessica Livingston to speak. Jessica is the co-founder of a small angel investment group called Y Combinator. Its model is to give a few thousand dollars to groups of two or three geeks to start tech companies. She has also written a book called Founders at Work, in which she interviews the founders of about 30 successful start-ups. When she asked me what she should speak about, I asked her to consider describing all the different ways a start-up can fail, rather than the usual stuff about lessons learned from people who succeeded.

“That would be boring,” she told me. “They all fail for the same reason: People just stop working on their business.” Um, yeah, well, sure, and most people die because their heart stops beating. But somehow dying in different ways is still interesting enough to support 40 hours a week of prime-time programming.

But the more I thought about it, the more I realized Jessica was onto something. Why do start-ups fail? As she pointed out, it’s usually a collapse of motivation — everyone wanders back to civilian life, and the start-up ends, not with a bang but a whimper.

I wrote on this topic earlier in “Beware advice from the successful” and “Good to Not so great“. Startups fail for lots of reasons, they also succeed for lots of reasons and what worked for Starbucks (even if you could rigorously determine why) won’t really help you much.

People have a strong tendency to attribute to skill everything they did right and luck to everything they did wrong. In reality, it’s best to think of the world as a very large roulette table (or Russian Roulette depending on your predicament) – skill will allow you to place more bets on the table but it’s not a guarantee of success. Couple this bias with the lack of data and an inability to reproduce the experiment – people generally don’t have the ability to rerun their lives to determine what would have happened if they had made some choice differently. This is where Survivor Bias appears – by asking the successful, your ignoring all the people who were unsuccessful.