About Trip Advisor and American Bomber Pilot Survival

In July 1980 the Center For Naval Analysis reprinted a paper called A Method of Estimating Plane Vulnerability Based on Damage of Survivors. This research paper was the work of Abraham Wald, an Austrian Jew who had narrowly escaped the holocaust. Wald wrote the paper in 1943, whilst he was employed by the American military in a top secret establishment called the Statistical Research Group - part of Columbia University operating from a non-descript apartment in New York.

During the second world war the survival rate of American bomber crews was a great concern. American bomber command preferred to fly missions in daylight, when crews could sight their targets better than the British planes, which flew at night. This made them more vulnerable to Nazi defences. At the height of the conflict there was only a 50% probability that American aircrews would survive a tour of duty.

Military chiefs knew that protecting the crews was vital but they could not armour whole planes since the extra weight would make it impossible for them to take off with a full load of bombs and fuel. They’d noticed that many of the planes that were shot but which made it back to base had most damage on their wings so understandably this is where they wanted to place armour. Wald and his team were asked to analyse the problem.

If you’re interested in complex mathematical equations, you can search on line and find the whole paper but in a nutshell what Wald found was that armour plating the wings would have had little effect. The military chiefs had analysed the wrong data, or more accurately they had not analysed all of the data. They’d neglected to consider what was happening to the planes that had not returned. It turned out that these planes had been hit in their fuselages and cockpits.

You’ll be starting to see how this affects your business and why any amount of market research may lead you in the wrong direction. Let me give an example.

A few years back I was working and writing in the Middle East. Dubai was on a roll and Abu Dhabi and other big cities in the Gulf were surging. It seemed like everyone wanted a chain of restaurants. I went to one meeting where I was told by an obnoxious man with a perfectly straight face that his company planned to open 500 outlets in 18 months. In the rush no-one wanted to invest time and money designing new concepts, instead they’d franchise from elsewhere and wanted everything at best price. No-one wanted to build a lasting restaurant chain by patiently coming to understand their customers whilst making endless incremental improvements for many years.

Today, if you visit the Mall of the Emirates in Dubai, or many others in the region, you’ll find most of the outlets are just the same as everywhere else only bigger and flashier. The chances are that when you’ve finished your meal you’ll be given a customer satisfaction form with check boxes and a place where you can write your comments. This is intended to help the restaurant owner (who is most likely a huge property-developer) understand what their staff have done wrong. I’ll leave you to judge the wisdom of this approach.

I found it offensive to be asked to do work for the restaurant after every meal. A good restaurant manager would have been present and would know without being so unsubtle.

But now we see why this approach and others like it can’t work. The feedback received is never all of the data. It’s only part of the story. The data is only provided by those want to complete the form, who tell the truth and most importantly who ate in the restaurant in the first place. In other words, it’s a subset of a subset of a subset of all the data. What the restaurant owners really needed to know was why customers of other restaurants had not eaten in theirs.

Nowadays we have Trip Advisor and other on-line reviews. Again these only provide part of the data. Sure it’s important to take comments seriously and try to improve but the real questions you need to ask is: how do I get all of the data?

You’ll find the answer is not so easy. This doesn’t mean you can safely ignore it.