Photo by McGeddon - Own work, CC BY-SA 4.0 , Link
Every day I’m amazed by the wisdom of my Russian and Ukrainian colleagues. Today while we were talking about automated testing and security topics, I shared an old anecdote that happened to me with Sergio Marino and Rodrigo Román 10 years ago where we were discussing the compatibility of the website we were working on with IE 5.5 or some other browser from yesteryear. Since that browser wasn’t in the site’s usage statistics, the idea arose that nobody used the site with that browser, and then we erroneously inferred that its compatibility wasn’t important; to which Sergio and Rodrigo logically refuted saying that if it wasn’t in the statistics, it was because simply people with that browser couldn’t use the site and therefore weren’t visible in the statistics.
Going back to 2018 when I finished telling this anecdote, my colleagues said that this had a name, and after googling a bit we came across the concept of Survivorship bias, which would be a logical error that consists of concentrating on the part of the sample that manages to pass through a selection process X, and ignoring the part of the sample that doesn’t achieve it due to lack of visibility. This error can lead to erroneous conclusions.
They immediately commented that they had read a Russian book that mentioned the brilliant example of planes that return with anti-aircraft fire holes in the fuselage and wings after going to bomb (something that doesn’t happen so often in South American countries). And the airplane technicians erroneously state: “We need to reinforce the parts of the plane where there are holes.” But they’re not considering the planes that haven’t returned after the bombing, so they should concentrate on reinforcing the areas that haven’t suffered anti-aircraft fire impacts, or at least try to analyze what happened to the planes that didn’t return.
Actually it would be better if no plane went out to bomb anything… but to understand the Survivorship bias concept I think it’s a good example.