Making Room for Validity – Valutrics
Both reliability and validity are important for an organization. Without validity, an organization has little chance of moving knowledge across the funnel. Without reliability, an organization will struggle to exploit the rewards of its advances. As with exploration and exploitation, the optimal approach to validity and reliability is not to choose but to seek a balance of both.
The precise method for balancing validity and reliability will vary from situation to situation and from organization to organization. It may be that some areas of an organization (accounting, for example) will emphasize reliable measures, while others (R&D) will embrace valid ones. Still other departments, marketing for instance, may seek to design new measures that in themselves strike a balance, incorporating reliable structures around qualitative research methods, for example. But, given that companies have very real and powerful reasons to favor reliability over validity, and that this preference for reliability is often enshrined in the organization’s structures, processes, and norms, the challenge will typically be to incorporate a validity orientation into a reliability culture. To do so, the organization must open up new definitions of proof, embrace some degree of subjectivity as not just inevitable but valuable, and acknowledge that getting the right answer is worth taking a little more time. It must open itself up to a new way of thinking. That skill is abductive reasoning, which drives the intuitive spark that leaps across the gap separating the world as it is from the world as it might be.
FIGURE 2-1
The predilection gap
Let us start with examining a reliable prediction. What could this mean? Well, a reliable prediction, which we’ll denote by a sentence, S, about some future event, is one that is produced by a reliable process. A reliable process, in turn, is one that has turned up a reliable prediction N times in the past, where N is a very large number. A reliable prediction of the effects of mortgage rates on housing prices is that they vary in inverse proportion. It is reliable because the process that generated it has worked in the past.
Now, what is a valid prediction? It is a prediction that turns out to be true. A prediction is always about the future, and, as such, it cannot be judged to be valid before the future about which it is predicated actually happens.
The “inductive fallacy” is that of inferring validity from reliability. In its strong (and most pernicious) form, the inductive fallacy states that the reliability of a prediction logically entails its validity. So, a reliable prediction would be that high-tech start-up valuations should increase as a large multiple of hits on the fledgling company’s Web site and the number of engineers employed by the company. Using this reliable method for predicting firm valuation post September 2000 would, however, have led to an invalid prediction and plenty of invalid investments, as it turned out.
The weak form of the inductive fallacy is that the probability of a prediction being valid, given that it is reliable, is greater than the probability of its being invalid, given that it is reliable. Bertrand Russell put this to rest with the simple fable of a chicken who expects (correctly) to be fed every time the farmer appears in the morning, and who predicts (reliably, but not validly) that it will also be fed on (what turns out to be) its last day on Earth, when the farmer comes to chop off its head.
The problem of going from reliability to validity is, of course, the “all things being equal” condition that appears in most experimental reports and empirical study results, which states that the supposed cause-effect relationship supported by the data will be operational in other contexts, “all things being equal.” But that is precisely the point of living in an open, uncontrolled system, also known as the world: all things are not equal from one experimental run to another. The step that safeguards the transition from reliability to validity is not a simple inferential step but, rather, a far more complex, abductive step. The validity seeker, unlike the reliability seeker, treats past predictive successes as hypotheses to be carefully tested before using them to generate predictions that are expected to be valid. Hence, the real empiricist is “a first-rate noticer” of precisely the anomalies that would cause him or her to throw out the “all things are equal” assumption.