How to Measure Anything: Finding the Value of Intangibles in Business

Perhaps the most frequent question from decision analysis team members is, “How do we get the inputs?” In most evaluations, there are several key variables about which we know little. Consider oil price, for example. We have abundant historical data, yet forecasting future prices is a daunting challenge.

Doug Hubbard has written an entire book about capturing quantitative judgments. His approach differs from the usual decision analysis process. In a conventional analysis, we assume that that a subject matter expert (SME) can be identified for each key variable. Then, a skilled interviewer carefully elicits the SME’s judgment through an interview process.

Hubbard takes a different approach. People familiar with the type project are assembled and given calibration training. Becoming calibrated might take perhaps a half-day of practice exercises and feedback. Basically, being “calibrated” means that one can consistently provide judgments of 90% confidence intervals that avoid the “overconfidence” bias. The book provides several example quizzes for the reader to self-assess. Example questions (from page 56; answers below):

1. In 1938 a British steam locomotive set a new speed record by going how fast (mph)?
    

2. The ancient Romans were conquered by the ancient Greeks (True or False).  Confidence that you are correct (Circle one) 50%  60%  70%  80%  90%  100%.

In a quiz in the form of question 1, we would expect a calibrated person to have approximately 90% the answers correct. In the question 2 format, the average of the circled percents should approximate the fraction of correct True/False answers.

Even though I was well-aware of the overconfidence bias, I still performed poorly on the self-assessment tests (history was never my strong subject!). Of course, the questions for a technical group would be crafted from topics within the area of interest. Whether (a) being an expert in the quiz subject matter or not and (b) being told in advance that most people tend to be overconfident about the quality of their knowledge doesn’t seem to affect the overconfident bias. Practice and feedback are the antidotes to this bias problem.

Hubbard’s training and consulting examples are engaging. It has been years since I’ve devoured a technical book so thoroughly. While the reader will pick-and-choose methods of most interest, the “measurement” topic is well-covered.

The book contains many shortcuts and heuristics for rapid problem-solving.  For example, a quick "value of perfect information" calculation for a variable may show there is no benefit to improving the measurement (judgment) about that variable. People often measure what is easy, rather than what is important.

Many people never attempt to quantify what they think are intangibles, such as risk of catastrophe, lost of corporate reputation, and security. Yet, most people are able to provide credible, quantitative judgments after brief "calibration" training .

Four Useful Measurement Assumptions (p. 31):

1. Your problem is not as unique as you think.

2. You have more data than you think.

3. You need less data than you think.

4. There is a useful measurement that is much simpler than you think.”

How to Measure Anything is well-written and carefully edited. The companion Web site, www.howtomeasureanything.com, offers additional calibration questions, several calculation spreadsheets, and additional information.

A sampling of topics in the book includes:

• Modeling and Monte Carlo simulation

• Designing experiments for measurement

• Decomposition in project modeling; also for better understanding and measuring variables

• Heuristics for obtaining simple statistics

• Value of perfect information, for screening which variables are worthwhile measuring

• Bayes’ rule (because we almost always have some prior information about the subject of the observation)

• Cognitive biases

Persons reading this book will be the better for it.  Doug’s e-mails end with this well-deserved recognition: “Since it was published in July 2007, Doug Hubbard's book "How to Measure Anything: Finding the Value of Intangibles in Business" has been the best selling business math book on Amazon!”