2.2 Characteristics of Good Measures

Measures should be easy to understand. The value of measures is helping managers track outcomes and performance. Thus, straightforward and understandable measures are better than complex or convoluted ones. A more specific definition of "simple" would be that few inputs are required to produce the same measure that could be produced with more inputs. However, just because a measure is simple does not mean that it is the right measure if it doesn't provide the meaning you are looking for. However, all other things being equal (ceteris paribus), a simpler measure is better.

An easily obtainable measure includes two parts: ease of collection and ease of calculation. 

Ease of collection: How easy are the measures to collect? Does it require human counting, or is it generated through automation? For example, at checkout stands, the process of running items over a scanner generates the information. Thus, the work itself is “sensed” by some sort of monitoring device. In baseball, the process is not automated, but it is not difficult to obtain counts for the different measures. Anyone with a basic knowledge of the game could collect measures that would cover all of the categories necessary.

Ease of calculation: How easily can you calculate useful information from the source data? Counts and simple ratios are easy to calculate. Some complex measures that take into account several variables are more complex to calculate. It is often beneficial to automate the collection and calculation tasks when possible; otherwise, it is unnecessarily arduous and costly, prone to error, and less likely to be done. This is especially true of measurements that are computation-intensive or that require an array of data to calculate.

Measures should be clearly defined so that they can be applied and evaluated consistently. Organizations need to establish and adhere to specific rules when collecting measures and ensure that these rules are being followed uniformly so that the integrity of the data is maintained. In baseball, for example, the total number of 'at-bats' a player has determines, in part, his batting average. Therefore, it is important to decide what constitutes an at-bat. When a player is issued a walk, is that considered an at-bat? How about when the result of the at-bat is a sacrifice? An error? These parameters have already been determined for baseball, and it is important that these measures be made uniformly for each player to maintain consistency.

Ideally, two or more qualified observers should arrive at the same value for the measure. Objective measurements will be referred to later when we discuss the need for administration of measurements. For the most part, baseball measurements are designed to be objective. Every once in a while, the official scorer will score something as a hit that could be considered an error or vice versa. These occurrences are rare and do not have a great impact on the outcome of the measurements.

The measure should actually reflect the property it is intended to. For example, how would you measure the success of a recording artist? If you wanted to measure commercial success, you could track the number of albums they sell and their total radio airtime. Conversely, if you wanted to measure a singer’s vocal ability, you may have a panel of experts assess whether the artist’s voice is consistently on key and the expanse of their vocal range. Often measures are designed to reflect specific process improvement constructs such as quality and efficiency.

Measures should be insensitive to insignificant changes in the process or product. Baseball measurements are robust. Unless the entire structure of the game was changed, the measurements would not be affected by small changes. For example, changing the height of the pitcher’s mound or the distance from the pitcher’s mound to home plate will not change the measurements used to assess a pitcher’s performance.

The characteristics of good measures (simple, easily obtainable, precisely definable, objective, valid, robust) are each desirable traits. However, it is unlikely that the "best" measures will incorporate each of these characteristics to the highest degree possible. Rather, you will likely need to make trade-offs among these traits. As a result, it is common to have multiple quantitative measures (i.e., represented on a dashboard) to depict a concept.

Consider the following two measures: (1) consumer purchase volume and (2) consumer satisfaction. Purchase volume is fairly simple; define the time frame and add up the purchases. It's easy to collect because you can simply query the operational database. It's precisely definable as long as you have the timeframe. It's objective because it's based on actual measurable behavior and not opinions. It's valid as long as you are recording purchases properly in your databases. It's also robust because if you change the time frame, you can still count on it being accurate. However, what does purchase volume actually represent? Does it indicate how much a consumer likes your product? Probably, but not always.

Now consider satisfaction. Is there a perfect measure that precisely defines exactly what satisfaction means to everyone? Not likely. Is purchase volume a perfect indicator of satisfaction? Actually, no. What if someone buys your product because they "have to?" For example, you may purchase Microsoft Office only because your instructor told you that you had to for this class. But in reality, you will switch to OpenOffice the moment you don't have to share files with other Microsoft Office users. So purchase volume is not always a valid measure of satisfaction. Okay, well, how about we ask a survey question like, "How satisfied are you with Microsoft Office?" and get respondents to select an option on a scale of 1 (very unsatisfied) to 7 (very satisfied). Does everyone interpret the number "2" on that scale to mean the same thing? Also, asking a survey question is MUCH more costly and difficult to collect than querying a database for purchase volume. Therefore, a smart manager will collect several measures that she or he believes to represent what a customer's satisfaction truly is.