Everyone who has lived within the world of analytics has seen cases where different parts of a business have made use of slightly differing definitions of core business metrics. Sometimes these differences lead to only minor and non-material disagreement. At other times, the differences in definition can cause massive divergence of reported results and related actions taken. Organizations must ensure that where differences exist in definitions those differences are either reconciled or clearly labeled and articulated to provide the proper context.
IT’S JUST A LENGTH OF STAY, RIGHT?
I recently came across a terrific example when a healthcare provider was discussing the seemingly simple issue of determining the length of stay that patients have at a hospital. Not only is length of stay important in and of itself, but it is also a component of other important metrics such as cost per stay.
Given its importance, one might assume that the formula for length of stay was standardized within the organization. But, when there are a number of hospitals acquired over time that still run in a largely autonomous fashion, it is easy to have different definitions creep in. Worse, each definition can be defended and there may be no true “right” answer.
To illustrate, let’s look at three examples of how “length of stay” might be defined for a patient that is admitted at 8:00 PM Monday and released at 8:00 AM the following morning.
Each calendar day a patient is there counts as a day. In this case, the patient would have 2 days in his stay since he was there part of Monday and Tuesday.
Each time a patient crosses midnight, it is counted as a day. In this case, the patient had a single day stay.
Hours as a percentage of a day. In this case, the patient was there for 12 hours and so would be counted as a ½ day stay.
In looking at the above, all the definitions can be defended. At the same time, there is a 4X difference between the low and high method (2 days vs 0.5 days). As stays increase in length the percentage differences will shrink. But, there are a lot of short hospital stays. Different hospitals could have vastly different reported results given the exact same business conditions.
THE COUNTER-INTUITIVE CASES FOR EACH LENGTH OF STAY COMPUTATION
While all three of the definitions above can be defended, they also lead to cases that can seem a bit silly, if not downright wrong.
In the case of counting calendar days, a patient who is admitted for just an hour from 11:30PM – 12:30AM gets counted as two days. That certainly seems inflated.
In the case of counting patients who cross a midnight barrier, the formula falls apart for someone admitted at 12:01 AM and released at 11:59 PM. In this case, the situation is counted as no hospital stay at all even though the patient was there for 23 hours and 58 minutes!
The percentage of hours method stays logically consistent at any length of stay. However, it leads to almost every patient having a stay with a fractional day component. This is harder to work with and goes against the tradition of giving each patient a specific length of stay that is a whole number.
The take away is that it is easy to both defend and poke holes in each approach to measuring length of stay. Understanding the differences and when each definition is being used is critical for those using the data.Using differing definitions in different contexts isn’t inherently bad as long as those being fed the information are well informed as to what they are looking at and why they are looking at it.
IT’S JUST A MEMBER COUNT, RIGHT?
For another example of divergent results, consider a health insurer keeping track of how many members it has. The member count then flows into a wide range of other metrics such as claims per member, premiums per member, and more. How might we decide who counts as a member for a given month?
Anyone who is a member at the start of the month is a member. A new member mid-month who has claims would not be counted as a member, but yet does have legitimate claims and premiums!
Anyone who is a member on the last day of the month is a member. Like option 1), a member who dropped mid-month and had claims prior to dropping would not be a member but would have valid claims and premiums.
Anyone who is a member for any portion of the month is a member. In this case, big employers coming in and out very early or late in the month can skew results as a large number of members would have limited time to actually make a claim but would count as a full member.
As with the hospital length of stay example, it is easy to see how unusual cases can make a seemingly reasonable definition fall apart and lead to counter-intuitive results.
PROTECTING YOUR ORGANIZATION
Few people find discussions about how to define metrics interesting. However, as the prior examples illustrate, it is critical to have those conversations. When different parts of an organization diverge in their approaches to defining key business metrics, the differences can lead to substantially different decisions being made and divergent business practices arising.
As also illustrated in the examples, even solid logic can have cases where it breaks down. It may not be possible to avoid those breakdowns completely, but to simply understand when they occur and choose definitions that minimize the damage that the unusual cases inflict.
When is the last time your organization took the time to really examine the strengths and weaknesses of the metric definitions being used on a daily basis? It might just be time to force the conversation again.