I’ve written elsewhere about the value of grounding your metrics program in semantics.1 This article goes well beyond that. We are just beginning to see a pattern of evolution that seems to be enabled, perhaps even encouraged, by the collapsing of silos into a data-centric knowledge graph. It is still early days, but the patterns are beginning to emerge, from our client work as well as our internal project to make Semantic Arts a showcase for what is possible. The serendipity moment occurred when I saw an outside presentation on a metric we were actively evolving (Accounts Receivable aging).
The plan of the paper is first to reexamine why we have metrics in our enterprises. From there, we lay out an evolutionary progression for metrics. Finally, in part 2, we wrap up with a detailed case study based on aging accounts receivable.
Why We Have Metrics in Our Enterprises
Most enterprises have a rabid fascination with measuring things, often driven directly or indirectly from this quote.
“If you can’t measure it, you can’t manage it.”2
— misattributed to Edward Deming
As John Hunter points out in the footnote here, this isn’t what Deming said or meant. Deming did understand the value of data, for sure, but he also understood there was much more to managing than merely measuring. But it’s this attitude we’re saddled with, and as a result, most enterprises have more metrics than employees.
We have metrics in our enterprise to attempt to manage them better. If you peel it back a bit, you find that “management” is really about influencing behavior. Some metrics aggregate a lot of behaviors. “Net income,” one of the most widely used metrics, covers a range of behaviors from customers loyalty to raw material substitution. The OKR (Objectives and Key Results) movement is all about using key results (metrics) to change behavior (toward the goal of the objective).
In this paper, we’re going to look at traditional enterprise metrics and consider how the lens of behavior change can affect the way the metric and its usage evolves.
An Evolutionary Progression for Metrics
In this section, we outline a progression we’ve seen in several settings and provide a few simple examples. This is to set up the framework we will use for the detailed case study.
The progression is:
- Measure and superficial clean up
- Formal semantics
- Calibration
- Prescriptive action
- Root cause
- Better metrics
Measure and Superficial Clean Up
Most metrics start with just wanting to measure something you’re already doing. Often, you start with data you think you have. Maybe you want to measure customer acquisition cost or sales per square foot, or number of meaningful interactions per salesperson per day.
Inevitably, the act of measuring shines a light on the fact that you either don’t have the data you thought you did, or it’s mostly crap. There are often several months and several cycles of getting to the bottom of why you have poor data quality (at least for this purpose, prior to starting the metrics project you thought the quality was just fine).
Formal Semantics
At some point, the data is at least clean enough as to not be embarrassing to share. At this point, people often start asking harder questions about what the data really means and where it came from. This is especially true in large BI (Business Intelligence) organizations where there may be hundreds, or in some cases, thousands of analysts working in parallel on business metrics and dashboards. Inevitably, this leads to managers in different divisions wanting to compare metrics and realizing they are comparing apples to oranges.
This is where formal semantics and the so called “headless BI” can help a lot. By taking the definition of the metric out of the graphics tool and putting it in a shared place, you end up with shareable and comparable metrics. They still have a way to go before they are optimized, but at least they can be compared.
Calibration
Up to this point, the data is merely “clean” and comparable. It seems odd, but often it takes a while before people start asking if it is right, or if it makes sense, or if it even measures what we hoped to measure.
One of our recent metrics was future cash predictions. You might think your accounting system would be good at this, but most are not. Most of the important inputs to this prediction are elsewhere. We gathered up everything we thought we knew about our next six months of activity (into a spreadsheet naturally) and began reporting on this number.
It was at that point we entered into what we called the “calibration phase.” When you have a million dollars in the bank, your projections could be off by many hundreds of thousands of dollars, and you can still make a $400K payroll. But we recognized that we wouldn’t always be this flush, cash flow seems to come and go. At that point, we made the observation that an altimeter can be off by hundreds of feet when you are at cruising altitude, and no one will notice, but if its off by hundreds of feet at best, you’ll have a very hard landing.
Some of the calibration activity we worked on at this point were:
- Realistic-ness of predictions — One of our early projections contained wildly alarming predictions. We could have just accepted that at face value (and either extended the line of credit or started massive layoffs), but instead we chose to question the veracity of the measure. (Turns out, there were many systemic problems with assumptions and logic.)
- Looking for false positives and false negatives — In this case, a false negative is something that should have shown up in the measure, but didn’t. A false positive is a signal without an underlying cause. The former requires you look outside the system. Look for things that exist in the world, that should throw a signal and then look for the signal. If it’s not there, you have a false negative. The false positive is a bit easier, in that you don’t have to go out of the system to find them, but you may have to go outside the system to find out what caused the false alarm. It usually takes a lot of cycles to get false positives and false negatives down to a manageable level.
- Checking our predictions — Most prediction systems are point in time. This is your prediction of cash levels every week for the next three months. In three months, you will have a new prediction for the next three months. What you won’t have, unless you plan for it, is feedback on whether your prediction was any good. You must keep your predictions somewhere for this to be effective. The best time to check your prediction is when the final reality arrives. When week 12 finally arrives, you will know exactly how much cash you have on hand. What will improve your prediction is to look at what was your cash prediction for week twelve in each of weeks 1-11.
- Calibrate at 0 — Inventory cycle counters know something we can all learn from: Things are easier to count and easier to reconcile when they are at 0. When the inventory system tells you that you have 2 items on hand, but you go to the bin and there are none, you know you have a problem. This is a false positive from above; the system thinks you have inventory that you don’t have. Note ,if the system thinks you have 30 on hand, but you really only have 28, you are far less likely to notice. Conversely, every time the system signals on hand has gone to 0, it’s a good time to count. If you find any you have a systemic problem. Note the relationship to our altimeter analogy.
This whole calibration phase is one of the most important in the life cycle of the metric and often takes the longest. We often find this sends us back to the semantic definition (we thought we were clear on the inputs to the metric, but often reality tells us otherwise).
The power of calibration is what it sets you up for. Until you have a highly calibrated metric, you are reluctant and slow to take action. Traditional metrics are mostly consumed by analysts. When the metric is calibrated, we are ready to turn them over to front line actors.
Prescriptive Action
This is where we use the metric to drive immediate action. If your metric systems if throwing false alarms all the time you really shouldn’t deploy notifications and especially not automated actions. Users of healthcare systems are all too familiar with “alert fatigue.” EMRs (Electronic Medical Record systems) typically throw off so many false positives that providers learn to ignore them (which is not the intended behavior).
If the signal is strong, with few false positives and few false negatives, it is a good candidate for notification and action.
It is at this moment that we begin to realize we may have the wrong user. Up to this point, the user was often some sort of analyst. Analysts are tolerant of bad data because they are doing research and figuring out what is really going on. We are finding, when we change the target user for our notifications, sometimes we need to change the metric again. As we will describe in the case study, sometimes at this point, we realize there is a different behavior we want to modify, and it may change again some details of the metric.
Root Cause
Switching to action mode gives us a new dilemma. We want to use this signal to take action, but what action? Often, the action is different depending on the root cause of the signal. But there is usually nothing in the signal to suggest the root cause.
This is where abduction comes in. Not as in alien abduction, but as in the third in the triad of induction, deduction and abduction. Deductive reasoning is where we start with well accepted rules and apply instances to them to reach new conclusions. Semantic inference is deductive. Induction is where rules come from a large set of instances. Most science is inductive, research is gathered, and conclusions are drawn. The intent with inductive reasoning is to have a sufficient number of instances that the conclusions are statistically valid.
Abductive reasoning comes into play when we don’t have a large set of instance data to drive a statistically valid conclusion, but we need to proceed anyway. Abductive reasoning is generally taking a cognitive leap and proposing a likely cause of the observation (in our case the signal). A very good abductive hypothesis is testable.
As we will see in our case study, we want to take our signal and then apply what is usually a small amount of contextual information to propose a testable hypothesis for the cause of the signal. If it is easily testable, then we should test it. By working this feedback loop our abductive muscles get stronger.
Better Metrics
Finally, as strong signals and good abductive postulates start to take over, we find that often we’ve moved on and there is a higher-level metric that makes more sense to track. At one point, we were going to call these “meta metrics” because they seem to operate at a higher level, but we realized that this was going to get easily conflated with meta data, so let’s just call them “better metrics.”
Classically, cash flow management was a way that small companies managed their finances. As in our above example, cash management never goes away, but most firms find that managing the firm on an accrual basis is more actionable for day-to-day management, and cash planning is a secondary concern.
That was a generalization of an evolution we are seeing in many scenarios. We understand it is a bit abstract. You will need to work through the details in your own examples. Part 2 works through the six phases in a common case study that may be familiar to many readers: aging accounts receivable.
Summary
We have laid out a progression that we think metrics in an enterprise could, and often do evolve through, as the firm becomes more data-centric and more capable of shifting the response to the metric from an analyst closer to someone who can affect the behavior change the metric is trying to influence.
In the next installment, we will work through this sequence in a single case study, the evolution of aging accounts receivable.
[1] The Data-Centric Revolution: Headless BI and the Metrics Layer https://tdan.com/the-data-centric-revolution-headless-bi-and-the-metrics-layer/29533
[2] https://deming.org/myth-if-you-cant-measure-it-you-cant-manage-it/
