Taking apart a saved thirty percent
The number in the board pack usually cannot survive three questions. Here are the three questions, and how to be ready for them.
A claimed improvement is only evidence if you can say where the number came from, what it is being compared against, and what else changed in the same period. Capture the baseline before the project starts, because it cannot be reconstructed afterwards.
Question one: where did the number come from
Take a claim that a new system saved thirty percent of the team's admin time. Ask how the before figure was measured. Very often the answer is that somebody asked the team to estimate, after the new system was already in use.
That is not a measurement, it is a recollection formed while looking at the answer. It may be roughly right. It cannot settle an argument, and it will not survive a finance review.
Question two: compared to what
Compared to last month is a comparison against a season. Compared to last year is a comparison against a different business. Neither is wrong, but the choice changes the number, so the choice has to be stated.
The comparison also has to include what else happened. A system that launched in the same quarter as a hiring round, a price change and a marketing campaign has three co-authors for any improvement it claims.
What to capture before you start
This is the whole discipline, and it takes a day. Two weeks before the build begins, write down what you will claim later and how you will know.
- The two or three numbers the project is meant to move, named in advance
- Their current value, measured rather than estimated
- How they were measured, in enough detail to repeat exactly
- What else is planned in the same period that could move them
- The date you will read them again, agreed with whoever will ask
Measures that survive scrutiny
Prefer numbers your existing systems already record: orders processed, tickets closed, days to invoice, error rate, time from request to fulfilment. They are unglamorous and they are already being collected consistently, which is what makes them usable as a baseline.
Be careful with satisfaction scores and self-reported time around a launch. Both move on the novelty of a new tool and drift back, so a reading taken in the first month tends to flatter the project.
- Volume: orders, tickets, applications processed
- Elapsed time: request to fulfilment, work done to invoice sent
- Error rate: corrections, returns, rework
- Cost per unit of the thing you actually produce
If there is no baseline
Say so, and make a smaller claim you can support. Process time per order measured this month against a sample of last year's records is defensible. Thirty percent of everybody's time is not, and asserting it invites a challenge that discredits the parts that were true.
The habit worth building is deciding what would count as failure before starting. A project with no stated failure condition cannot succeed in any way that is checkable.
What to take from this
- 01Measure the baseline before the build, because it cannot be recovered later
- 02Name the comparison period and everything else that changed in it
- 03Prefer numbers your systems already record over self-reported time
- 04Decide in advance what would count as the project failing
Ask the question directly
- What if the improvement is real but hard to measure?
- Then measure something adjacent that moves with it and say that is what you are doing. A stated proxy is honest. An unstated one gets discovered by whoever is reviewing the claim.
- How long after launch should we read the numbers?
- Long enough for the novelty to pass and a full cycle of the business to run. For most operational systems that is one to three months, and it should be agreed before launch rather than chosen afterwards.
- Should the supplier report on this?
- They should supply the data. The reading should not be theirs alone, for the same reason you would not ask a supplier to sign off their own invoice.
Related answers
Growth transformation
Organic accounts, reputation, lifecycle and conversion run as one programme, with fractional leadership above it for the companies that need the seniority before they can justify the salary.
AI transformation
Most AI projects fail because they add a chat box next to the problem instead of removing the step that costs the hours. We start from the workflow, and from the systems and the data underneath it.
Bring the decision you are stuck on
A call, forty five minutes, no deck. We will tell you if we are the wrong firm.