Measuring AI productivity in your MSP: why saved time proves nothing

Measuring AI productivity right now feels like someone proudly announcing he is cycling faster, without checking whether he is heading for the right village. Your engineers are happy with Copilot, your service desk closes tickets in half the time, and still you do not see it at the bottom of your P&L. That is exactly the question almost every MSP owner is chewing on at the moment.

Ask any MSP owner what AI brings his business and you almost always get the same answer: "people are faster". Fine. But faster at what? And what happens to the time that is freed up? Does it disappear into extra coffee, into more meetings, or into new revenue? In nine out of ten conversations we have, nobody can really say.

The problem is that most MSPs measure AI at task level. How many seconds shorter a ticket is. How many minutes someone saves on a quote. Those figures are quick to collect and they feel good. But if the decision making behind it stays slow, if your sales process still takes six weeks, if your onboarding still takes a month, then you have put a fast task inside a slow chain. The bottleneck has simply shifted a few metres.

On top of that comes a second trap. We hear it constantly from mid-sized clients: "we need to do something with AI", without anyone able to say what exactly needs solving. So as an MSP you start running demos on a problem nobody has named out loud, and then you measure whether the demo did anything. That is not measuring. That is hoping, with a dashboard next to it.

Measure the process, not the task

What does work in practice is simple but uncomfortable: you stop looking at time saved per action and start measuring at process level. Take one process, first line support or lead qualification for example, and measure four things: throughput time from start to finish, quality of the output, client experience and revenue. Only when those four move together do you have a real gain.

An example. In our own customer service we now resolve 50 per cent of cases autonomously. That is a nice number, but it only means something once you put three other figures beside it: what happens to CSAT on those cases, what is the effect on second and third line, and what does it cost in tokens versus what it saves in salary. Without all four together you do not know whether your business is better off or merely different.

The same goes for the sales end. You see lead qualification getting much sharper, BDR work running surprisingly well through voice agents, and demos going out faster. Good. But if conversion per demo does not rise and your deal size stays the same, you have only lowered the cost base of sales and widened your marketing funnel. Not a bad thing, but something very different from "AI makes us more productive".

The core question we ask in every mapping session is this: which KPI is going to move because of AI that genuinely matters to this business? For a scale-up that is speed of innovation. For a traditional MSP it is margin per engineer. For a security player it is time between signal and response. Only once that KPI has been named may anyone start on an AI use case. Not before.

Follow the pain first

Internally we hold to an order that almost nobody keeps: first eliminate, then simplify, then standardise, and only then automate. That is not a fancy framework, it is simply what works. Rubbish in, rubbish out, and if you speed up a bad process with AI you get a fast bad result. Or worse: a fast bad result nobody dares argue with because "AI said so".

A sharp example from our own bookkeeping. Every month we close the books and determine work in progress. Last year I had a solid argument with our finance man: his figure differed significantly from what I worked out in my head. Twelve years of the same spreadsheet, the same formula. It turned out Copilot had quietly adjusted the formula, because it thought that was smarter. We were missing a sizeable amount in the reporting. The greatest value of a human in an AI process is not speed, but the ability to keep saying "this is not right".

Three use cases that move

If we are honest, we currently see three places where AI structurally delivers something inside an MSP. One: first line support, where 40 to 50 per cent of cases can be resolved autonomously provided your data and governance are in order. Two: lead qualification and BDR work, where voice agents are now so good that clients on a call do not notice there is no human on the line. Three: code generation in context, where the model understands which platform it is working on, Salesforce versus Oracle versus Workday.

All three share one condition: there is a data question underneath. Data coverage, data labelling, data security, data ownership. This is also why data management is finally coming out of the boring corner. Where we used to flog a dead horse at clients, they now knock on the door themselves: data governance, we need to do something about it. Not because they suddenly enjoy governance, but because they realise their AI ambition will not get past a demo without clean data.

And then there is the token economy. At the moment Salesforce, for instance, has unlimited tokens, so everybody happily burns away. That will change. The Anthropics and OpenAIs of this world have to make money at some point, and then the conversation shifts from "what can AI do" to "what does this use case cost per month and does it return more than it burns". That is precisely where an MSP can specialise: FinOps for AI tokens, because no mid-sized client is going to work that out alone.

Measuring AI productivity in your MSP does not start with a dashboard, but with one honest question per process: which KPI should this AI use case move, and are we actually measuring it. Whoever has that conversation internally about his own business will have it effortlessly with clients later. Whoever skips it burns tokens, time and the trust of his team. Inside the MSP Late Night community, leaders swap every month which metrics do and do not work for them. Join in if you want to hold your own figures up against theirs.