Simple metrics

A pilot with no number is an anecdote. A pilot with fifteen charts is another anecdote, slower. In an SME four figures you already understand, taken the same way before and after, are enough.

The temptation is to measure what the tool shows (prompts, tokens, “conversations”). That measures the vendor. You care whether last week’s work changed.

This article sets a minimum board, how to take a baseline week, and what to do when the number does not move.

The four numbers

• Hours (or minutes) per unit of work. Per invoice, per ticket, per order. Taken with a clock or a one-week sample, not with a feeling.
• Errors that reach the customer or the tax office. Duplicates, bad VAT, missed lead times. If you were not counting them, start this week: no “before”, no pilot.
• Human accept rate. Share of outputs sent or posted without a rewrite. Low at first is normal. Zero at day 30 means the flow is badly designed.
• Cycle time. From input arriving to validated output leaving. That is what the customer feels, even if there is “AI” inside.

How to take the baseline week

Before you switch anything on, one sample week: ten or twenty real units. Same person, same document type. You log minutes and errors. That is the “before”. Without that week, any improvement is an impression.

You do not need a BI tool. A sheet with date, id, minutes, error yes/no, and later accept rate. If the sheet is abandoned, so is the pilot: the metric is part of the habit.

What does not count (even if it looks nice)

Workshop satisfaction. Seat count. Tokens. Screenshots. Dashboard visits. “The model is smarter”. None of those decide whether a second flow opens.

A metric only the vendor can compute also does not count. If they vanish, you must still be able to measure with your sheet.

If the number does not move

First: is it used? If it is not opened, that is training or scope, not the model. Second: is everything rewritten? Then the template or the sources are wrong. Third: it is used and time does not drop? The bottleneck may be the next step (ERP, signature, email).

At day 30 you decide: cut, train again on real cases, or stop. Continuing “because we already paid” is how tool graveyards start.

Showing the number in a meeting

One slide: baseline week, week 4, sample size, and a sentence of what you did. No word clouds. If you need a method note (how minutes were taken), it should fit in a paragraph.

Kit Consulting and training justification travel better with this board than with a diploma. A reviewer understands “minutes per invoice, n=40” better than “we trained on generative AI”.

Sheet template (columns)

Date, document or ticket id, minutes, error yes/no (and one word: VAT, duplicate, tone, lead time), accept (as-is / rewritten / discarded), one-line notes. Nothing else in month one. If you add twelve columns, nobody fills it.

Friday, five minutes: average minutes, error count, accept rate. That is pasted in the team channel. You do not wait for the month-end committee to know whether the pilot is breathing.

When the second flow is born, another sheet. Do not mix invoices and tickets in the same rows: the number stops meaning anything.

A sample size you can defend

Ten units already show a disaster. Twenty let you compare weeks. Fifty, if the volume exists, close the “it seems to me…” debate. Under ten, the number is an anecdote: say so and do not take it to the committee as proof.

Do not mix types (clean invoice + till receipt + credit note) in the same average. Split them or the “success” will be an artefact.

Publish the number in the same place every Friday, even if it is ugly. A metric hidden in a mail to leadership does not change the habit. The one the team sees does: they correct, ask, or demand a cut. A secret board is theatre for a committee, not a pilot.

If on Friday nobody has filled the sheet, the metric is not the problem: the habit does not exist. Do not “estimate” minutes by eye to avoid a gap. Leave the gap visible. The empty cell teaches more than an invented number, and it avoids the committee theatre this article is trying to cut.

Frequently asked questions

Do we need a data analyst?

No. You need someone who fills the sheet and an owner who looks at it every Friday.

Do we also measure perceived quality?

Yes, as a comment, not as the king number. Perception without accept rate and hours is easy to game.

What if volume is low?

Then the pilot may not belong. A flow of four documents a month does not even pay for measurement. Go back to the matrix.

Next: mistakes to avoid and training. Request an assessment and we will return the board for the first flow.