Vice President · TheGreyMatter.ai
In brief
Measure the finished unit of work, including the effort required to trust and use it.
A draft produced in seconds can still take longer to use than a document prepared manually. The missing measurement is often the work between generation and acceptance: checking sources, correcting errors, resolving omissions, and deciding whether the result is safe to act on.
Define an accepted unit
Choose a unit the business already recognizes, such as a completed support response, an approved account brief, or a reconciled reporting packet. Write down its acceptance criteria. A response might need the right policy, accurate account information, an appropriate tone, and approval from a qualified reviewer. Generating text is an intermediate event, not the unit of value.
Separate quality from speed. Track important errors even when reviewers catch them before release. Also record cases where the workflow appropriately declines or escalates. A refusal that prevents an unsupported answer can be a successful outcome; a plausible answer that slips through review can be a serious failure despite its short completion time.
Use a fair baseline
Compare similar tasks under similar conditions. Record case difficulty, reviewer experience, available information, and interruptions. If the assisted group receives easier cases or more complete source material, the result does not isolate the contribution of the workflow. Use enough observation to include ordinary variation rather than selecting the best demonstration.
An illustrative example makes the calculation concrete. Suppose a manual task takes 30 minutes. An assisted version takes 4 minutes to prepare, 12 to review, and 6 to correct. The net time difference is 8 minutes, not 26. These are hypothetical numbers, not a customer result. Additional support or failure recovery could reduce that difference further.
Count costs at the right boundary
Include model access, infrastructure, indexing, maintenance, monitoring, and the people who handle exceptions. Separate initial setup from recurring operation so both are visible. Report cost per accepted unit alongside total spend and volume. A lower unit cost can coexist with a higher total bill when the organization produces more work.
Recovered time is not automatically cash savings. Ask what changed operationally: was a backlog reduced, a service improved, overtime avoided, or capacity redeployed? Keep the evidence for that change separate from the time estimate. Do not claim a revenue or valuation effect merely because a team completed drafts faster.
Keep the measurement useful
NIST’s AI RMF Core includes measurement within a continuing risk management process. A practical scorecard can therefore retain quality and failure measures alongside efficiency. Name who reviews the results, how often the sample is refreshed, and which changes require a new comparison. Model upgrades, source changes, and new users can alter the result.
The decision should follow the evidence. Expand a workflow that repeatedly meets acceptance criteria at a worthwhile cost. Narrow one that works only on simpler cases. Repair a source problem when reviewers spend their time correcting stale information. Stop a workflow whose correction burden or failure consequences outweigh its benefit. Useful measurement makes all four decisions possible, including the decision not to automate.
Sources and evidence
AI assisted editorial analysis. Sources checked October 10, 2026. Practical examples are proposed workflows, not reported customer outcomes or personal experience.
One useful next step: Choose one idea from this note and test it at the smallest scale that could teach you something this week.
