Measure AI Productivity Without Counting Prompts or Logins

To find out whether workplace AI improves productivity, test one repeatable workflow against a baseline and a credible comparison. Measure acceptable output per unit of total effort, then check speed, corrections and employee experience. Prompt counts and logins show exposure to a tool; they do not show whether the finished work improved.

The unit of analysis matters. A faster draft can leave a reviewer with more errors, while a higher case count can reflect easier incoming work. Keep the task mix, quality threshold and labor included in the denominator visible before drawing a conclusion.

Define one workflow and its unit of output

Choose a workflow with a clear start, finish and quality standard. Customer-support case resolution is an illustrative example: a request enters the queue, an employee responds, and the outcome can be checked against the same service standard in both groups. Separate routine requests from difficult ones so a change in case mix does not appear to be an AI effect.

Write a small measurement worksheet before the pilot starts:

• Eligible work: Define the requests included, exclusions, case difficulty and what counts as a completed resolution.
• Primary outcome: Use acceptable resolutions per paid staff hour, or another output measure with a fixed quality threshold. Include handling, review and correction effort.
• Speed and rework: Record intake-to-acceptable-resolution time, reopened cases, correction rates and reviewer minutes under the same rules for each group.
• Employee experience: Ask the same brief questions about workload, confidence and the ability to do meaningful work before and during the test.
• Context: Record access to AI, actual use, staffing, training and process or policy changes. Give each measure an owner and a data source.

Case records, quality reviews, staffing data and short surveys can supply these measures. The worksheet is tied to the work being done, so it does not require a particular vendor dashboard.

Build a baseline and a credible comparison

Measure the selected workflow before introducing AI, with the definitions and quality rubric intended for the pilot. Capture ordinary variation across case types, shifts and staff experience, and set the observation window in advance. Record any concurrent changes to staffing, service policy or demand.

The UK government’s AI impact evaluation framework recommends measuring real human behavior and real-world outcomes, using a control or credible comparison group where possible, and treating adoption as contextual evidence. It also lists quality-adjusted output, comparison with unassisted work and structured expert review among illustrative productivity measures.

Where feasible, randomly assign comparable cases or teams to AI access and the usual process. Both groups need the same intake rules, review standard and observation window; the AI group’s training and checking time belongs in its effort total. Choose an assignment unit that limits spillover: when employees share techniques across cases, assigning whole teams or shifts may give a cleaner comparison than switching the same person between conditions.

If random assignment is impractical, compare a similar team or a staged rollout with its own baseline. Examine the change in each group over the same period, while recording remaining differences in case mix and staffing. A before-and-after improvement in one team alone can also reflect demand, a staffing change or a new policy.

Count usable output, not activity

Microsoft’s workplace impact guide separates adoption from impact and examines output quality, task efficiency, business outcomes, behavioral changes and user sentiment. In the worksheet, AI use establishes that the intervention reached employees; productivity comes from work that meets the agreed quality bar for the effort spent.

Consider a hypothetical pilot with equal paid hours in each group. The AI-assisted group closes 120 cases, of which 84 pass review without revision; the usual-work group closes 100, of which 90 pass. Gross closures favor AI, but acceptable first-pass output favors the usual process. If paid hours differ, compare acceptable cases per paid hour rather than the raw totals.

Set a short quality rubric suited to the task: factual accuracy, completeness, policy compliance and whether the customer’s issue was actually resolved. Apply it to comparable samples from both groups, ideally without telling reviewers which work used AI. Keep serious failures distinct from minor edits, because an average score can hide costly exceptions.

Time the journey to an acceptable outcome, not only the first response. Corrections, reopened cases and reviewer labor can absorb apparent drafting gains. If quality improves but completion slows, report both changes rather than folding them into a single productivity claim.

Check where the effort went

Track where the work moves. Less handling time can coincide with more verification, overtime or a growing backlog. The OECD’s review of AI at work records worker concerns about increased work intensity and the collection and use of data, making workload and monitoring practices relevant safeguards alongside output.

Ask employees whether the workload feels manageable, whether they trust the material they must check and whether the work remains meaningful. Leave room for unexpected burdens in their own words. Aggregate responses sufficiently to protect individuals, and explain what usage data is collected and who can see it. Survey answers can help explain observed changes, but self-reported minutes saved should remain separate from recorded labor time.

Make a claim the design can support

A randomized comparison supports a claim about the effect of AI access on the tested workflow, staff and period, provided assignment and measurement held up. Report the quality threshold, hours counted, case mix and any material changes during the trial. Break out results by task difficulty or experience when an overall average conceals different effects.

A matched but nonrandom group leaves open the possibility that the groups differed in ways the records missed. A short test may also miss learning effects or delayed rework. Estimated time savings are not automatically cash savings: that claim needs evidence of a change in paid capacity, cost or service output. These boundaries matter when deciding whether the observed improvement warrants a wider rollout.