In this article

A monthly payroll tool has little reason to bring its customers back every day. A search tool can serve someone well by ending a session quickly. An automated monitoring service may do its best work while nobody is looking at it.

These products expose the weakness of treating activity as a universal measure of engagement. More visits can mean more value, but they can also mean repeated attempts to finish the same job. Before setting a target for activity, describe the behavior that would make the product useful at its intended cadence.

A workable engagement definition has to recognize a successful task even when an improvement removes clicks, sessions, or minutes from the experience.

Put a customer outcome behind the event

“Report exported” is an observable event. “The team received an accurate report in time to make a decision” is a customer outcome. The event may be a useful approximation, but the gap between them matters.

An exported file might be empty. It might require extensive repair in a spreadsheet. It might arrive after the meeting. Instrumentation that counts all three as successful engagement rewards the product for getting work out of its interface, regardless of whether the customer can use it.

Start with a plain description of the completed job. Then identify what the product can observe directly and where additional evidence is needed. For an expense tool, submission is visible, but later rejection or correction may tell you whether the claim was usable. For learning software, finishing a lesson says less about understanding than performance on a relevant task.

Decide what a better experience means before selecting the event that will stand in for it. Google Research’s HEART paper describes this progression from goals to observable signals and metrics.

Write the limitation beside the metric definition. “Claim submitted without a validation error” is more honest than “accurate claim” when your system cannot observe accounting accuracy. Better naming prevents a proxy from acquiring authority it has not earned.

Let the work determine the clock

Frequency belongs to the task. A weekly planning routine, a quarterly review, and an occasional insurance claim should not share an arbitrary daily target.

Choose a window in which a customer reasonably has an opportunity to complete the job. For payroll, the pay cycle provides a natural reference. For an expense product, the opportunity may depend on whether someone incurred reimbursable expenses. A person with none this month should not automatically count as a failed user.

That distinction becomes harder when the product itself affects how much work enters the system. An expense-capture feature may help people record claims they previously forgot. Defining the population only as people who uploaded a receipt after the change would select on behavior the feature influences. Preserve a suitable stable comparison population and examine task opportunity separately.

Cadence can vary within one product. An employee submits expenses occasionally; a finance reviewer handles them daily. Their activity should be interpreted against those roles. Averaging both into one engagement target can encourage unnecessary employee reminders while hiding an overloaded reviewer.

Read the effort that surrounds completion

Consider an illustrative expense workflow, with invented numbers for comparison:

MeasureManual matchingProposed automatic matching
Sessions per eligible employee42
Claims submitted without detected errors per eligible employee0.80.9
Median active minutes per completed claim148
Submitted claims later returned for correction6%5%

The second column of results would look worse on a sessions dashboard and more promising on a task dashboard. The figures are not research findings or proof of an effect. They show why the interpretation needs the completed job and its quality.

Even this view is incomplete. If automation saves the employee six minutes but creates ten minutes of review for finance, the product has shifted work. Examine the people who receive the output, especially where an account-level result combines several roles.

Measure effort carefully. Elapsed time includes interruptions, waiting, and time away from the screen; it is not identical to active work. A shorter median among completed tasks can also conceal people who never finish. Read completion and effort together rather than dropping unsuccessful attempts from the story.

For some products, time is part of the desired experience. A reader may want to spend an evening with a book. Even there, extra minutes cannot establish satisfaction on their own. Difficulty finding a passage can extend the session too.

Use unusual activity as a research lead

Very high activity deserves investigation. It may identify expert users doing more work, or people caught in retries. Look at what precedes and follows the activity before celebrating it.

Microsoft Research describes a button-change experiment in which an initially large click increase diminished over time; the team connected the pattern with confusion and repeated clicking. The example is a reminder that an engagement event can record a user’s attempt to recover from a change.

Sample recent successful and difficult tasks. Ask customers to show what happened after your recorded endpoint. Did an export become useful immediately, or did it begin another hour of cleanup? Include people with low activity where possible; interviewing only frequent users leaves the meaning of absence unexplored.

Surveys can add context, but their timing and audience shape the answer. A confidence question after reviewing a result is different from a general satisfaction prompt during setup. People who respond may differ from those who leave. Keep those limitations attached to the interpretation.

Recognize useful work that moved out of the interface

Automation changes the relationship between visits and value. When a scheduled job replaces a manual workflow, the old completion event may disappear even though the underlying job still happens.

Update the measurement before rollout. Observe whether the job ran, whether its output was delivered, and which quality signals are available. Keep manual and automated paths comparable at the outcome level. Do not require someone to open the application just to produce an engagement event.

This is also where the definition should be allowed to evolve. If customers no longer need to inspect every successful run, a falling visit rate may be expected. A rise in exception handling could be healthy when it reflects newly visible problems, or harmful when the automation creates them. Investigate the source of that work.

A useful test of an engagement metric is to imagine a better version of the product that completes the same job with half the interaction. Would the metric reward it? If the dashboard would call that improvement a decline, change the definition before asking the team to optimize it.

Sources

Something we should clarify?

Suggest a correction