In this article
A reminder earns clicks, and some recipients return to the product. That does not tell you how many would have returned without it. To answer the question, withhold the reminder from a comparable, randomly assigned group and measure the same useful behavior for everyone.
The design should be narrow enough to operate correctly. Start with one reminder, one eligible population and one return outcome. A test of an entire messaging program needs a broader holdout that persists across the program; a single unsent email cannot answer that larger question.
Write the decision before configuring the send
Consider a fictional personal budgeting app. The team wants to know whether one reminder helps established users complete another weekly review. Eligible users have completed a review before, have not completed one in the previous seven days, and are permitted to receive the reminder. This is a proposed teaching example, not a report of an actual experiment.
Define the primary outcome as completing a valid review within 14 days after assignment. A login, email open or click does not qualify. Count each assigned person once, even if they complete several reviews. Record additional reviews separately if they matter to the product decision.
Write down the treatment exactly: one reminder with a specified destination and no follow-up sequence. The control receives no such reminder. Ordinary service messages continue under the same rules in both groups. Do not withhold security or task-critical communications to make a growth test cleaner.
Before launch, choose the enrollment period, follow-up window, minimum effect worth acting on and sample size using a plausible baseline. Set the analysis and stopping method in advance. The small worked result below is arithmetic for interpretation, not a sample-size recommendation.
Assign people once and preserve that assignment
Create an eligibility record before the reminder is sent, then randomly assign each eligible user 50:50 using a stable user identifier and an experiment-specific seed. Store the assignment and timestamp. A person who becomes eligible again during the test keeps the same assignment and receives no second experimental reminder.
Keep the eligibility calculation identical in both arms. Do not enroll treatment when an email is sent but enroll control when a user opens the app. Those rules would select different populations before outcomes are measured.
Microsoft’s experimentation guidance discusses identifier stability and the need to choose a randomization unit suited to the product. Its warning about effects between collaborators matters here. The example uses personal accounts because one person’s reminder is unlikely to change another person’s review. A shared household or team product would need a design that addresses that dependence; treating every member as independent could misstate uncertainty.
Inspect assignment counts and known pre-treatment characteristics for implementation problems. Investigate missing identifiers, duplicate assignments and unexpected traffic ratios. A balanced-looking dashboard alone cannot prove randomization works.
Know what your messaging tool is holding out
Customer.io’s holdout documentation describes a message variant that is generated internally but not sent to the recipient. It recommends using a conversion goal because an unsent message cannot receive opens or clicks. This provides a practical mechanism for a single-message test.
Check where the split occurs and whether the person can enter another branch or campaign that sends an equivalent reminder. A control member who receives the same nudge elsewhere weakens the intended contrast. Log competing messages and decide before launch which campaigns must be suppressed equally or isolated for the test.
The implementation must also preserve the control’s assignment time. “Completed within 14 days” needs the same starting point for people who receive no delivery. Do not use email delivery time for one arm and a later campaign event for the other.
Read the result with the assigned population intact
Suppose the illustrative test enrolls 2,000 people, with 1,000 assigned to each group. The treatment sends successfully to 900 people; 90 people click. Within their full 14-day windows, 180 treatment users and 150 control users complete a valid review.
| Measure | Reminder assignment | No-reminder assignment |
|---|---|---|
| Assigned users | 1,000 | 1,000 |
| Users completing a review | 180 | 150 |
| Completion rate | 18% | 15% |
The observed difference is 3 percentage points, equivalent to 30 additional completions per 1,000 assigned users in this invented result. Relative to 15%, the point estimate is 20% higher. Report the absolute difference first so the effect’s practical size is visible.
Keep the 100 treatment users whose message was not delivered in the primary assigned-group comparison. Delivery is part of implementing the reminder policy. Removing those users after assignment can break the comparison. Delivery and click rates remain useful diagnostics, but a clicker-only analysis selects people partly through their response to treatment.
Sampling uncertainty still matters. Using a simple independent-binomial normal approximation, the illustrative difference has an approximate 95% interval from −0.25 to 6.25 percentage points. The calculation is the difference plus or minus 1.96 times the square root of the two estimated proportion variances. It includes zero. This example therefore does not establish a positive effect at that conventional confidence level, despite the attractive relative point estimate.
That approximation assumes independent users and a fixed analysis. It is not suitable unchanged for clustered assignment, repeated observations or a sequential stopping rule.
Measure the cost of the return
A reminder can produce useful activity while also causing unwanted contact. Predefine guardrails such as notification opt-outs, complaints or support contacts, and measure them over comparable windows.
Suppose, again illustratively, that 12 treatment users and five control users opt out during follow-up. The observed difference is 0.7 percentage points. That is a signal to interpret with its own uncertainty, not permission to label every extra opt-out as caused by the reminder. The product decision should consider both useful returns and the cost of obtaining them.
Do not redefine success after looking at a favorable slice. If the primary result is inconclusive, a high click rate does not rescue it. A segment discovered afterward can suggest the next test, with fresh assignment and a stated question.
Decide what the test has earned
At the end, separate three issues: whether the assignment and logging were trustworthy, what range of effects remains plausible, and whether that range supports the business decision. An underpowered test may leave the team uncertain. Extending it opportunistically until a preferred number appears changes the analysis problem.
A successful short-window reminder also does not prove durable retention. If the team’s real concern is repeated monthly use, define that later outcome and wait for it. Use the retention-window guide to keep immature observations out of the comparison.
The useful result is a decision about this reminder for this population. It might justify keeping the message, changing the proposition or stopping a send that adds no demonstrated value. That is a stronger foundation than treating every return after an email as a return caused by the email.
Sources
Something we should clarify?
Suggest a correction


