In this article
The retention chart improved overnight. No release shipped, nobody changed the reminder campaign, and the customers did exactly what they had done before. The analytics team had repaired an identity join.
That can be a real measurement improvement. It can also be an accidental rewrite of who counts as a customer. Before celebrating the new curve, separate the events people produced from the rules that attach those events to a person. A retention percentage can move because its returning population changed, its original cohort changed, or both.
This guide uses a deliberately small example so every person can be followed. The numbers are illustrative; they are not results from an analytics vendor or a customer account.
Follow six people before calculating a percentage
Suppose a product defines its starting cohort as people who completed a useful task during week zero. A person is retained if they complete that same kind of task in week one. Both weeks are complete, and the event definition is unchanged.
Six people qualify. Their task events arrive under the identifiers below. Separate authentication records establish the valid joins; signing in does not itself count as a successful task.
| Person | Week-zero task identifiers | Week-one task identifier | What the identity records establish |
|---|---|---|---|
| A | a-anon and a-known | a-known | Both starting identifiers belong to A |
| B | b-anon | b-known | B signs in after the anonymous starting task |
| C | c-known | c-anon | C uses another device, then authenticates there |
| D | d | No task | D has no qualifying return |
| E | e | e | Stable identity across both weeks |
| F | f | f | Stable identity across both weeks |
A raw query that treats every identifier as a separate person finds seven starting identities: A occupies two rows. Only a-known, e and f appear in both weeks. The resulting retention rate is 3 ÷ 7, or 42.9%.
Apply the valid person mapping and the starting population becomes six people. A, B, C, E and F return. Retention becomes 5 ÷ 6, or 83.3%. There are no additional task events. A’s duplicate starting identity disappears from the denominator, while B’s and C’s returns become connected to their starting activity.
The difference is large because this is a tiny teaching example. It is not an estimate of the error in your product. Its value is that each change has an explanation a reviewer can inspect.
Know what your analytics system joins
Amplitude documents three identifiers: device ID, your assigned user ID, and its own Amplitude ID. Its dashboard applies a mapping for merged anonymous identities. That mapping is not applied to raw Redshift event logs. It also says two different assigned user IDs cannot be merged through this mechanism.
Those details matter when someone says the dashboard and warehouse disagree. Counting distinct raw IDs is not necessarily the same operation as counting resolved people. The six-person example above illustrates that distinction; it does not reproduce every rule in Amplitude’s implementation.
Write down the identity rule beside the metric definition. Name the identifier used for the cohort, the mapping used for later activity, and whether the mapping is evaluated using what was known at the time or what is known now. Without that last detail, a historic retention number may appear stable while its membership is being revised.
Choose the retention window separately. Correct identity resolution cannot rescue a comparison between one cohort’s calendar week and another cohort’s first seven days.
Keep the raw event and the interpretation
Preserve the original event identifiers in your analysis inputs. Add a resolved-person field or a separate mapping rather than overwriting the only record of what arrived. A useful join record says which identifiers were connected, the supporting authenticated action, when that action occurred, and when the mapping became available to reporting.
This gives an analyst two legitimate views. A reconstructed view can answer, “Given the identities we now recognize, how many people returned?” A frozen reporting view can answer, “What did the published report say when we made the decision?” Both can be useful. They should have different names.
Do not use a shared browser, household address, company domain or similar-looking email as sufficient proof that two records represent the same person. In this example, the join is supported by authentication into a stable product account. A person using two accounts is a different problem from one account appearing under an anonymous and an authenticated identifier.
There is another boundary to preserve: a person can participate in several workspaces. Joining the person’s identity does not mean their workspaces should be collapsed into one customer. Keep account membership available for analyses that use the account as the unit.
Reconcile the rows, not just the headline
Start a discrepancy review with the smallest cohort that reproduces the problem. Export its qualifying events and compare four lists: raw starting IDs, resolved starting people, raw returning IDs, and resolved returning people. For every change in membership, require one explicit reason.
If an event is missing, investigate collection and export completeness before altering identity logic. Amplitude’s Export API selects ranges by server upload time, rather than the event’s occurrence time. A late-arriving event can therefore fall outside an export selected using the wrong clock. That is a different failure from an incorrect join.
Once the event sets agree, reconcile the cohort denominator first. Then reconcile which cohort members have a qualifying return. A single overall percentage conceals the fact that two errors can offset each other: duplicate starters might push the rate down while wrongly merged returns push it up.
Save a short reconciliation table with counts before and after each rule. Do not round until the final percentage. A useful explanation reads, “One duplicate starter was removed and two previously disconnected returns were linked,” rather than “The identity fix added 40 points.”
Protect the next release from repeating the problem
Treat a change to the product’s stable user ID as a measurement migration. A switch from an internal ID to an email address, or a new prefix introduced by another client, can change the reporting population even if account creation is unchanged.
Before rollout, follow representative histories through sign-in, sign-out, a second device, a shared device and an account switch. Check the emitted identifiers and the resulting resolved person. Include one person who never returns, so the test cannot pass simply by merging everyone into active accounts.
After rollout, watch the number of new identifiers attached to existing accounts and the share of events that cannot be resolved. Review a small sample of unexpected joins before accepting a new historical curve.
The goal is a retention result whose movements can be explained by customer activity or by a documented measurement change. When the latter occurs, annotate the report and keep the previous result available. A more accurate chart is useful precisely because the team can explain why it changed.
Sources
Something we should clarify?
Suggest a correction


