Held-back groups
A held-back group is a share of an audience that is set aside and never sent. Because it is drawn at random, it shows what would have happened to the people sent had they not been sent, which is the comparison a causal estimate needs (Rubin, 1974; Holland, 1986). See research note 7.
It is a switch
Section titled “It is a switch”The held-back group is an option on each Outcome. It is off unless the Outcome’s switch is turned on. When it is off, nothing is withheld from any list or export.
How it is drawn
Section titled “How it is drawn”- At creation, not at export. The held-back share is drawn when an audience is created for an Outcome, never at the moment of export.
- Fixed per person. Assignment is fixed for each person and Outcome, so a person held back once stays held back for that Outcome. It cannot drift between exports.
- Removed from every export. Held-back people are left out of every file, for every destination.
- The share can rise, not fall. The held-back share is set on the Outcome. It can be raised, and cannot be lowered.
Response, not lift
Section titled “Response, not lift”A result is called response: what happened among the people who were reached. Reports print no lift figure and never say a campaign caused a result. Field experiments show that advertising effects are small relative to the noise in individual behavior, so a credible comparison needs very large groups (Lewis and Rao, 2015).
Comparisons stay inside the campaign
Section titled “Comparisons stay inside the campaign”Learning and reporting compare within the flight that produced the result, never against the general population. A general-population rate is a different population and would flatter or punish a campaign for reasons it did not cause.