Measuring what a campaign caused

Moonbrush research note 7

Version

1.0

Published

October 20, 2026

Checked against the product

October 19, 2026

Authors

Moonbrush

Keywordscausal inference; potential outcomes; randomized controlled experiments; intent to treat; advertising effectiveness; statistical power; attribution; preregistration

Abstract

Moonbrush reports response: what happened among the people a campaign reached, counted under rules fixed before launch. Its reports print no lift figure and never say a campaign caused a result. This note sets out the research behind that choice and behind the held-back group, the design that makes a causal comparison possible.

A causal effect compares what happened with what would have happened without the campaign, which can never be observed for the same person. Randomized comparison is the standard answer. Large field experiments in advertising show that observational methods, even with rich data, often fail to recover the effects randomized experiments find, and that advertising effects are small relative to the noise in individual behavior, so a credible comparison needs very large groups. A calculation in this note shows how large. Moonbrush’s held-back group is drawn at random when an audience is created, fixed per person and removed from every export, which matches the design the experimental literature recommends. Its counting rules are fixed before launch, the principle behind preregistering a study.

1. Introduction

Two numbers are easily confused. Response is how many people reached by a campaign then did the thing it was for. Effect is how many did it because of the campaign. A campaign aimed at people already likely to act can show high response and no effect at all.

Moonbrush counts response under a precise definition: “did it from our ads” counts browsers that arrived on the client’s site through an ad link and then completed the Outcome, attributed to the latest arrival within a fixed window. See How a result is counted. This note explains why that is not an effect, what it would take to measure one, and why reports do not print one.

2. The causal question

For each person $i$, let $Y_i(1)$ be the outcome if sent the campaign and $Y_i(0)$ the outcome if not. The effect of the campaign on that person is the difference, and the average effect over the audience is (Rubin, 1974):

$$\tau = \mathbb{E}\left[\,Y_i(1) - Y_i(0)\,\right]$$

Only one of the two outcomes is ever observed for any person, which is the fundamental problem of causal inference (Holland, 1986). Randomizing who is sent solves it on average: if assignment is random, the people held back show what would have happened to the people sent, and the difference in their outcome rates estimates the effect:

$$\hat{\tau} = \bar{Y}_{\text{assigned to be sent}} - \bar{Y}_{\text{held back}}$$

Two details decide whether this comparison means anything.

• Compare assigned groups, not exposed people. Moonbrush decides who is sent; the ad platform decides who is actually shown an ad. The comparison is therefore between everyone assigned to be sent and everyone held back, whether or not they saw an ad. This is an intent-to-treat comparison: it estimates the effect of being sent, diluted by the people the platform never served (Johnson et al., 2017).
• Count the outcome the same way in both groups. “Did it from our ads” requires an ad click, which no held-back person can make, so it cannot be compared across the two groups. A comparison needs an outcome counted the same way for everyone, such as completing the Outcome by any route.

3. Why observational measurement misleads in advertising

Advertising is aimed at people likely to act, so those who see ads differ from those who do not before any ad runs. The field evidence on what this does to measurement is consistent.

• In 15 randomized experiments at Facebook, covering 500 million user-experiment observations and 1.6 billion ad impressions, observational methods applied to the same campaigns often failed to recover the effects the experiments found, even after conditioning on extensive demographic and behavioral data (Gordon et al., 2019).
• In large field experiments at eBay, ads on searches for the brand’s own name had no measurable short-term benefit, and the returns to other search ads were concentrated among new and infrequent users (Blake et al., 2015).
• Across 25 large field experiments with retailers and brokerages, individual purchases were so variable relative to the cost of the ads that the median confidence interval on return on investment was more than 100 percentage points wide (Lewis and Rao, 2015).

Attribution rules that give the whole result to the last touch before an action have been analyzed as a source of bias in their own right (Berman, 2018). Moonbrush’s “did it from our ads” is a last-arrival counting rule. It is a precise count of a defined event, not an estimate of effect, and is reported as such.

The same issue arises in targeting. The people a model ranks highest for an outcome are not necessarily the people a campaign changes most: in field data, targeting customers by predicted risk was less effective than targeting by predicted sensitivity to the offer (Ascarza, 2018). Uplift modeling addresses that question directly, typically from randomized data (Gutierrez and Gérardy, 2017).

4. The held-back group

The held-back group is a share of an audience set aside and never sent. Its design matches what the experimental literature recommends for a controlled comparison (Kohavi et al., 2009; Kohavi et al., 2020). Figure 1 shows the design.

Table 1. Design choices for the held-back group, and why each matters for a comparison.

Choice	Why it matters
Drawn at random when the audience is created, not at export	Assignment must come before any exposure and cannot depend on anything that happened after
Fixed per person and Outcome	A person who moved between groups across exports would belong to neither; assignment is keyed on the durable ID, so it survives monthly rebuilds
Removed from every export, for every destination	A held-back person reached through another path would contaminate the comparison
Off unless switched on	A comparison costs reach, so it is a choice made per Outcome

The share held back is set on the Outcome, and can be raised but not lowered. See Held-back groups.

Figure 1. The held-back design. Because assignment is random and made before anything is sent, the two groups differ only by chance and by the campaign. The outcome must be counted the same way in both, for example completing the Outcome by any route; "did it from our ads" needs an ad click, which no held-back person can make. Everyone assigned to be sent is compared, whether or not the ad platform showed them an ad, so the difference estimates the effect of being sent (Rubin, 1974; Holland, 1986). Moonbrush reports print response, not this difference: see section 7.

Schematic, not data

5. How big a comparison must be

Advertising effects are small relative to the noise in individual behavior, so a comparison must be large to detect them (Lewis and Rao, 2015). A common rule of thumb for an experiment with two equal groups, at a 5% significance level and 80% power, puts the number of people needed in each group at (Kohavi et al., 2009):

$$n \approx \frac{16\,\sigma^{2}}{\Delta^{2}}$$

where $\sigma^{2}$ is the variance of the outcome and $\Delta$ the difference to be detected. For a held-back share $h$ of an audience of $N$ people with base rate $p$, the smallest true difference the comparison would detect is, by the normal approximation,

$$\mathrm{MDE} = c\,\sqrt{p(1-p)\left(\frac{1}{(1-h)N} + \frac{1}{hN}\right)}$$

where $c = z_{1-\alpha/2} + z_{1-\beta}$, which is $z_{0.975} + z_{0.80} \approx 2.80$ at a two-sided 5% level and 80% power. Figure 2 plots it as a share of the base rate, with 10% of the audience held back.

• Base rate 0.5%
• Base rate 2%
• Base rate 5%

Figure 2. How big a difference a held-back comparison can see. The smallest true difference in response rate, as a share of the base rate, that a comparison would detect four times in five at two-sided α = 0.05, with 10% of the audience held back. Dots mark where each line reaches a 10% difference: about 166K people at a 5% base rate, 427K at 2% and 1.7M at 0.5%. Effects smaller than a line cannot be told apart from chance at that size.

Computed from a stated formula

Data behind figure 2

Audience	Held back	Base rate 0.5%	Base rate 2%	Base rate 5%
10,000	1,000	132%	65%	41%
20,000	2,000	93%	46%	29%
50,000	5,000	59%	29%	18%
100,000	10,000	42%	21%	13%
200,000	20,000	29%	15%	9.1%
500,000	50,000	19%	9.2%	5.8%
1,000,000	100,000	13%	6.5%	4.1%
2,000,000	200,000	9.3%	4.6%	2.9%
5,000,000	500,000	5.9%	2.9%	1.8%
10,000,000	1,000,000	4.2%	2.1%	1.3%

At a 2% base rate, with 10% held back, detecting a 10% relative improvement (from 2.0% to 2.2%) four times in five needs an audience of roughly 430,000 people. Many campaign audiences are smaller. Methods exist to make comparisons more precise, such as adjusting for each person’s behavior before the experiment, which cut variance by about half in one large application (Deng et al., 2013), or recording which control-group people would have seen an ad (Johnson et al., 2017). These docs do not describe Moonbrush as using them.

6. Fixed before launch

A comparison is only credible if the rules for reading it were set before the data existed. When analytic choices can follow the data, a study can find an effect that is not there; registering the questions and the analysis in advance guards against this (Nosek et al., 2018).

Moonbrush applies the same principle to counting. Before a campaign launches, its Outcome fixes what counts, how a result is attributed and its target. See How a result is counted. This is not a preregistered experiment, but it removes the same freedom to choose the reading after the results are in. The design for Outcome models, which are being built, follows the principle too: the one way a campaign result could reach the shared catalog is a request to re-judge a single model, filed under a rule registered before any data existed. See Outcome models.

7. Why reports print response, not lift

The evidence above is consistent with the reporting rule.

1. A difference needs a design. Without a randomized held-back group, a difference between people sent and not sent mixes the campaign’s effect with who was targeted, which section 3 shows can be large.
2. A design needs size. With a held-back group, many campaign audiences are too small to tell a realistic effect from chance, as Figure 2 shows. An underpowered difference printed in a report would be read as an effect.
3. Response is precisely defined. Response is a count of a defined event under rules fixed in advance. It means the same thing every week. It is also conservative: results whose records cannot be matched to a person are lost rather than guessed, and people who saw an ad without clicking are not counted.

Ranking results from model evaluation are kept apart for the same reason. A model that ranks well shows who is likely to act, not who a campaign changes. See note 6.

8. Questions an auditor should ask

1. When a held-back group is on, how is the random assignment generated, and can it be reproduced?
2. How is a held-back person kept out of every destination, including files sent before the switch was turned on?
3. Which outcome, counted the same way in both groups, would a comparison use?
4. What base rate and audience size would a given Outcome need before a comparison is worth reading?

9. Conclusion

What a campaign caused can only be measured against what would have happened without it. The field experiments in advertising show that this requires randomization and very large groups, and that methods without randomization can mislead badly. Moonbrush counts response under precise rules fixed before launch, builds the held-back group the way the experimental literature recommends, and does not print a causal number its design and sizes cannot support.

References

1. Ascarza, E. (2018). Retention futility: Targeting high-risk customers might be ineffective. Journal of Marketing Research, 55(1), 80–98. doi.org/10.1509/jmr.16.0163
2. Berman, R. (2018). Beyond the last touch: Attribution in online advertising. Marketing Science, 37(5), 771–792. doi.org/10.1287/mksc.2018.1104
3. Blake, T., Nosko, C., & Tadelis, S. (2015). Consumer heterogeneity and paid search effectiveness: A large-scale field experiment. Econometrica, 83(1), 155–174. doi.org/10.3982/ECTA12423
4. Deng, A., Xu, Y., Kohavi, R., & Walker, T. (2013). Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining (pp. 123–132). ACM. doi.org/10.1145/2433396.2433413
5. Gordon, B. R., Zettelmeyer, F., Bhargava, N., & Chapsky, D. (2019). A comparison of approaches to advertising measurement: Evidence from big field experiments at Facebook. Marketing Science, 38(2), 193–225. doi.org/10.1287/mksc.2018.1135
6. Gutierrez, P., & Gérardy, J.-Y. (2017). Causal inference and uplift modelling: A review of the literature. In Proceedings of the 3rd International Conference on Predictive Applications and APIs, Proceedings of Machine Learning Research 67 (pp. 1–13). proceedings.mlr.press/v67/gutierrez17a.html
7. Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81(396), 945–960. doi.org/10.1080/01621459.1986.10478354
8. Johnson, G. A., Lewis, R. A., & Nubbemeyer, E. I. (2017). Ghost ads: Improving the economics of measuring online ad effectiveness. Journal of Marketing Research, 54(6), 867–884. doi.org/10.1509/jmr.15.0297
9. Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R. M. (2009). Controlled experiments on the web: Survey and practical guide. Data Mining and Knowledge Discovery, 18(1), 140–181. doi.org/10.1007/s10618-008-0114-1
10. Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. doi.org/10.1017/9781108653985
11. Lewis, R. A., & Rao, J. M. (2015). The unfavorable economics of measuring the returns to advertising. Quarterly Journal of Economics, 130(4), 1941–1973. doi.org/10.1093/qje/qjv023
12. Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. doi.org/10.1073/pnas.1708274114
13. Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701. doi.org/10.1037/h0037350

How to cite this note

Moonbrush. (2026). Measuring what a campaign caused (Research note 7, version 1.0). https://docs.moonbrush.com/research/measuring-effects/