Evaluating ranking models

Moonbrush research note 6

Version

1.0

Published

October 20, 2026

Checked against the product

October 19, 2026

Authors

Moonbrush

KeywordsROC analysis; area under the curve; top-decile lift; held-out evaluation; data leakage; dataset shift; model documentation

Abstract

Gravity scores are rankings, so Gravity models are judged on how well they rank. This note explains why a model is graded on a held-out test set, what the two published measures from one full retrain of a higher-education model mean (an AUC of 0.87, and a top tenth of the test set in which the outcome rate was 5.04× the rate across the whole set), and how both relate to the established literature on ROC analysis and lift.

AUC is the probability that a randomly chosen person who did the outcome is ranked above a randomly chosen person who did not. It does not depend on the base rate or on any order-preserving rescaling of the scores, which is why it suits a ranking. The top-tenth rate answers the practical question for a team working down a list. Neither measure says how likely any one person is to act, and neither says what a campaign would cause. The note sets out the threats that can make a held-out result look better than it is (leakage, dependence between training and test data, and shift over time), states which controls are published and which are available under NDA, and ends with the questions an auditor should ask.

1. Introduction

A Gravity model sorts the people it can see. Its score is a rank from 0 to 1, not a probability: a higher score means a person ranks higher on the model, and it makes no claim about that person’s chance of acting. See Reading a score.

The claim a ranking model makes is therefore narrow and testable. If the model is good, the people who later do the thing it ranks for should sit near the top of the list. Everything in this note follows from testing that claim honestly: on people the model has never seen, with measures that reward ranking and nothing else, and with the limits of those measures stated next to the result.

2. Out-of-sample evaluation

A model fitted to data will describe that data better than it describes new data, because some of what it learns is noise that will not recur. Measured performance on the training data is therefore an optimistic estimate of performance on anyone else (Hastie et al., 2009). The distinction matters most when the purpose is prediction rather than explanation: a predictive claim is a claim about new cases, and it can only be assessed on new cases (Shmueli, 2010; Breiman, 2001).

Moonbrush evaluates models on a held-out test set: people kept out of training, so the model is graded on people it has never seen. This is the standard design for out-of-sample evaluation (Hastie et al., 2009).

2.1 Leakage

A held-out test can still mislead if information about the outcome reaches the model through the features. Leakage is the use of information at training time that would not be available when the model is used, and it can make a weak model look strong (Kaufman et al., 2012). The common cases in marketing data are features recorded after the outcome window opened, and features that are by-products of the outcome itself.

2.2 Dependence between training and test data

A random split assumes the test people are independent of the training people. When records are related, by household, by place or by time, a random split lets the model learn the relationship and then be rewarded for it on the test set, so the error estimate is too small. Splitting by blocks, such as by time period, gives a more honest estimate (Roberts et al., 2017).

Whether and how the published retrain controlled for leakage and dependence is part of its study design, which is available under NDA. Section 7 lists the questions to ask.

3. AUC: the probability of a correct ordering

3.1 Definition

Take one person who did the outcome and one who did not, both at random from the test set. Let their scores be $S^{+}$ and $S^{-}$. The area under the ROC curve is the probability that the model orders them correctly, with ties counted as half (Hanley and McNeil, 1982):

$$\mathrm{AUC} = \Pr\left(S^{+} > S^{-}\right) + \tfrac{1}{2}\Pr\left(S^{+} = S^{-}\right)$$

On a test set with $n^{+}$ people who did the outcome and $n^{-}$ who did not, it is estimated by counting correctly ordered pairs, which is the Mann-Whitney statistic scaled to the unit interval (Mann and Whitney, 1947; Hanley and McNeil, 1982):

$$\widehat{\mathrm{AUC}} = \frac{1}{n^{+} n^{-}} \sum_{i=1}^{n^{+}} \sum_{j=1}^{n^{-}} c_{ij}$$

where each pair scores 1 if it is ordered correctly and a half if tied:

$$c_{ij} = \mathbf{1}\left[s_i^{+} > s_j^{-}\right] + \tfrac{1}{2}\,\mathbf{1}\left[s_i^{+} = s_j^{-}\right]$$

An AUC of 0.5 is a random order. An AUC of 0.87 means that in 87 of every 100 such pairs, counting ties as half, the person who did the outcome was ranked above the person who did not.

3.2 Why it suits a ranking

AUC depends only on the order of the scores. Any strictly increasing rescaling leaves it unchanged, and it does not depend on the share of people who did the outcome (Fawcett, 2006; Bradley, 1997). Both properties match what a Gravity score claims: an order, not a calibrated chance.

Figure 1 shows what curves with these areas look like under the textbook equal-variance binormal model, in which the two groups’ scores are normal with the same spread and their means are $d$ standard deviations apart. The difference between one score from each group is then normal with mean $d$ and variance 2, so the chance that it is positive is

$$\mathrm{AUC} = \Phi\!\left(\frac{d}{\sqrt{2}}\right)$$

• AUC 0.87, the published value
• AUC 0.75, for comparison
• Chance, AUC 0.50

Figure 1. What an AUC of 0.87 looks like. Each curve traces every possible cut through a ranked list: how much of the group that did it sits above the cut, against how much of the group that did not. The area under a curve is its AUC. Curves follow the equal-variance binormal model, in which AUC = Φ(d/√2), so d = 1.593 for 0.87 and 0.954 for 0.75. The shaded area under the rust curve is 0.87 of the square. Only the AUC value is published; the shape of the real model's curve is not.

Computed from a stated formula

Data behind figure 1

False positive rate	AUC 0.87	AUC 0.75	Chance
0%	0.0%	0.0%	0.0%
5%	47.9%	24.5%	5.0%
10%	62.2%	37.2%	10.0%
20%	77.4%	54.5%	20.0%
30%	85.7%	66.6%	30.0%
50%	94.4%	83.0%	50.0%
70%	98.3%	93.0%	70.0%
100%	100.0%	100.0%	100.0%

3.3 Uncertainty

An AUC measured on a finite test set has sampling error. Standard methods give its confidence interval and compare two correlated AUCs measured on the same people (DeLong et al., 1988), or estimate the interval by resampling (Efron and Tibshirani, 1993).

3.4 What AUC leaves out

AUC averages performance over every possible cut through the list, including cuts no one would use. It has also been criticized on a different ground: read as an average loss, it weights misclassification costs differently for different models, so comparing models by AUC compares them with different yardsticks (Hand, 2009). When very few people do the outcome, a precision-recall view can be more informative than the ROC view (Saito and Rehmsmeier, 2015). For a marketer, the cut that matters is near the top of the list. That is what the second measure reads.

4. The top tenth: what a list holds where it is used

4.1 Definition

Sort the test set by score and take the top 10%. The top-tenth rate is the outcome rate in that tenth divided by the outcome rate across the whole set:

$$L_{10} = \frac{\text{rate of the outcome in the top 10\% by score}}{\text{rate of the outcome in the whole test set}}$$

This is the top-decile lift, a common accuracy measure for marketing models; it has been used, for example, to compare customer defection models in a modeling tournament (Neslin et al., 2006). Lift-based measures were proposed for ranked direct-marketing lists because plain accuracy says nothing about how responders are spread down a list (Ling and Li, 1998).

4.2 Reading the published value

Because the top tenth holds one tenth of the people, the share of everyone who did the outcome that lands in the top tenth is one tenth of $L_{10}$:

$$\text{share of outcomes captured in the top 10\%} = 0.1 \times L_{10}$$

At the published 5.04×, the top tenth of the held-out test set held about 50% of the people in it who did the outcome. A random order would hold 10%.

4.3 The ceiling

No ranking can do better than putting every person who did the outcome at the top. If $p$ is the base rate of the test set,

$$L_{10} \le \min\!\left(10,\ \frac{1}{p}\right)$$

so the top-tenth measure is bounded at 10 when at most one person in ten did the outcome, and lower when more did. The base rate of the published test set is not published, so its true ceiling lies somewhere between 5.04 and 10. Figure 2 places the published value between the floor and the highest possible ceiling.

The word lift here names a ranking measure on a test set. It is not the campaign lift that Moonbrush reports never print; see note 7.

Figure 2. The top tenth of the held-out test set, against the whole set. The rate of the outcome among the top-ranked tenth of the held-out test set, divided by the rate across the whole set. A random order scores 1×. No order can score more than 10× in its top tenth, or more than 1 divided by the base rate when more than one person in ten did it. The published value comes from one full retrain of a higher-education model. The base rate of that test set is not published, so its true ceiling lies between the published value and 10×.

Moonbrush published figure

Data behind figure 2

Ordering	Top tenth against the whole set	What it means
Random order	1×	the top tenth holds its share and no more
Published retrain	5.04×	top tenth of the held-out test set
Ceiling	10×	every person who did it ranked in the top tenth, when at most 1 in 10 did it

5. What these measures do not show

Table 1. Claims the published measures support, and claims they do not.

Question	Answered by AUC and the top tenth?	Where it is addressed
Does the model put people who later did it near the top?	Yes, on the test set it was graded on	This note
How likely is one person to act?	No. Scores are not calibrated probabilities	Section 5.1
What would a campaign aimed at the top tenth cause?	No. A rank is not an effect	Note 7
Will the model rank as well next quarter?	Not by itself. Performance can drift	Section 5.2
Do all catalog models perform this well?	Not answered. This is one retrain of one model	Section 5.3

5.1 Ranking is not calibration

A model can rank perfectly and still assign numbers that are poor estimates of probability. Some common learning methods produce scores that need a separate calibration step before they can be read as probabilities (Niculescu-Mizil and Caruana, 2005), and calibration is judged with different measures, such as the Brier score (Brier, 1950). Moonbrush makes no probability claim for catalog scores, and no page in these docs should be read as one.

5.2 Shift over time

A model is fitted to one period and used in the next. The relationship between behavior and outcome can change, as can the population and the base rate. The literature calls these dataset shift and concept drift (Moreno-Torres et al., 2012; Gama et al., 2014). The controls Moonbrush publishes are structural: catalog models change only by release, each release scores a sample first and must pass its gates before it swaps in, and every score carries its model version and date. See Quality, coverage and freshness.

5.3 One result is one result

The published figures come from one full retrain of one higher-education model. They show what that model achieved on its test set. They are not an average over the catalog, and they are not a forecast for another client or category.

5.4 A good rank does not mean a good target

The people a model ranks highest for an outcome are not necessarily the people a campaign changes most. In a field study of customer retention, targeting the customers at highest predicted risk was less effective than targeting the customers most responsive to the intervention (Ascarza, 2018). A ranking result answers which people are likely to do something. Whether reaching them changes what they do is a causal question, taken up in note 7.

6. Recording, so a result can be traced

Evaluation is only auditable if the result can be traced to the exact model and data that produced it. Model documentation practice recommends a card for each model that states its intended use, its evaluation and its limits (Mitchell et al., 2019), and equivalent records for datasets (Gebru et al., 2021). Untracked versions and configuration are a known source of silent failure in production systems (Sculley et al., 2015).

At Moonbrush, every training run records its metrics, every release records the version that produced each score, and every delivered package records the score version each person was sent under. Every model carries a card with its definition, its limits, its evidence, its measured size, its window and its current reading. See How models work.

7. Questions an auditor should ask

These are the questions this note cannot answer in public. Each can be raised in a technical session under NDA.

1. How was the test set drawn: at random, by time period or by another block? Could any person, or household, appear on both sides?
2. Were all inputs recorded before the outcome window opened?
3. What is the confidence interval on the published AUC?
4. What was the base rate of the test set, and so what was the ceiling for the top tenth?
5. How does the model rank on a period later than the one it was trained on?
6. What acceptance bands did the release pass, and what happens when a gate fails?

8. Conclusion

The published measures are ranking measures, read on people the model never saw, and they are reported with what they cannot show. AUC says how often the model orders a pair correctly; the top tenth says how much of the outcome sits where a list is actually used. Both are established measures in the literature. The remaining questions, about leakage, dependence, uncertainty and drift, are the right ones to ask, and the evaluation record that answers them is available to qualified reviewers.

References

1. Ascarza, E. (2018). Retention futility: Targeting high-risk customers might be ineffective. Journal of Marketing Research, 55(1), 80–98. doi.org/10.1509/jmr.16.0163
2. Bradley, A. P. (1997). The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30(7), 1145–1159. doi.org/10.1016/S0031-3203(96)00142-2
3. Breiman, L. (2001). Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical Science, 16(3), 199–231. doi.org/10.1214/ss/1009213726
4. Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3. doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2
5. DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics, 44(3), 837–845. doi.org/10.2307/2531595
6. Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. Chapman & Hall.
7. Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874. doi.org/10.1016/j.patrec.2005.10.010
8. Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014). A survey on concept drift adaptation. ACM Computing Surveys, 46(4), Article 44. doi.org/10.1145/2523813
9. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., & Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86–92. doi.org/10.1145/3458723
10. Hand, D. J. (2009). Measuring classifier performance: A coherent alternative to the area under the ROC curve. Machine Learning, 77(1), 103–123. doi.org/10.1007/s10994-009-5119-5
11. Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1), 29–36. doi.org/10.1148/radiology.143.1.7063747
12. Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer. doi.org/10.1007/978-0-387-84858-7
13. Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. (2012). Leakage in data mining: Formulation, detection, and avoidance. ACM Transactions on Knowledge Discovery from Data, 6(4), Article 15. doi.org/10.1145/2382577.2382579
14. Ling, C. X., & Li, C. (1998). Data mining for direct marketing: Problems and solutions. In Proceedings of the Fourth International Conference on Knowledge Discovery and Data Mining (KDD-98). AAAI Press. www.csd.uwo.ca/~xling/papers/kdd98.pdf
15. Mann, H. B., & Whitney, D. R. (1947). On a test of whether one of two random variables is stochastically larger than the other. Annals of Mathematical Statistics, 18(1), 50–60. doi.org/10.1214/aoms/1177730491
16. Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019). Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* ’19) (pp. 220–229). ACM. doi.org/10.1145/3287560.3287596
17. Moreno-Torres, J. G., Raeder, T., Alaiz-Rodríguez, R., Chawla, N. V., & Herrera, F. (2012). A unifying view on dataset shift in classification. Pattern Recognition, 45(1), 521–530. doi.org/10.1016/j.patcog.2011.06.019
18. Neslin, S. A., Gupta, S., Kamakura, W., Lu, J., & Mason, C. H. (2006). Defection detection: Measuring and understanding the predictive accuracy of customer churn models. Journal of Marketing Research, 43(2), 204–211. doi.org/10.1509/jmkr.43.2.204
19. Niculescu-Mizil, A., & Caruana, R. (2005). Predicting good probabilities with supervised learning. In Proceedings of the 22nd International Conference on Machine Learning (pp. 625–632). ACM. doi.org/10.1145/1102351.1102430
20. Roberts, D. R., Bahn, V., Ciuti, S., Boyce, M. S., Elith, J., Guillera-Arroita, G., Hauenstein, S., Lahoz-Monfort, J. J., Schröder, B., Thuiller, W., Warton, D. I., Wintle, B. A., Hartig, F., & Dormann, C. F. (2017). Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography, 40(8), 913–929. doi.org/10.1111/ecog.02881
21. Saito, T., & Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE, 10(3), e0118432. doi.org/10.1371/journal.pone.0118432
22. Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., & Dennison, D. (2015). Hidden technical debt in machine learning systems. In Advances in Neural Information Processing Systems 28 (pp. 2503–2511).
23. Shmueli, G. (2010). To explain or to predict?. Statistical Science, 25(3), 289–310. doi.org/10.1214/10-STS330

How to cite this note

Moonbrush. (2026). Evaluating ranking models (Research note 6, version 1.0). https://docs.moonbrush.com/research/evaluating-ranking-models/