Evaluation and release
A model reaches a client only after it passes a test that was written down before it ran. Models are evaluated on a held-out test set: people kept out of training, so a model is graded on people it has never seen. Performance measured on the data a model was fitted to overstates how it will do on anyone else, which is why predictive claims are tested on new cases (Shmueli, 2010; Hastie et al., 2009).
Catalog releases
Section titled “Catalog releases”Catalog models are released together, as a version.
- A sample of the population is scored first and graded against each model’s acceptance band.
- The population run is checked against the sample.
- The release swaps in only after every gate passes, and each surface can roll back on its own.
- Each model must also pass the evidence, size, overlap, determinism and language gates.
See Quality, coverage and freshness.
Outcome models
Section titled “Outcome models”Outcome models are being built. They are designed so that:
- a person approves a model before it builds any audience;
- the audience it suggests must beat the audience groups the team built by hand, judged by the next campaign’s cost per inquiry.
What is recorded
Section titled “What is recorded”- Every training run records its metrics.
- Every release records the version that produced each score.
- Every delivered package records the score version each person was sent under.
Together these let a result be traced back to the exact model version and data behind the audience.
A published result
Section titled “A published result”From one full retrain of a higher-education model:
| Measure | Result |
|---|---|
| AUC on the held-out test set | 0.87 |
| Top tenth of the held-out test set, against the whole set’s rate | 5.04× |
AUC is the probability that a randomly chosen person who did the outcome is ranked above one who did not (Hanley and McNeil, 1982). The second measure is top-decile lift, a common measure for marketing models (Neslin et al., 2006): at 5.04×, the top tenth of the test set held about half of the people in it who did the outcome. It is a ranking measure on a test set, not the campaign lift that reports never print. Research note 6 explains both measures, what they cannot show, and the questions to ask about them.
What we do not publish
Section titled “What we do not publish”The model types, feature families, training design, exact gate values and acceptance bands are not published. We walk qualified reviewers through them under NDA, together with the evaluation record for any model they ask about. Request a technical session.