Personalization Rules Built Against a Holdout

A personalization rule earns its place only when a holdout that never sees it performs worse. I keep 10 percent of eligible traffic out of every rule, permanently, and report the gap each quarter. In the rule sets I have taken over, close to half of what was running turned out to be doing nothing measurable, and it had been running for more than a year. What we start from: Your existing rules, not a blank page. Each one gets a verdict, keep, retire or rebuild, and the test is whether its eligibility can be written as a query. Rules firing on engaged users or high intent are retired unless somebody can produce the SQL behind those words. That first pass usually removes more than it adds, and the page gets faster because there are fewer decisions to resolve before render. The build: For each surviving rule I write the eligibility condition, the precedence order for when two rules claim the same slot, the decay window, the fallback the unmatched majority sees, and a kill switch a marketer can pull without waiting for a deploy. Decay is set per signal, not globally: a deposit signal is stale at 14 days, a category browse at 30. Precedence is where inherited rule sets fail. Three rules qualify, whichever loaded last wins, and nobody can reconstruct what a given visitor actually saw. Where it runs: Your stack. I have built this into VWO, Optimizely Web, Dynamic Yield, and plain feature flags fed by segments computed in BigQuery or Snowflake and synced through Hightouch. If audiences already live in Segment or RudderStack they stay there and I build only the delivery and measurement layer. I sell no platform and take nothing from any vendor. Measurement: Every rule logs an exposure at decision time carrying rule id, version and holdout flag, not at render. Without that you cannot separate a rule that fired and did nothing from a rule that never fired at all. Ratio metrics are read at user level using the delta method, because personalization changes how many sessions a person starts and a session-level read then reports a win that is only a moved denominator. What I refuse: No recommendation models. If you want collaborative filtering that is an ML team's work and I will say so on the first call rather than in month three. In gambling funnels I do not build prompts triggered by a loss, countdowns that are not real, or reactivation rules aimed at a player who has just cooled off after a heavy session. Italian regulation and my own judgement land in the same place on that. For kids-and-family products I do not build behavioural profiles on users under the Italian digital consent age of 14; those rules run on context, meaning device, session and catalogue position, never on the child. Who this is not for: Teams whose largest addressable segment sees under 8,000 sessions a month. The rule may well be right, but you will never know whether it was, and I would rather say that before the invoice than after. Also not for anyone who wants a personalization strategy deck. This ends in shipped rules with named owners. What you get: A rule register, one row per rule, carrying eligibility SQL, precedence rank, decay window, holdout share, owner and retirement date. The rules themselves, live in your stack. And a quarterly readout naming which rules paid, which broke even and which I recommend retiring, including the ones I built.