MB · The Lab

EXP. 001 · The Allocator

The Allocator

Three strategies spend the same traffic on the same noisy market — sequential A/B testing, bandits that reallocate every visitor, and auto-research that finds the biggest lever first. Without JavaScript the race can't run; the standings are described in the notes.

The market you’re optimizing — 1998
few big levers, still 2026
many small, drifting

EXP. 001 — Notes

The Allocator

The market is a factored conversion problem: four levers, four variants each, each variant contributing to a 2% base rate; every visitor is a real Bernoulli draw, seeded and replayable. Three strategies spend identical traffic. Sequential A/B runs the classical program — equal-split test on one lever for a fixed horizon, lock the observed winner, move down the roadmap. Bandits is Thompson sampling with a forgetting factor: it holds a posterior per variant and reallocates on every single visitor. Auto-Research spends a probe budget first, ranks the levers by measured sensitivity, then optimizes them largest-first and re-probes when its own conversion rate sags. The dashed ceiling is the expected rate of the true best combination; regret — conversions left on the table — is each lane's distance below it.

Drag the era dial and the argument makes itself. At the left — few levers, big effects, a still market — the quarterly cadence is honestly fine: a thousand visitors per variant detects a three-point spread, the winner stays the winner, and discipline beats cleverness. That is the world the ritual was built for.

At the right — many small effects, drifting — the same ritual fails twice. The fixed horizon is set by the calendar, not by statistical power, so it locks wrong winners; and drift means even right winners spoil while the roadmap looks elsewhere. Watch the bandits lane: its edge isn't intelligence, it's cadence — reallocating every visitor instead of every quarter. Then watch which lever auto-research spends on first. Roadmaps run in presentation order; sensitivity runs in magnitude order. Finding the highest-order bit before polishing the low ones is most of the job now — yours shifts from running the tests to designing the space the machine searches.

Three gamblers, one table, the same deck. The first bets by the quarter and calls his patience rigor. The second bets by the hand. The third asks the only question that compounds: which game here is worth playing? The house keeps whatever hesitation leaves — and certainty, held too long, is just inventory spoiling.

  1. Mechanism under test: experiment strategy itself. Not descent on a fixed landscape (that is EXP. 007) but the allocation of scarce, noisy traffic across a live market — exploration against exploitation, run honestly. Every conversion on the scoreboard is an actual seeded Bernoulli event; nothing is staged, including the losing.
  2. The fair fight: all three lanes face the same ground truth, the same drift, the same budget of 60,000 visitors, and start from the same incumbent. The exhibit knows its own truth — it is the market — which is how it can draw the ceiling and outline each lever's true best variant in ember. The lanes never see any of that; they see conversions.
  3. The era dial: one control, dragged across a historical argument. It scales effect sizes down and drift up — from a 1998-shaped problem (spread ≈ 3.5 points, still) to a 2026-shaped one (spread ≈ 1.2, moving). The gap between the lanes is a function of the dial, and that function is the exhibit.
  4. The highest-order bit: auto-research's first act is to spend ~7% of its budget learning which lever is worth the rest. Its probe estimates are shown in its own status line, mistakes included — with 100 visitors per variant the ranking is honest but fallible, and occasionally it bets the quarter's way by accident. That fallibility stays in.
  5. What might graduate: the race-against-a-ceiling grammar (regret as the visible quantity, not raw score), and the era dial — a single slider that carries a thesis across a phase change.

PHYSICS — canvas 2D, typed-array posteriors, no libraries, single file · seeded mulberry32 (era + deal replay identically; RE-RUN advances the deal) · rAF only while the race runs; at 60,000 visitors the loop stops cold — settle means zero frames · reduced motion = the completed race, computed synchronously, drawn once · no-canvas = designed flat edition · ?freeze renders visitor 30,000 deterministically, ?norun exercises the fallback · zero console errors · native scrollbar and skip link intact.

← The Lab  ·  ← morganbrown.co