MB · The Lab

EXP. 007 · The Loss Landscape

SGD — · MOM — · ADAM —
tap: drop the runners · drag: raise ground

The Lab — EXP. 007 · machine learning, on foot

Three ways down the same mountain.

A loss surface, surveyed — and three optimizers released on the same ridge: SGD stumbles honestly, momentum overshoots and rings, Adam renormalizes every step. Tap to drop all three somewhere new; drag to raise the ground under them mid-descent.

EXP. 007 — Notes

The Loss Landscape

The terrain is a loss surface — error as a function of choices, lower is better — and the three hikers are real optimizers, not animations: SGD takes small noisy steps straight downhill; Momentum accumulates velocity and overshoots turns; Adam adapts its stride per direction. None can see the map. Each knows only the slope under its feet, which is the honest condition of optimization everywhere.

Your testing program is one of these hikers. Each experiment measures the local slope; your strategy is how you step. Many small cheap tests are SGD — noisy, but hard to trap. Committed roadmap bets are Momentum — fast on a long slope, expensive at a turn. Adaptive budget per lever is Adam.

The trap has a name: the local minimum where every small test loses and the team concludes it's done — when a jump (a redesign, a repricing, a new market) would find a deeper valley. Budget for occasional jumps. And notice you can drag the ground here mid-race: markets do that too.

Three temperaments loose on a mountain of error, each certain its next step is down. The mountain doesn't care. It keeps its deepest valley wherever you aren't looking, and moves it while you climb.

  1. Mechanism under test: the three canonical update rules, run for real. SGD: x ← x − η∇L + seeded gaussian noise, with a reduce-on-plateau learning-rate schedule (η anneals only once L stops improving — what lets constant-noise SGD honestly converge — and re-warms when you change the problem). Momentum: v ← βv − η∇L, x ← x + v (β = 0.9). Adam: per-coordinate first and second moments m, v with bias correction m̂ = m/(1−β₁ᵗ), v̂ = v/(1−β₂ᵗ), then x ← x − α·m̂/(√v̂+ε). Gradients are analytic — the surface is a known sum of gaussians, a bowl, and one saddle, so ∇L is written in closed form, not finite-differenced; the shader and the JS evaluate the same expression.
  2. Renderer lineage: the survey read is EXP. 002's — hillshade, hypsometric tint, fwidth-AA contours with heavier index contours, a jittered hachure-stroke grid aligned down the gradient. Here the field is analytic, so height and slope are evaluated in-shader instead of sampled from a dye texture; contour lines are true level sets of L.
  3. The mind-bend: optimizers have personalities — the trails are character studies, not decoration. And the landscape is the hard part: drag raises a gaussian ridge under the pointer while they descend, and you watch three different theories of motion cope with a problem that changed under their feet.
  4. Settle honesty: an optimizer is converged when |∇L| and its step both stay below ε for a run of frames (both read through a short EMA, so one noisy step doesn't reset the clock) (the domain is clamped at the map edge, and a runner pinned there is retired as EDGE). When all three are down, the race is over: final losses freeze in the legend and the loop stops at zero frames — waking only for your tap or drag.
  5. What might graduate: optimizer-trail figures for essays on learning — a settled race is already a publishable diagram (three colored descents on a contour sheet). Also: the analytic-field variant of the survey renderer, cheaper than the texture-fed one wherever the terrain is known in closed form.

PHYSICS — raw WebGL2, RGBA8 canvas only (no float textures), no libraries, single file · seeded terrain + seeded SGD noise (mulberry32); ?freeze renders the finished race deterministically · analytic gradients, fixed substeps · loop runs only while an optimizer is still moving or the pointer is down, then stops — zero idle frames, visibility- and viewport-gated · reduced motion = the settled race as a still · no-GL = designed fallback · zero console errors.

← The Lab  ·  ← morganbrown.co