CritterSnaps — The Rigorous Version

How Scoring Works — The Rigorous Version

Elo as maximum-likelihood inference under the Bradley–Terry model. For the statisticians in the house.

1. The generative model

Each critter i carries a latent strength parameter R_i (its rating). A head-to-head is a Bernoulli trial: the probability that i is preferred to j is a logistic function of the rating gap,

(1) E_ij = P(i ≫ j) = 1 ÷ (1 + 10^{−(R_i − R_j)/400}) = σ ₁₀((R_i − R_j)/400).

Writing log-strengths β_i = (ln 10 / 400) R_i, this is exactly the Bradley–Terry model P(i ≫ j) = π_i / (π_i + π_j) with π_i = e^β_i = 10^R_i/400. The base 10 and scale 400 are pure convention: a 400-point gap is one log₁₀ unit, i.e. 10 : 1 odds.

2. The update rule is online gradient ascent

After a game with outcome S_i ∈ {0, 1}, Elo does

(2) R_i ← R_i + K (S_i − E_ij).

This is not ad hoc. Take the Bernoulli log-likelihood of that single game, ℓ = S_i ln E_ij + (1−S_i) ln(1−E_ij). Because ∂E_ij/∂R_i = E_ij(1−E_ij) (ln 10/400), the score function collapses to

(3) ∂ℓ / ∂R_i = (S_i − E_ij) · (ln 10 / 400).

So (2) is a stochastic gradient-ascent step on the Bradley–Terry log-likelihood, with the constant ln 10/400 absorbed into the step size K. Elo is single-sample SGD for logistic regression on pairwise-comparison data.

3. Conservation and identifiability

Since S_j = 1−S_i and E_ji = 1−E_ij, the loser’s gradient is the negative of the winner’s: ΔR_i = −ΔR_j. Every update is zero-sum, so ∑_i R_i is invariant — the dynamics live on a fixed-sum hyperplane. Only rating differences are identified; the model has a one-parameter gauge freedom (an additive constant), which we fix by the arbitrary origin R = 1200.

4. Fixed points and the role of K

A stationary point of the expected update requires E[S_i − E_ij] = 0 for all i — the model’s win probabilities equal the empirical ones. That is the maximum-likelihood / method-of-moments condition. But K is the learning rate, and with K held constant the iterates do not converge to the MLE; they reach a stationary distribution fluctuating around it with variance O(K) — the familiar constant-step-SGD trade-off:

Large K → fast adaptation, high variance (tracks non-stationarity — useful if true strengths drift). Small K → slow, low-variance, near the MLE. Almost-sure convergence needs a Robbins–Monro schedule, ∑K_t = ∞, ∑K_t² < ∞. We use the standard chess value K = 32.

The Bayesian descendants — Glicko (rating + a rating-deviation) and TrueSkill (a Gaussian factor graph solved by expectation propagation) — are precisely this idea made adaptive: the effective step per player shrinks as its posterior variance does.

5. Why we match critters of similar rating

Here is the part that governs the leaderboard. The Fisher information a single comparison carries about the gap Δ = β_i − β_j is the Bernoulli variance

(4) ℇ(Δ) = σ(Δ) (1 − σ(Δ)) = p (1−p), maximised at p = ½ (Δ = 0).

An evenly-matched game (p ≈ ½) is maximally informative; a blowout (p ≈ 0 or 1) carries almost none — the outcome was a foregone conclusion, so observing it barely moves the estimate. This is the D-optimal design result for logistic regression: place your design points where the variance function p(1−p) is largest.

Consequently, a scheduler that pairs opponents at random spends most of its samples on lopsided, low-information games — and, fatally for the ranking, never plays the top contenders against one another, leaving the head of the order non-identified in finite samples (a cluster of mutually-untested, undefeated leaders). CritterSnaps therefore biases opponent selection toward equal current ratings, maximising information per comparison and resolving the ordering exactly where it matters most.

The coverage heuristic (favour least-played critters for one slot) handles the complementary concern — that every parameter receives samples at all, including new uploads sitting at the 1200 prior.

← Back to the plain-English version