Does consensus predict the right answer?

Whole-exam prediction tests

Every reported prediction excludes all official answers from its own exam. Vote percentages and respondent counts are inputs; the independent labels are questions, grouped into only five exams.

Log loss penalizes confident wrong answers heavily. The multiclass Brier score sums squared errors across the answer choices. April 2023 has five choices; later exams have four. Every method uses the correct choice count for each exam.

Where confidence still fails

Explore the withheld predictions below. All probabilities shown here were computed without that exam’s answer key. Special-grading rows are retained for inspection and are excluded from the single-answer fit.

Do the probabilities match outcomes?

Agreement within these groups is descriptive. The same five exams support every group, so this is not proof of calibration on October 2026 or on future exams.

Can special grading be predicted?

A preliminary test compares one overall event rate with separate rates for questions below 50%, from 50% to below 75%, and at least 75% consensus. Both use the same fixed Beta(1,19) prior and hold out an entire exam.

The agreement groups improve log loss slightly but do not improve Brier score. Three universal-credit cases across two exams are too few to support reliable question-specific cancellation probabilities. The comparison model therefore estimates a single accepted answer and does not add special-grading credit.

What the evidence suggests changing

1. Calibrate vote margins across exams. Equal vote shares understate how often a clear leader is right. The fitted margin model improves both pooled scoring measures over raw shares and the constant leader baseline. It beats the leader baseline on log loss in all five exams, but on Brier score in only three. Against an April-only fit on equal votes, the improvement is small and mixed.
2. Keep shared mistakes in the fit. The five high-confidence failures, including April 2026 Q19, remain in the evaluation. Removing them would make probabilities look more reliable than they are.
3. Recover aligned response matrices before changing voter weights. The archive can evaluate consensus probabilities, but not top-10% selection, similarity discounts, or individual score-error bands.
4. Collect more special-grading cases. Universal credit and multiple accepted answers are separate outcomes. More exams are needed before adding their probabilities to an individual score.

No voting weights or total-score error bands were changed by this evaluation. The historical probability comparison is available in each answer’s details on the estimator.

Model, sources, and reproducibility

P(a | s, β, ε) = (1 − ε) × exp(βsₐ) / Σⱼ exp(βsⱼ) + ε / K

The model averages over the posterior for vote-margin sensitivity β and error-mixture ε. K is four or five according to the exam. Priors remain half-normal(σ=10) and Beta(1,9), with the same integration grid as the April model. These settings were not optimized on the held-out exams. The leader baseline uses its training exams’ leader success rate with Beta(1,1) smoothing and distributes the remaining probability equally among other choices.

Four exams come from CBLEInsights. All 320 accepted-answer sets were checked against copies of CBP’s published keys, linked below. April 2026 uses the estimator’s previously audited response sheet and key. The public evidence contains question totals and accepted answers, without participant names or individual response records.

Survey snapshots may contain changes made after the key was released. Collection timing could not be verified. Published keys also do not establish the outcomes of later individual appeals. Candidate comparisons guide model development; confirmation requires another exam not used in that process.

Download anonymous question evidence · Download all withheld predictions and model parameters