Proctor project · 13 November 2026

An Adaptive Reasoning Test

A high-range test that picks each item to fit the answers so far, built in a day on shared web hosting. The items are drafted by one language model, checked by a blind solve by another, and banked rather than made on demand. This page explains how it works, what the checking catches, how the bank grows on a budget of $10 a month, why the score it reports is not yet an IQ, and why no test like it can reach the one-in-a-million level.

The test is at questrel.com/oracle/irt/.

The starting point

The design began as a sketch of a C++ server. On each request it asked Gemini to write a new item at a target difficulty, scored the candidate with item response theory, nudged the item's difficulty after each test, and sat behind an nginx proxy at /oracle/irt/. Three parts of that sketch survive unchanged: the two-parameter model, the ability estimate, and the rule that updates an item's difficulty. The rest changed, for two reasons: where the test had to run, and how reliable machine-made items turned out to be.

Why it is a CGI program

questrel.com is shared cPanel hosting: Apache on CloudLinux, with no nginx, no root access, and no long-running processes, which the host stops. A server that stays up and listens on a port has nowhere to live. Apache will, though, run a program once per request (CGI), and the host has a C++ compiler. So proctor is a CGI program. It starts, answers one request, and exits, and everything it must remember is in a SQLite database outside the web folder: the items, each sitting, and each answer.

Two hosting details cost time and are worth knowing:

How a sitting works

Each item has two numbers: a difficulty b and a discrimination a, on a scale where an ability θ of 0 corresponds to IQ 100 and each unit to 15 points. The chance that someone of ability θ answers it correctly is

P(θ) = 1 / (1 + e^(−a(θ − b)))

and the item tells you most about θ where that chance is near one half. The measure is its information, a²P(1 − P).

  1. Start. The candidate may list scores on other tests (below). The server creates a session with a random 128-bit id.
  2. Choose. For each item, the server works out the current ability estimate from the answers so far. It takes the domain (verbal or mathematical-spatial) seen less often, ranks the unused items there by information at that estimate, and picks at random among the best three, so that the same few items are not shown to everyone.
  3. Answer. The server records when it sent the item and when the answer came back, so response times cannot be faked. An answer under three seconds is counted as fast. An item already sent comes back again on reload, so it cannot be skipped by refreshing.
  4. Score. After up to 20 items, ability is estimated by maximum a posteriori: Newton steps on the likelihood with a normal prior of mean 0 and standard deviation 2. The prior keeps the estimate finite when every answer is right or every answer is wrong. The standard error comes from the curvature at the estimate. The page reports 100 + 15θ with a 95% range.
  5. Recalibrate. If the sitting had at least five answers, each item's difficulty moves a little: easier when it was answered right more often than the model expected, harder when less often. The step shrinks as the item collects responses. Discrimination stays at 1.5 for now.

Nothing is generated while a candidate waits, and the answer key never leaves the server. A visitor can start at most 30 sittings a day.

Where the items come from

A separate program, itemgen, drafts items offline on a desktop machine through OpenRouter, so no API key sits on the public host. For each domain and each target level from θ = 0 to 5 (IQ 100 to 175) it asks Gemini 2.5 Flash for an item of a given kind, as JSON. The kinds are:

The prompt rules out figures, published puzzles, trick wording and obscure knowledge, and asks for exactly one defensible answer and a worked solution. The options are then shuffled, because models like to put the right answer first.

Each draft then goes to a blind solve: a second model, with reasoning turned up, sees only the question and the options. It must pick one and say whether the item is sound: exactly one option defensible by a clear rule, no other equally defensible, and unambiguous wording. The item goes live only if the solver picks the drafted key and judges it sound; otherwise it is kept, inactive, with the solver's one-line reason. The very first draft shows why a check is needed:

Which of the following best continues the series? READ, BOOK, LIBRARY, ARCHIVE, …

ENCYCLOPEDIA · INTERNET (drafted key) · MUSEUM · DATABASE

The blind solve chose DATABASE. Neither answer is defensible over the other.

One snag: at first, over a quarter of the blind solves came back as prose, ignoring the instruction to answer in JSON. Those items were being held back for a formatting failure, not because their keys were wrong. Asking the API for JSON mode fixed it.

Which model should check

At first the drafting model, Gemini 2.5 Flash, also did the blind solve. "Confirmed" then meant only that the same model got the same answer twice, and a mistake both runs share passes. So three stronger models were tried as the solver, on the same six items:

SolverJudged unsoundCost, 6 items
Gemini 2.5 Pro1 (and disagreed on 1)$0.25
Claude Sonnet 5.52$0.03
Claude Opus 5.52$0.05

Both Claude models saw that an analogy (FILTER : ?) had several defensible answers, and that SHUT and CLOSE are equally good opposites of OPEN. They were also cheaper, because they spend fewer reasoning tokens. Claude Opus 5.5 became the solver: about 1 cent a check, from a different maker than the drafter, so the two are less likely to share blind spots. It re-checked the whole bank:

OutcomeItems
Drafts requested302
Malformed or duplicate, discarded81
Items written222
Passed by Flash checking itself183
… of those, passed by Claude143
Rejected by Flash but passed by Claude4
Live: passed by Claude147
Judged unsound by Claude66
Claude picked a different answer9

So Flash checking itself let through about one item in five that a stronger, independent solver rejects, almost always as unsound rather than as simply mis-keyed. Its reasons are specific. Three examples:

Identify the next word in the series: ONE, TWO, FOUR, EIGHT, SEVEN, FIVE, ONE, TWO, FOUR, ?

TEN · EIGHT (key) · SIXTEEN · ONE

Passed by both: doubling, reduced to a digit sum (16 → 7, 32 → 5, 64 → 1, …). Sound, but labelled IQ 175, which it is not.

A square has vertices A, B, C, D in clockwise order. A point P strictly inside it is 5 from A, 5 from B and 7 from C. What is the area of the square?

62 · 37 + √1081 (key) · 37 − √1081 · 49

Passed by both, correctly: P lies on the perpendicular bisector of AB, which gives s⁴ − 74s² + 288 = 0, and only the larger root puts P inside.

What is the next number in the following sequence: 1, 4, 13, 52, 161, 644, ?

2576 · 1933 (key) · 2585 · 1936

Passed by Flash, rejected by Claude: "52 × 3 + 1 = 157, not 161 … no option follows from a consistent rule."

Claude also rejected an odd-one-out (pivot, foundation, apex, capstone) because pivot and foundation are both defensible. And it was right where a human spot check was too harsh: two cryptic clues that looked like nonsense parse after all. ABST (the opening of "abstract") plus RUSE (a trick) gives ABSTRUSE, "hard to grasp", and SENSE plus E plus C (a "sharp" consonant) is an anagram of ESSENCE.

What no solver fixes is difficulty. Sound items tend to be easier than their labels: the IQ 175 word series above is perhaps a 130 item, and two number series labelled IQ 145 share one rule, ×2 plus a counter. The labels are only starting guesses, which the recalibration is there to correct. Human review of the bank is still worth doing.

Why not generate each item on demand?

Generation belongs offline, feeding a bank.

Growing the bank on a budget

The bank starts with an initial build of about 300 drafts, 25 for each domain and level, run as four parallel workers. After that it grows by a weekly job with a budget of $10 a month. The aim is to add items where they help distinguish the people who actually take the test:

  1. The server reports every live item's current, recalibrated difficulty and response count, and the final ability of every finished sitting of five or more items.
  2. The planner sorts these into levels at whole values of θ from −1 to 6 (IQ 85 to 190). Demand at a level is the number of recent candidates who ended there, each weighted by recency (the weight falls with a 90-day time constant), on top of an even prior strong enough that a few candidates cannot swing it. A level's target, per domain, is 10 items plus 80 times its share of demand, raised by half where items average over 40 responses each, to limit exposure.
  3. This week's spend is what is left of the month's $10, spread over the weeks left in the month. Unspent money does not roll over. Drafts go one at a time to the domain and level with the largest shortfall, allowing for the share of drafts that fail the check. When nothing is short, nothing is spent.
  4. The drafter writes those slots, each checked by Claude, stops at the budget, and logs OpenRouter's own cost figures to a ledger, from which the planner learns the real cost per draft. The bank is then loaded on the server; existing items keep their calibration.

The first dry run, with one finished sitting, allowed $3.33 for the week and planned 95 drafts, weighted toward the hard mathematical levels, where the bank was thinnest.

Scores, norms and the scores candidates report

The number the test reports is labelled provisional, because nothing yet ties its scale to a population. Each item's difficulty started as the drafting model's target, and the recalibration moves it relative to the candidates who happen to take the test. With people from high-IQ societies taking it, an item "for IQ 145" may simply be one that half of them get right. Calling 100 + 15θ an IQ would require norms.

Norms come from overlap with tests that have them. So the first page asks, optionally, for scores on other tests, with year: SAT (split at the April 1995 recentering), GRE (split at August 2011), ACT, LSAT, GMAT, Miller Analogies, WAIS, Stanford-Binet, Raven's, Cattell, the Mensa admission test, the Mega, Titan, Ultra and Power tests, or any other. They are stored with the sitting, without a name. An export lists each finished sitting's θ beside the scores reported, so once enough people report the same established test, θ can be mapped onto its scale. That is the step that would turn the provisional number into a normed one.

Can a test like this reach the Mega level?

No, and not for want of items. One in a million is about IQ 176, θ ≈ 4.75. Four things stand in the way:

  1. No item is that hard. The labels are the drafting model's guesses, and the sound items run easier than labelled. Of 26 items labelled IQ 175, Claude passed 13; the one shown above is not near that level.
  2. Every live item can be solved by an AI. An item goes live only if Claude solves it blind, so anyone taking an unproctored online test can paste each item into a chatbot. At the top of the scale that matters more than anything else: a top score would show who used AI.
  3. Four options put a floor under every score. A guess is right a quarter of the time, and the two-parameter model ignores guessing, so lucky guesses on hard items inflate the estimate most where it matters most. Hoeflin's Mega and Titan tests used free-response problems for this reason.
  4. The top cannot be calibrated from the people who come. Calibrating an item at θ = 5 needs candidates near θ = 5, and there are few of them in the world.

The arithmetic itself would allow it: twenty genuinely hard items at the cutoff, with discrimination 1.5, would measure there to about ±0.3 θ (±4 IQ points). But a short test cannot do it. With five such items the error is about ±0.57 θ at best (±0.45 even if every item discriminated as sharply as a = 2) and nearer ±0.7 in practice, and the base rate then dominates. Suppose applicants average IQ 145 with a spread of 10 points, and the test measures to ±9 points: only about one in eight of those who pass a one-in-a-million cutoff would truly be above it.

A test that can reach that level would need to be proctored, so that no AI can help; free-response, so that guessing does not count; long enough at the top, perhaps as a final stage given only to those who pass an online screen; built from problems whose answers are guaranteed, for instance generated by programs from proven problem types rather than drafted freely; and anchored on people with established high-range scores, as Hoeflin anchored his norms. The online test here could serve as that first-stage screen, where it only needs to work up to about IQ 145.

Where it stands

Source: ~/projects/proctor — proctor.cpp (the CGI program and admin commands), itemgen.cpp (drafting and blind solve), bankplan.cpp and bankgrow.sh (budgeted growth), web/ (the test page).