Proctor project · 13 November 2026
An Adaptive Reasoning Test
A high-range test that picks each item to fit the answers so far, built in a day on shared web hosting. The items are drafted by one language model, checked by a blind solve by another, and banked rather than made on demand. This page explains how it works, what the checking catches, how the bank grows on a budget of $10 a month, why the score it reports is not yet an IQ, and why no test like it can reach the one-in-a-million level.
The test is at questrel.com/oracle/irt/.
The starting point
The design began as a sketch of a C++ server. On each request it asked Gemini to write a new item at a target difficulty, scored the candidate with item response theory, nudged the item's difficulty after each test, and sat behind an nginx proxy at /oracle/irt/. Three parts of that sketch survive unchanged: the two-parameter model, the ability estimate, and the rule that updates an item's difficulty. The rest changed, for two reasons: where the test had to run, and how reliable machine-made items turned out to be.
Why it is a CGI program
questrel.com is shared cPanel hosting: Apache on CloudLinux, with no nginx, no root access, and no long-running processes, which the host stops. A server that stays up and listens on a port has nowhere to live. Apache will, though, run a program once per request (CGI), and the host has a C++ compiler. So proctor is a CGI program. It starts, answers one request, and exits, and everything it must remember is in a SQLite database outside the web folder: the items, each sitting, and each answer.
Two hosting details cost time and are worth knowing:
- Apache's suexec refuses to run a CGI program in a directory that its group can write to. The visitor gets a bare 500 error and the error log says nothing. The directory must be mode 755.
- Turning on URL rewriting in a subdirectory drops the site's own http-to-https rule. The obvious fix, inheriting the parent's rules, sends visitors to the site's home page instead, so the rule is written out again in the subdirectory.
How a sitting works
Each item has two numbers: a difficulty b and a discrimination a, on a scale where an ability θ of 0 corresponds to IQ 100 and each unit to 15 points. The chance that someone of ability θ answers it correctly is
and the item tells you most about θ where that chance is near one half. The measure is its information, a²P(1 − P).
- Start. The candidate may list scores on other tests (below). The server creates a session with a random 128-bit id.
- Choose. For each item, the server works out the current ability estimate from the answers so far. It takes the domain (verbal or mathematical-spatial) seen less often, ranks the unused items there by information at that estimate, and picks at random among the best three, so that the same few items are not shown to everyone.
- Answer. The server records when it sent the item and when the answer came back, so response times cannot be faked. An answer under three seconds is counted as fast. An item already sent comes back again on reload, so it cannot be skipped by refreshing.
- Score. After up to 20 items, ability is estimated by maximum a posteriori: Newton steps on the likelihood with a normal prior of mean 0 and standard deviation 2. The prior keeps the estimate finite when every answer is right or every answer is wrong. The standard error comes from the curvature at the estimate. The page reports 100 + 15θ with a 95% range.
- Recalibrate. If the sitting had at least five answers, each item's difficulty moves a little: easier when it was answered right more often than the model expected, harder when less often. The step shrinks as the item collects responses. Discrimination stays at 1.5 for now.
Nothing is generated while a candidate waits, and the answer key never leaves the server. A visitor can start at most 30 sittings a day.
Where the items come from
A separate program, itemgen, drafts items offline on a desktop machine through OpenRouter, so no API key sits on the public host. For each domain and each target level from θ = 0 to 5 (IQ 100 to 175) it asks Gemini 2.5 Flash for an item of a given kind, as JSON. The kinds are:
- Verbal: analogy, odd one out, verbal series, word construction, cryptic definition, and relations between word pairs.
- Mathematical-spatial: number series, combinatorics, plane geometry, probability, solids described in words, and arithmetic deduction.
The prompt rules out figures, published puzzles, trick wording and obscure knowledge, and asks for exactly one defensible answer and a worked solution. The options are then shuffled, because models like to put the right answer first.
Each draft then goes to a blind solve: a second model, with reasoning turned up, sees only the question and the options. It must pick one and say whether the item is sound: exactly one option defensible by a clear rule, no other equally defensible, and unambiguous wording. The item goes live only if the solver picks the drafted key and judges it sound; otherwise it is kept, inactive, with the solver's one-line reason. The very first draft shows why a check is needed:
Which of the following best continues the series? READ, BOOK, LIBRARY, ARCHIVE, …
ENCYCLOPEDIA · INTERNET (drafted key) · MUSEUM · DATABASE
The blind solve chose DATABASE. Neither answer is defensible over the other.
One snag: at first, over a quarter of the blind solves came back as prose, ignoring the instruction to answer in JSON. Those items were being held back for a formatting failure, not because their keys were wrong. Asking the API for JSON mode fixed it.
Which model should check
At first the drafting model, Gemini 2.5 Flash, also did the blind solve. "Confirmed" then meant only that the same model got the same answer twice, and a mistake both runs share passes. So three stronger models were tried as the solver, on the same six items:
| Solver | Judged unsound | Cost, 6 items |
|---|---|---|
| Gemini 2.5 Pro | 1 (and disagreed on 1) | $0.25 |
| Claude Sonnet 5.5 | 2 | $0.03 |
| Claude Opus 5.5 | 2 | $0.05 |
Both Claude models saw that an analogy (FILTER : ?) had several defensible answers, and that SHUT and CLOSE are equally good opposites of OPEN. They were also cheaper, because they spend fewer reasoning tokens. Claude Opus 5.5 became the solver: about 1 cent a check, from a different maker than the drafter, so the two are less likely to share blind spots. It re-checked the whole bank:
| Outcome | Items |
|---|---|
| Drafts requested | 302 |
| Malformed or duplicate, discarded | 81 |
| Items written | 222 |
| Passed by Flash checking itself | 183 |
| … of those, passed by Claude | 143 |
| Rejected by Flash but passed by Claude | 4 |
| Live: passed by Claude | 147 |
| Judged unsound by Claude | 66 |
| Claude picked a different answer | 9 |
So Flash checking itself let through about one item in five that a stronger, independent solver rejects, almost always as unsound rather than as simply mis-keyed. Its reasons are specific. Three examples:
Identify the next word in the series: ONE, TWO, FOUR, EIGHT, SEVEN, FIVE, ONE, TWO, FOUR, ?
TEN · EIGHT (key) · SIXTEEN · ONE
Passed by both: doubling, reduced to a digit sum (16 → 7, 32 → 5, 64 → 1, …). Sound, but labelled IQ 175, which it is not.
A square has vertices A, B, C, D in clockwise order. A point P strictly inside it is 5 from A, 5 from B and 7 from C. What is the area of the square?
62 · 37 + √1081 (key) · 37 − √1081 · 49
Passed by both, correctly: P lies on the perpendicular bisector of AB, which gives s⁴ − 74s² + 288 = 0, and only the larger root puts P inside.
What is the next number in the following sequence: 1, 4, 13, 52, 161, 644, ?
2576 · 1933 (key) · 2585 · 1936
Passed by Flash, rejected by Claude: "52 × 3 + 1 = 157, not 161 … no option follows from a consistent rule."
Claude also rejected an odd-one-out (pivot, foundation, apex, capstone) because pivot and foundation are both defensible. And it was right where a human spot check was too harsh: two cryptic clues that looked like nonsense parse after all. ABST (the opening of "abstract") plus RUSE (a trick) gives ABSTRUSE, "hard to grasp", and SENSE plus E plus C (a "sharp" consonant) is an anagram of ESSENCE.
What no solver fixes is difficulty. Sound items tend to be easier than their labels: the IQ 175 word series above is perhaps a 130 item, and two number series labelled IQ 145 share one rule, ×2 plus a counter. The labels are only starting guesses, which the recalibration is there to correct. Human review of the bank is still worth doing.
Why not generate each item on demand?
- Waiting. A draft with its check takes about 30 seconds, and only about half of drafts end up live (147 of 302), so a good item takes about a minute on average and sometimes several. Over 20 items that is a quarter of an hour or more of watching a spinner, and shared hosts cut off long requests.
- Keys. Of the items that came back well-formed, a third were unsound or mis-keyed in the stronger solver's judgement. Without a check, a 20-item test would carry several such items; with one, a candidate waits for the rejects to be redrafted. A banked item can also be reviewed by people, and dropped once strong candidates keep failing it; an item made on demand is used once and thrown away before anyone can notice it is broken.
- Difficulty. A one-off item never gets a second response, so its difficulty is the model's guess forever, and the reported precision means nothing.
- Cost and exposure. Each sitting would make 20 to 40 paid calls (about a dollar), the key would sit on the public host, and an outage or a quiet model update would change the test without notice.
Generation belongs offline, feeding a bank.
Growing the bank on a budget
The bank starts with an initial build of about 300 drafts, 25 for each domain and level, run as four parallel workers. After that it grows by a weekly job with a budget of $10 a month. The aim is to add items where they help distinguish the people who actually take the test:
- The server reports every live item's current, recalibrated difficulty and response count, and the final ability of every finished sitting of five or more items.
- The planner sorts these into levels at whole values of θ from −1 to 6 (IQ 85 to 190). Demand at a level is the number of recent candidates who ended there, each weighted by recency (the weight falls with a 90-day time constant), on top of an even prior strong enough that a few candidates cannot swing it. A level's target, per domain, is 10 items plus 80 times its share of demand, raised by half where items average over 40 responses each, to limit exposure.
- This week's spend is what is left of the month's $10, spread over the weeks left in the month. Unspent money does not roll over. Drafts go one at a time to the domain and level with the largest shortfall, allowing for the share of drafts that fail the check. When nothing is short, nothing is spent.
- The drafter writes those slots, each checked by Claude, stops at the budget, and logs OpenRouter's own cost figures to a ledger, from which the planner learns the real cost per draft. The bank is then loaded on the server; existing items keep their calibration.
The first dry run, with one finished sitting, allowed $3.33 for the week and planned 95 drafts, weighted toward the hard mathematical levels, where the bank was thinnest.
Scores, norms and the scores candidates report
The number the test reports is labelled provisional, because nothing yet ties its scale to a population. Each item's difficulty started as the drafting model's target, and the recalibration moves it relative to the candidates who happen to take the test. With people from high-IQ societies taking it, an item "for IQ 145" may simply be one that half of them get right. Calling 100 + 15θ an IQ would require norms.
Norms come from overlap with tests that have them. So the first page asks, optionally, for scores on other tests, with year: SAT (split at the April 1995 recentering), GRE (split at August 2011), ACT, LSAT, GMAT, Miller Analogies, WAIS, Stanford-Binet, Raven's, Cattell, the Mensa admission test, the Mega, Titan, Ultra and Power tests, or any other. They are stored with the sitting, without a name. An export lists each finished sitting's θ beside the scores reported, so once enough people report the same established test, θ can be mapped onto its scale. That is the step that would turn the provisional number into a normed one.
Can a test like this reach the Mega level?
No, and not for want of items. One in a million is about IQ 176, θ ≈ 4.75. Four things stand in the way:
- No item is that hard. The labels are the drafting model's guesses, and the sound items run easier than labelled. Of 26 items labelled IQ 175, Claude passed 13; the one shown above is not near that level.
- Every live item can be solved by an AI. An item goes live only if Claude solves it blind, so anyone taking an unproctored online test can paste each item into a chatbot. At the top of the scale that matters more than anything else: a top score would show who used AI.
- Four options put a floor under every score. A guess is right a quarter of the time, and the two-parameter model ignores guessing, so lucky guesses on hard items inflate the estimate most where it matters most. Hoeflin's Mega and Titan tests used free-response problems for this reason.
- The top cannot be calibrated from the people who come. Calibrating an item at θ = 5 needs candidates near θ = 5, and there are few of them in the world.
The arithmetic itself would allow it: twenty genuinely hard items at the cutoff, with discrimination 1.5, would measure there to about ±0.3 θ (±4 IQ points). But a short test cannot do it. With five such items the error is about ±0.57 θ at best (±0.45 even if every item discriminated as sharply as a = 2) and nearer ±0.7 in practice, and the base rate then dominates. Suppose applicants average IQ 145 with a spread of 10 points, and the test measures to ±9 points: only about one in eight of those who pass a one-in-a-million cutoff would truly be above it.
A test that can reach that level would need to be proctored, so that no AI can help; free-response, so that guessing does not count; long enough at the top, perhaps as a final stage given only to those who pass an online screen; built from problems whose answers are guaranteed, for instance generated by programs from proven problem types rather than drafted freely; and anchored on people with established high-range scores, as Hoeflin anchored his norms. The online test here could serve as that first-stage screen, where it only needs to work up to about IQ 145.
Where it stands
- Live: 147 items at questrel.com/oracle/irt/, 62 verbal and 85 mathematical-spatial, each passed by Claude Opus 5.5 as correctly keyed and sound, from 302 drafts.
- Cost so far: about $9.50 in API charges: $6.80 for drafting and the first checks with Gemini 2.5 Flash, $2.40 for the Claude re-check of the bank, and $0.30 comparing solvers. It is outside the monthly budget.
- Growing: weekly from 16 November, within $10 a month, aimed at the levels real candidates need.
- Known weaknesses: difficulty labels run high; discrimination is fixed at 1.5; the model has no guessing term; the scale is not normed; an unproctored candidate can use AI. Human review of the bank and a three-parameter model are the next improvements.
Source: ~/projects/proctor — proctor.cpp (the CGI program and admin commands), itemgen.cpp (drafting and blind solve), bankplan.cpp and bankgrow.sh (budgeted growth), web/ (the test page).