Simulation
Set to an error bar of 0.20. Delivered at 0.21 once judge harshness is corrected at each level.
Simulation: 15,000 decisions on judge data from our own runs, seed 20260908. With harshness left in, the same run came in at 1.16. Error bars in logits.
Final read after 139 questions: measured level 12.40, error bar 0.153 logits (one standard error).
AIM measures the judge and locks the questions. A move bigger than its error bar points at the model. Band: a recorded session on a scripted test performer, 139 locked questions, 2026-08-27.
Judge measured. Questions locked.
Judges change under the same name. In one published study, GPT-4's accuracy at telling prime from composite numbers fell from 84% to 51% between March and June 2023 (Chen, Zaharia and Zou, 2024).
- Model: free to move.
- Judge: measured.
- Questions and keys: locked, with a fingerprint.
- Grading prompt: held constant. Its record is pending.
Your own task mix in production is outside this ruler.
In a simulation on judge telemetry from our own runs, a run set to a standard error of 0.20 came in at 1.16 with judge harshness left in. With it corrected, 0.21.1
Simulation.
Each level, its own error bar.
Each level carries its own error bar. No average sits on top, so a failure at one level stays visible.
Pick any score. It resolves to one question in a locked question set, with its answer key and a fingerprint.
One of our gates failed. A 200-question wave failed the test for one shared dimension. We published it as a finding.
The run behind the 0.21, in the whitepaper
If your domain is not on the list, we say so before the run.
One ruler. Every mind.
Every mind on one ruler. Each reading has an error bar. The bar tightens as evidence comes in.
Illustration, not measured results. The figures drift and overlap; no ranking is implied.
Has the ruler moved?
Two questions to ask any ruler.
Think of a physical gauge you trust. It is traced to a standard and states its error. Those are the two checks to ask of any ruler.
Has the ruler moved?
The question set is frozen, fingerprinted and dated. Every run checks the fingerprint first. Whether the ruler holds steady over months is the next exhibit. It is not done yet.
Re-measurement run: not yet published.
Is the ruler circular?
Each level of the ruler is defined before any answer is graded. The model that writes a question never grades it. A check against outside judgment is the next exhibit. It is not done yet.
Validity study: not yet shown.
The question set is frozen, fingerprinted and dated. Every run checks the fingerprint first. Whether the ruler holds steady over months is the next exhibit, and it is not done yet.
Re-measurement run: not yet published.2
Is the ruler circular?
Defined before anything is scored.
Each level is fixed before any answer is graded. New questions stay sealed, so no model could have learned them. For new questions, the writer, the attacker and the judges come from separate companies. Each judge is checked against the others.
The judges for levels 7 to 9 are a certified set of 12. Their credentialing run is in progress, with no results yet.
A check against outside human experts: still ahead.
Measured, not averaged.
Each level carries its own error bar. No average sits on top. A failure at one level stays visible.
Pick any score. It resolves to one question in a locked question set, with its answer key and a fingerprint.
If your domain is not on the list, we say so before the run.
Know the cheapest judge that can.
Two judges have published ceilings. Today you pick the judge and see how far up the ruler it grades reliably. The run stops when the error bar is tight enough, or at the question limit you set. Hand-off to a stronger judge is in build.
Cheapest judge first. A stronger judge on stand-by, in build. Stops you set: the widest error bar, the most questions, the most spend.3
In build.
Open to reviewers. Closed to training.
- Engine
- Open to reviewers under Apache-2.0, so they can check the method.
- Question banks
- Closed, so the questions stay out of public training data. Each bank has a sha256 fingerprint.
If you run an eval loop, AIM is one step in it. It replaces nothing you built.
One outside view.
“I've always stressed the importance of ethics in persuasion, and Dr. Matt Barney's AI assessment tool brings unprecedented scientific rigor to this domain. I am optimistic that his method holds immense promise in proactively preventing the misuse of persuasion techniques, both by people and emerging technologies, and augmenting their long-term use correctly”
No score without its error bar.
That is the standard we hold every number here to. Each of our own measures names its status beside it. Today each is still in calibration.
Free to copy: the rule we use to decide if a judge's score counts. One paragraph. No form. Read the rule
Tell us which number you do not trust.
You will get a short note from Matt when a measurement slot opens. Nothing else, no drip sequence.
We store what you send with the few providers our Privacy Policy names, and use it to reply to you. We do not sell it. Email matt@xlnc.co and we delete it. Read our Privacy Policy.
- Ceiling rule met. Criterion: a judge's ceiling is the highest level it passes before a break, capped at 13; passes above a break do not count, and the level-14 questions are exploratory and never set a ceiling. Unit: one judgment on one offered question. Source: the reasoning-ceiling probe run of 2026-08-19, checked by our internal quality review.
- The simulation used 15,000 judge decisions, dated 2026-09-09. Method and results are in the whitepaper. Whitepaper
- Fifteen artifacts sit under sha256 fingerprints, checked at the start of every run.
- Two ceilings are published. Both sit at level 11, with a stated condition. Ceiling rule met
- Method: Chapter 3 of a book on models and metrology, by Barney and Barney. Edited by Fisher and Pendrill. Published by De Gruyter in 2024, pages 103 to 132. doi.org/10.1515/9783111036496-003