← Crucible results

About Crucible

This page was created and written by or with the help of AI.

Crucible is a small benchmark harness for testing models on the work people actually hand them: building and fixing code, editing writing, answering everyday questions, reasoning through a schedule, diagnosing a leak. The aim is to answer a practical question: which model, through which tool, on which hardware, is good enough for this job?

How a run works

1. Test. Each test is a YAML file: the prompt, a rubric of weighted criteria, and anything it needs, such as a seeded repo, fixture files or a mock email-and-calendar server.
2. Harness. The harness hands the prompt to the model. Chat tests always use direct, a single API call with no tools. Agentic tests use a coding agent, opencode or pi, that can read and write files, run commands and browse. The same model can do very differently depending on the agent, which is why the Agentic results let you switch between them.
3. Model. Local models run through LM Studio or Ollama; cloud models run through OpenRouter as a reference point.
4. Run folder. Every run writes a folder with the prompt, the full transcript, the files the model produced, and the metrics: time, time to first token, tool calls, tokens and cost.
5. Score. Each run is scored 0–10 against its rubric, by a person or by an AI judge (an agent score). Some tests have a critical failure, like wrong interest math or an invented news story, that caps the score.
6. Export. Scored runs are flattened into one JSON file that drives the results page.

Sections and areas

Every test belongs to one section, Chat or Agentic, and the sections are never mixed: not in a run, not in a score. Within each, a test's area is the kind of work it covers, and every test ID is its area's code and a number.

Tiers

Tests are also sorted by how hard they are, so a small model isn't judged on work meant for a frontier one.

• Spark: the basics. Does it load, follow a format, call a tool, read code and make a small correct change? This is a gate: a model that fails Spark isn't ranked.
• Temper: real everyday work you'd actually use. This is the headline score.
• Forge: long, multi-step work you'd normally need a frontier model for.

Programmatic checks

Writing, math & logic, geography, history, general knowledge, evals, coding, everyday tests are scored with no judge at all: each rule below runs on the model's final answer, passes or fails, and is worth a fixed number of points. Anyone running the same answer through Crucible gets the same score. The answer that was checked is saved with the result, so every score can be audited later. Tests that need judgment (does the code actually work, is the advice safe) use an AI judge instead, with deterministic checks wherever a rule can be stated exactly.

Log in and recover fully deterministic CODE-4

• The app's audit log shows a login as testuser whose session then served the dashboard and was not logged out afterwards. Without it the score is capped at 3.
• The audit log shows the documented reset flow for testuser (reset requested, password reset with the token, then a login), in that order.
• server.js, public/, SKILL.md and accounts.json are exactly as seeded (git diff against the seeded commit), and the audit log shows the app was used. Otherwise the score is capped at 3.
• The last FAILURE: line names the seeded failure mode, wrong password (the skill's name for it is enough).

Grade a palindrome function fully deterministic EVAL-1

• The grade is close to the reference grade: full points for an exact match, fewer the farther off it is (a bell curve), none when it is missing.

Grade a two-sentence summary fully deterministic EVAL-2

• The grade is close to the reference grade: full points for an exact match, fewer the farther off it is (a bell curve), none when it is missing.

Grade a binary search answer fully deterministic EVAL-3

• The grade is close to the reference grade: full points for an exact match, fewer the farther off it is (a bell curve), none when it is missing.

Capitals, not the largest city fully deterministic GEOG-1

• Each country line names its capital and nothing else: Australia: Canberra, Canada: Ottawa, Brazil: Brasília, Nigeria: Abuja, Türkiye: Ankara, Switzerland: Bern; one item per country.

Landlocked countries fully deterministic GEOG-2

• Lists each of the 11 landlocked countries on the Landlocked line, and not on the Not landlocked line: Bolivia, Kazakhstan, Austria, Laos, Paraguay, Serbia, Ethiopia, Uzbekistan, Zambia, Hungary, Mongolia; one item per country.
• Lists each coastal country on the Not landlocked line, and not on the Landlocked line: Croatia, Eritrea, Mozambique, Vietnam, Slovenia, Jordan, Bosnia and Herzegovina, Togo, Iraq, DR Congo, Belgium; one item per country.
• The doubly landlocked country is Uzbekistan.

Geography trick questions fully deterministic GEOG-3

• Answer 1 is Brazil.
• Answer 2 is Saudi Arabia.
• Answer 3 is Reykjavík (Nuuk's Greenland isn't sovereign).
• Answer 4 is France.
• Answer 5 is Maine.
• Answer 6 is Lesotho.

Put events in order fully deterministic HIST-1

• Puts the events in order, oldest first: Magna Carta, Gutenberg Bible, French Revolution, First World War, Moon landing, Berlin Wall; one item per pair of events (15).
• Gives each event its year, on its own line: Magna Carta 1215, Gutenberg Bible 1450-1456, French Revolution 1789, First World War 1914, Moon landing 1969, Berlin Wall 1989; one item per event.

Fact-check twelve history claims fully deterministic HIST-2

• Lists each inaccurate claim (2, 3, 6, 7 and 10) in the ERRORS: block; one item per line.
• Corrects each one to its value alone: 2 to (King) John, 3 to 1492, 6 to 1815, 7 to 1912, 10 to 1989; one item per correction.
• Leaves the accurate claims (1, 4, 5, 8, 9, 11 and 12) out of the ERRORS: block; one item per claim.

Anachronisms in an 1863 letter fully deterministic HIST-3

• The ANACHRONISMS line includes c (Gettysburg, July), f (telephone, 1876), g (barbed wire, 1874), j (West Virginia, June 20) and k (draft riots, July); one item per letter.
• The ANACHRONISMS line leaves out a, b, d, e, h, i (all possible by June 10, 1863); one item per letter.

Fire truck color fully deterministic KNOW-1

• Says US fire trucks are red (lime-yellow or white variants can be mentioned too).
• Red is the first color it names (other colors mentioned afterward as variants are fine).
• Answers in a sentence or two (at most 60 words).

News briefing from feeds fully deterministic KNOW-4

• Cites each of the nine sources from today exactly once: S1, S2 and S9 together in one paragraph (the levy), S3 and S4 together (the fire), S5 and S6 together (the rally), and S7 and S8 each in a paragraph of their own; one item per source.
• Leaves out the week-old school board item: S10 is not cited, and neither Okoro nor "school board" appears.
• Marks the Bugle column as opinion in its citation, as in [S9 opinion].
• In the paragraph found by its citations: the levy has $41.6 million and the 7-2 vote; the fire has the corrected count of one firefighter; the rally gives all three estimates, each attributed (about 300 to the Ledger or its reporters, more than 1,000 to organizers, 400 to police); the teacher pay bill is "expected to sign" and not signed; the rate story has 3.75% and 0.25. One item per fact.
• Every number in the briefing appears in the sources (whole numbers under 11 aside), so no figure was invented or altered.

Recipe scaling and a discount fully deterministic MATH-1

• Answer 1 is 750 g.
• Answer 2 is $71.91.

Household money math fully deterministic MATH-2

• Answer 1 is $5,636.36 (monthly compounding, 36 periods).
• Answer 2 is 87 boxes (2,035 sq ft, rounded up).
• Answer 3 is $47.14.
• Answer 4 is $54.34 each (tip on the pre-tax amount, added to the taxed total).

Competition counting and probability fully deterministic MATH-3

• Answer 1 is 3809.
• Answer 2 is 49.
• Answer 3 is 2731.
• Answer 4 is 37/216.
• Answer 5 is 13/27.

Seating logic puzzle fully deterministic MATH-4

• The full seating is right, one item per chair (name and colour): Alice green in 1, Bob blue in 2, Dave red in 3, Carol yellow in 4.
• The last line names Dave as the person in chair 3.

Meeting slots from two calendars fully deterministic TASK-1

• slots.json parses as a JSON object (a code fence around it is tolerated, prose around it is not).
• For each of the 8 days Aug 31 to Sep 7, valid_slots lists every start where a 60-minute meeting fits for Alex and Jamie (half hours inside 9-5, computed from the fixture) and none that overlaps an event; one item per day.
• None of the starts in valid_slots or top3 overlaps an event of either person. Any overlap caps the score at 5.
• Every start listed is on the half hour inside 9-5 within the window (nothing at 16:30, 08:30, 09:15 or after Sep 7).
• top3 is a list of exactly three entries.
• Every top3 start is one of the true slots, with no repeats. The order of the three is not scored.
• Every top3 entry has a non-empty explanation string.

Morning brief with Spanish practice fully deterministic TASK-2

• The reply has a ## heading for Emails, Calendar, News and Spanish practice, in that order; one item per section.
• The Spanish practice section is Spanish: Spanish function words (de, la, que, el, en...) are at least 15% of its words and more than twice the share of English ones. If it is not, the score is capped at 5.
• The Emails, Calendar and News sections are English (not Spanish by the same function-word test).
• Each of yesterday's four email subjects in fixtures/mock-data is mentioned (the title, or most of its distinctive words together; a summary naming the Q3 expense report, Jamie's notes or the revised proposal counts), one item per email.
• Each of the three calendar events on 2026-09-02 (Planning, Vendor demo, Budget meeting) is mentioned, one item per event.
• At least one of the three headlines in fixtures/mock-data is mentioned.

Morning brief, prioritized fully deterministic TASK-3

• The reply has a ## heading for Weather, Outstanding work, Emails, Calendar, News and Top 3 things to do today, in that order; one item per section.
• The Top 3 section holds exactly three list items (sub-bullets under an item do not count).
• The brief mentions each must-act item: the 10:00 Vendor demo overlapping the 9:30 Planning (named together with a word like conflict or overlap), the Q3 expense report due at 5 PM, and the client's unanswered question about the proposal timeline. If none of the three is mentioned, the score is capped at 5.
• Every figure of 11 or more in the reply appears in the mock weather, email, calendar or news data (list numbers and small counts are ignored; whole hours also count as 24-hour times).

Am I free at 3:30? fully deterministic TASK-4

• The ANSWER: line says free (the Design review ends at exactly 3:30, which is not a conflict).
• The reply says which time it checked: PM, this afternoon or today (3:30 AM has already passed).
• The run called get_current_time and list_events (read from the session, not the reply).

An hour with Jamie fully deterministic TASK-5

• Every OPTIONS line is a 60-minute start that is free for both people inside weekday 9-5 and after now (computed from the dataset), with no repeats. Any other option caps the score at 4.
• The OPTIONS block has two or three lines.
• At least one valid option is Thursday 13:00 or on Friday (it needs Alex's declined and transparent events read as not blocking), and no option is before Monday 8:30 AM.

An hour for a team of 16 fully deterministic TASK-6

• Every OPTIONS line is a 60-minute start free for all 16 Platform calendars inside each person's 9-5 weekday (computed from the dataset), with no repeats. Any other option caps the score at 4.
• The OPTIONS block has exactly three lines.
• No option falls on Labor Day (9/7) or on 9/10-9/11, and none ends after 16:00 Central (Marcus works 9-5 in New York).
• The run's get_freebusy calls (read from the session) named all 15 Platform calendars other than the user's own.

Proofread a customer email fully deterministic WRIT-1a

• Lists each of the three errors in the ERRORS: block as → : havent → haven't, crush → crushed, and 'items were damaged' → 'was damaged'; one item per error.
• Fixes each error in the CORRECTED: email: haven't, crushed, and 'one of the two items was damaged'; one item per error.
• Keeps the five facts in the CORRECTED: email: order 84721, three days late, 5-7 business days, the mug and its handle; one item per fact.
• Adds no apology to the CORRECTED: email (no 'sorry', 'apologize' or 'regret').
• Lists only the three real errors (havent, crush and 'items were damaged') in the ERRORS: block; a style suggestion (a comma before 'but', 'seem' or 'seems', the greeting, the en dash) listed there fails.

Proofread a literary paragraph fully deterministic WRIT-1b

• Fixes 'louder then' to 'louder than' and 'talking to fast' to 'talking too fast' in the CORRECTED: paragraph; one item each.
• Keeps these verbatim in the CORRECTED: paragraph: 'Not waiting for her, exactly.', 'That would have been too kind.', 'And she didn't know whether to laugh, or leave.' (comma included), the dash in 'Still—she was back' and 'smelled like rain, hot asphalt, and somebody's breakfast'; one item each.
• Lists at least one intentional choice (a fragment, the dash, the comma in 'laugh, or leave' or the opening imagery) in the POSSIBLY INTENTIONAL: block, not changed silently.

Proofread an economics essay fully deterministic WRIT-1c

• Fixes each planted language error with its corrected word alone: 3 "is" to "are", 6 "it's" to "its", 12 "then" to "than"; one item per fix.
• Flags each planted overstatement in the FLAGS: block, with any of the four labels: 1 (Inflation is when ... because), 4 (legally required), 8 (lazy), 11 (inevitably causes hyperinflation), 14 (proves ... completely ineffective), 18 (definitively debunked ... at all); one item per sentence.
• Leaves the accurate sentences (2, 5, 7, 9, 10, 13, 15, 16 and 17) out of the FLAGS: block; one item per sentence.

Names for one character fully deterministic WRIT-2b

• The NAMES: block has 20 numbered lines, no name twice, a reason (at least three words after a dash) on every line, and no famous or caricature name (blocklist) among the names.
• The TOP FIVE: block has exactly five names, each one in the NAMES: list.

River haiku without water fully deterministic WRIT-3

• Never uses water, waves, ripples, wet or bank in any form (riverbank, waterfall, wavy, wetland...).
• Answers with exactly three lines (a haiku, nothing around it).
• Uses at least one river image (current, stone, bend, heron, flow, valley...).

Keeping scores comparable

Every test is versioned. If a prompt or its files change, old runs of it stop counting until they're rerun; if only the rubric changes, the saved outputs are re-scored. Published scores are always on a release, a pinned list of tests and versions, so adding a test never quietly moves anyone's score. New tests start provisional and only count once they've been checked against small, mid-size and frontier models to confirm their tier.

Run it yourself

Crucible is open source. You need Python 3.9+, Node.js for the agent harnesses (opencode or pi) and, for local models, LM Studio or Ollama; cloud models run through OpenRouter.

git clone https://github.com/schaferandrew/crucible.git
cd crucible
uv sync
uv run crucible run

crucible run asks which model, what to run (everything, Chat, Agentic, an area, a tier or single tests) and, for agentic tests, which harness, then prints the one-line command to repeat it. Without uv: python3 -m venv .venv, activate it, pip install -e ..

schaferandrew/crucible on GitHub has the full setup for each harness and provider.

Share your results