Rust coding eval · omp on OpenRouter · October 2026

Mercury 3 vs GPT-6 Luna and Claude Haiku 5.5 on 10 real Rust changes

NOTE: very low n, and highly non-scientific. Each model made one attempt at each of 10 cases, in one codebase. Read this as a structured anecdote, not a benchmark or a ranking. Single differences such as one defect more or less can be chance.

Wall-clock times were measured while other jobs ran on the same machine, without controlling ambient load. They are rough relative indications only.

This is AI generated with light supervision.

TLDR

What was tested

The point of the eval was to see how Mercury 3 holds up against two established models, GPT-6 Luna and Claude Haiku 5.5, on real coding work. The cases are 10 merged PRs from scode/treeward, a file-tree checksumming CLI with several thousand lines of Rust, a SPEC.md, and strict integrity rules. They cover a CLI flag, a fingerprint security bug, partial-write recovery, a TOCTOU race, a timestamp edge case, terminal-escape hardening, non-UTF-8 handling, symlink semantics, parse validation, and one large feature (status --diff, about 1.3k lines in the original). Each model received the same task text for a case, written in the maintainer's voice, which describes the problem and the required behavior but not the implementation. Each run worked in its own copy of the repo at the PR's parent commit, without git history.

Snapshottreeward at the PR'sparent commitgit archive, no historyfresh 1-commit repo Mercury 3 GPT-6 Luna Claude Haiku 5.5 only --model differs thinking high, no task tool Mechanical checksown cargo test · fmt · clippy · dprintheld-out acceptance tests, validatedagainst the real PR and its parent Defect search, reviewing agentone Opus agent per case, sees A / B / Cno model, cost or timing shownreproduces serious defects in copies
Every run went through the same pipeline, from byte-identical snapshots per case. Runs went through a queue of 12 at a time, ordered so a case's three runs started close together: within milliseconds for c02, c03, c06 and c10, within about 2 minutes for c01, c04, c07, c08 and c09, and about 7 minutes apart for c05.

What is reported

Acceptance tests
Tests written for each case after the fact, mostly ported from the real PR's own tests, and never shown to the models. Once a run finished, the tests were copied into its tree and run. There are 81 across the 10 cases (5 to 14 per case). Every case's suite was first checked to pass in full on the real merged change. 61 of the 81 tests fail on the code before it; the other 20 are regression guards that pass on both and check that a fix does not break existing behavior. Two extra tests are reported separately and not counted: one in c05 that pins a design choice of the original (all three models failed it), and one in c06 that checks behavior the task arguably excludes (all three passed it). The reviewing agents' notes call these tests "hidden tests".
Defects
Bugs found by a reviewing agent (one Opus agent per case), each with a concrete failing scenario. The reviewing agent saw the task, the original PR's diff, and the three runs as A, B and C, with no model, cost, or timing information. Critical means a core requirement or the tool's integrity guarantee is broken. Major means a stated requirement is unmet in a realistic case, or a significant regression. Minor covers edge cases, weak error text, and small documentation drift. The definitions given to the reviewing agents list a panic on reachable input as critical, but the reviewing agents rated a defect low when the original PR has the same one; that is why a reproduced panic in Mercury's c10 run (on a corrupt ward file, shared with the original) is listed as minor. The reviewing agent reproduced every critical and major defect in a copy of the run's tree. The critical one was also reproduced independently, outside the reviewing agent, and the same scenario passes on Luna's c03 build.
Hygiene and conventions
Read mechanically from each run's final tree and diff: whether its own cargo test passes, fmt and dprint cleanliness, clippy diagnostics compared with the starting commit, and which files it touched. The repo's agent instructions ask for SPEC.md to be kept current when behavior changes, and list the checks to run before finishing. They do not mention CHANGELOG.md; the repo generates it with git-cliff at release time, so a hand edit is not how changes reach it, and the reviewing agents treated such edits as a convention slip.
Cost and time
Token usage and cost from each run's omp event stream (omp's figure at OpenRouter list prices, not a bill), and wall-clock time under uncontrolled machine load.

Results

Mercury 3GPT-6 LunaClaude Haiku 5.5
Correctness
Acceptance tests passed
of 81, across all 10 cases
81/8181/8169/81
Cases with every acceptance test passing
of 10
10109
Critical defects
found and reproduced by a reviewing agent
100
Major defects
found and reproduced by a reviewing agent
012
Minor defects found
a lower bound; see caveats
13810
Hygiene, checked mechanically
Own test suite green at the end
runs, of 10
101010
cargo fmt --check clean
runs, of 10
101010
No new clippy diagnostics
runs, of 10, against each case's starting point
101010
Repo docs dprint-clean
runs, of 10
10109
Repo conventions, read from the diff
SPEC.md updated
runs, of the 7 cases whose repo has one
7/77/77/7
Hand-edited CHANGELOG.md
the repo generates it with git-cliff
230
Edited CLAUDE.md
the repo's agent instructions
100
Test functions added
#[test] attributes on added lines
723948
Lines added / deleted
excluding the run's own summary file
+3874 / −308+2935 / −403+2735 / −281
Cost and time
Cost, all 10 cases
omp's figure at OpenRouter list prices
$0.958$0.719$1.28
Median wall time per run
under uncontrolled machine load
3.8 min10.7 min11.0 min
Total wall time
sum of the 10 runs
0.8 h1.9 h2.3 h
Output tokens708k397k1.38M
CaseMercury 3GPT-6 LunaClaude Haiku 5.5 c01--log-level flag, CLI beats RUST_LOGMercury 3 on c01-loglevel: 13 of 13 acceptance tests passed; defects found: 0 critical, 0 major, 1 minor tests 13/13 1 minorGPT-6 Luna on c01-loglevel: 13 of 13 acceptance tests passed; defects found: 0 critical, 0 major, 1 minor tests 13/13 1 minorClaude Haiku 5.5 on c01-loglevel: 13 of 13 acceptance tests passed; defects found: 0 critical, 0 major, 2 minor tests 13/13 2 minorc02fingerprint misses repeated editsMercury 3 on c02-fp-repeated-edits: 13 of 13 acceptance tests passed; defects found: 0 critical, 0 major, 2 minor tests 13/13 2 minorGPT-6 Luna on c02-fp-repeated-edits: 13 of 13 acceptance tests passed; defects found: 0 critical, 0 major, 2 minor tests 13/13 2 minorClaude Haiku 5.5 on c02-fp-repeated-edits: 13 of 13 acceptance tests passed; defects found: 0 critical, 0 major, 2 minor tests 13/13 2 minorc03partial init/update leaves misleading wardsMercury 3 on c03-partial-writes: 5 of 5 acceptance tests passed; defects found: 1 critical, 0 major, 1 minor tests 5/5 1 critical · 1 minorGPT-6 Luna on c03-partial-writes: 5 of 5 acceptance tests passed; defects found: 0 critical, 0 major, 0 minor tests 5/5 no defects foundClaude Haiku 5.5 on c03-partial-writes: 5 of 5 acceptance tests passed; defects found: 0 critical, 1 major, 2 minor tests 5/5 1 major · 2 minorc04file changes between listing and checksumMercury 3 on c04-list-read-race: 5 of 5 acceptance tests passed; defects found: 0 critical, 0 major, 3 minor tests 5/5 3 minorGPT-6 Luna on c04-list-read-race: 5 of 5 acceptance tests passed; defects found: 0 critical, 0 major, 1 minor tests 5/5 1 minorClaude Haiku 5.5 on c04-list-read-race: 5 of 5 acceptance tests passed; defects found: 0 critical, 0 major, 0 minor tests 5/5 no defects foundc05pre-1970 directory mtimes rejectedMercury 3 on c05-dir-mtime: 6 of 6 acceptance tests passed; defects found: 0 critical, 0 major, 1 minor tests 6/6 1 minorGPT-6 Luna on c05-dir-mtime: 6 of 6 acceptance tests passed; defects found: 0 critical, 0 major, 0 minor tests 6/6 no defects foundClaude Haiku 5.5 on c05-dir-mtime: 6 of 6 acceptance tests passed; defects found: 0 critical, 0 major, 0 minor tests 6/6 no defects foundc06control characters leak to stderrMercury 3 on c06-stderr-escape: 9 of 9 acceptance tests passed; defects found: 0 critical, 0 major, 1 minor tests 9/9 1 minorGPT-6 Luna on c06-stderr-escape: 9 of 9 acceptance tests passed; defects found: 0 critical, 0 major, 1 minor tests 9/9 1 minorClaude Haiku 5.5 on c06-stderr-escape: 9 of 9 acceptance tests passed; defects found: 0 critical, 0 major, 0 minor tests 9/9 no defects foundc07non-UTF-8 symlink targetsMercury 3 on c07-symlink-utf8: 6 of 6 acceptance tests passed; defects found: 0 critical, 0 major, 0 minor tests 6/6 no defects foundGPT-6 Luna on c07-symlink-utf8: 6 of 6 acceptance tests passed; defects found: 0 critical, 1 major, 1 minor tests 6/6 1 major · 1 minorClaude Haiku 5.5 on c07-symlink-utf8: 6 of 6 acceptance tests passed; defects found: 0 critical, 0 major, 0 minor tests 6/6 no defects foundc08root .treeward that is not a regular fileMercury 3 on c08-root-ward-type: 5 of 5 acceptance tests passed; defects found: 0 critical, 0 major, 1 minor tests 5/5 1 minorGPT-6 Luna on c08-root-ward-type: 5 of 5 acceptance tests passed; defects found: 0 critical, 0 major, 1 minor tests 5/5 1 minorClaude Haiku 5.5 on c08-root-ward-type: 5 of 5 acceptance tests passed; defects found: 0 critical, 0 major, 1 minor tests 5/5 1 minorc09malformed sha256 in ward filesMercury 3 on c09-sha256-validate: 5 of 5 acceptance tests passed; defects found: 0 critical, 0 major, 0 minor tests 5/5 no defects foundGPT-6 Luna on c09-sha256-validate: 5 of 5 acceptance tests passed; defects found: 0 critical, 0 major, 0 minor tests 5/5 no defects foundClaude Haiku 5.5 on c09-sha256-validate: 5 of 5 acceptance tests passed; defects found: 0 critical, 0 major, 1 minor tests 5/5 1 minorc10status --diff (large feature)Mercury 3 on c10-status-diff: 14 of 14 acceptance tests passed; defects found: 0 critical, 0 major, 3 minor tests 14/14 3 minorGPT-6 Luna on c10-status-diff: 14 of 14 acceptance tests passed; defects found: 0 critical, 0 major, 1 minor tests 14/14 1 minorClaude Haiku 5.5 on c10-status-diff: 2 of 14 acceptance tests passed; defects found: 0 critical, 1 major, 2 minor tests 2/14 1 major · 2 minor All 10 casestests 81/81 1 critical · 0 major 13 minortests 81/81 0 critical · 1 major 8 minortests 69/81 0 critical · 2 major 10 minor
Each cell is one run: how many of the case's acceptance tests it passed, then the defects the reviewing agent found in it. Critical defects and failed tests are in red, major defects in orange, and a cell with either is tinted.
05101520253035$0.00$0.05$0.10$0.15$0.20$0.25$0.30 wall time per run, minutes cost per run Claude Haiku 5.5 · c01-loglevel: 5.7 min, $0.043Mercury 3 · c01-loglevel: 2.4 min, $0.043GPT-6 Luna · c01-loglevel: 5.6 min, $0.027Mercury 3 · c02-fp-repeated-edits: 6.8 min, $0.186Claude Haiku 5.5 · c02-fp-repeated-edits: 32.7 min, $0.271GPT-6 Luna · c02-fp-repeated-edits: 18.7 min, $0.149Mercury 3 · c03-partial-writes: 4.7 min, $0.103GPT-6 Luna · c03-partial-writes: 16.4 min, $0.111Claude Haiku 5.5 · c03-partial-writes: 17.0 min, $0.161Mercury 3 · c04-list-read-race: 13.2 min, $0.137Claude Haiku 5.5 · c04-list-read-race: 14.7 min, $0.111GPT-6 Luna · c04-list-read-race: 10.2 min, $0.067Mercury 3 · c05-dir-mtime: 2.6 min, $0.055GPT-6 Luna · c05-dir-mtime: 11.3 min, $0.049Claude Haiku 5.5 · c05-dir-mtime: 5.6 min, $0.065Mercury 3 · c06-stderr-escape: 4.8 min, $0.095GPT-6 Luna · c06-stderr-escape: 10.0 min, $0.067Claude Haiku 5.5 · c06-stderr-escape: 13.7 min, $0.148Mercury 3 · c07-symlink-utf8: 3.2 min, $0.082GPT-6 Luna · c07-symlink-utf8: 10.2 min, $0.053Claude Haiku 5.5 · c07-symlink-utf8: 7.5 min, $0.069Mercury 3 · c08-root-ward-type: 2.7 min, $0.051GPT-6 Luna · c08-root-ward-type: 4.3 min, $0.029Claude Haiku 5.5 · c08-root-ward-type: 6.1 min, $0.064Claude Haiku 5.5 · c09-sha256-validate: 8.3 min, $0.104GPT-6 Luna · c09-sha256-validate: 16.6 min, $0.074Mercury 3 · c09-sha256-validate: 3.5 min, $0.116Claude Haiku 5.5 · c10-status-diff: 27.1 min, $0.241Mercury 3 · c10-status-diff: 4.1 min, $0.091GPT-6 Luna · c10-status-diff: 13.4 min, $0.093 Mercury 3 median GPT-6 Luna median Claude Haiku 5.5 median

Mercury 3GPT-6 LunaClaude Haiku 5.5

Cost against wall time for all 30 runs; rings mark each model's median. Wall time was not measured under controlled load, so only large differences mean anything.

Critical and major defects

All four, each reproduced by the reviewing agent in a copy of the run's tree.

critical Mercury 3 c03-partial-writes

Rollback deletes a newly created child ward whose pre-existing parent ward was already rewritten to list it

init with p/x.txt; mkdir p/n + p/n/f.txt; add r2.txt; chmod 555 root; update -> writes p/n/.treeward (created), rewrites p/.treeward (now lists n), root fails, rollback removes p/n/.treeward; p/.treeward vouches for an unwarded n and status shows 'A p/n/f.txt' instead of 'A p/n'. Same with a failing sibling q at the same depth as p.

major Claude Haiku 5.5 c03-partial-writes

Staging keeps one open NamedTempFile per changed ward, so large runs exhaust file descriptors

Tree with 1100 directories, soft RLIMIT_NOFILE 1024 (common Linux default); init -> exit 255 'wrote 0 of 1101 ... Too many open files (os error 24)'; re-running never helps. Baseline and A/B succeed.

major GPT-6 Luna c07-symlink-utf8

Moving the NotInitialized/AlreadyInitialized checks after compute_status(WardUpdate) makes those cheap precondition failures checksum the whole tree first

Uninitialized dir with a 1 GiB file, `treeward update` -> 52 s of hashing before 'Not initialized' (A/C: 0.01 s); `init` on an initialized tree with a new 1 GiB file -> 48 s before 'Already initialized'. Running `update` in the wrong large directory now hashes all of it

major Claude Haiku 5.5 c10-status-diff

Status code is right-aligned instead of left-aligned, contradicting the explicit `A new.txt` / `. file.txt` examples

Any status/verify listing with a single-char code -> ` A new.txt` instead of `A new.txt`; status lines now start with whitespace, so a script cannot tell status lines from detail lines by indentation

A further behavior is worth knowing, though the reviewing agent rated it minor: in c02, Luna and Mercury made the default status read and checksum every added or M? file. The documented default is metadata-only, so this adds time and new failure modes on large trees.

Cost and tokens

Usage is summed over every assistant model call in each run. Haiku's higher cost comes mostly from output tokens (about 3.5× Luna's). The providers split input differently: Luna and Haiku report nearly all new input as cache writes, while Mercury reports no cache writes at all. Counting uncached input plus cache writes as new input, Luna took in 1.15M tokens, Mercury 1.61M, and Haiku 2.27M. Whether Mercury's numbers reflect how its provider caches or only how it reports usage was not checked.

ModelCostPer runUncached inputCache readCache writeNew input
(uncached + cache write)
OutputModel callsTool calls
Mercury 3$0.958$0.0961.61M55.00M01.61M708k702938
GPT-6 Luna$0.719$0.0721k37.71M1.14M1.15M397k453840
Claude Haiku 5.5$1.28$0.1281k30.17M2.27M2.27M1.38M563795
All 30 runs$2.95$0.098

Per case

Each case lists the facts per run, every defect the reviewing agent found, its one-line note on each run, the exact task text, and a link to the full review.

c01-loglevel --log-level flag, CLI beats RUST_LOG

Reference PR: Add --log-level; make it and -v override RUST_LOG env var (#89). Snapshot at parent 0949eaa.

ModelAcceptance testsCriticalMajorMinorTests addedLines +/−CostTime
Mercury 313/130015+147/−12$0.0432.4m
GPT-6 Luna13/130017+201/−13$0.0275.6m
Claude Haiku 5.513/130029+220/−11$0.0435.7m

Defects found by the reviewing agent:

Reviewing agent's one-line note per run (its opinion, not a measurement):

Mercury 3
Correct, reference-like implementation, but its five tests skip the replace-not-merge direction, -vv, the warn default, global placement and help output.
GPT-6 Luna
Correct implementation with good test coverage (including global placement and the suppression direction), slightly less thorough help text and no invalid-value test.
Claude Haiku 5.5
Clean, well-documented implementation that folds -v into a single explicit level, with a solid test set including the replace-not-merge direction and exit-code checks.

Full unblinded review: reviews/c01-loglevel.md (candidate letters: A = Claude Haiku 5.5, B = Mercury 3, C = GPT-6 Luna).

Exact task text (appended to the shared preamble in the Method section):

Add an explicit `--log-level` flag, and make the command-line logging flags win over `RUST_LOG`.

Right now `-v`/`-vv` only set the *default* log level: if `RUST_LOG` is set in the environment it silently wins, so
`RUST_LOG=warn treeward -v init` prints nothing at info level. That's backwards for a CLI; whatever I type on the command
line should beat whatever happens to be in my environment.

What I want:

- A new global flag `--log-level <LEVEL>` accepting `error`, `warn`, `info`, `debug`, `trace`. Like `-C` and `-v` it
  should work as a global option (e.g. `treeward --log-level info init`).
- If `--log-level` or `-v`/`-vv` is given, it decides the level and `RUST_LOG` is ignored entirely (not merged with it).
  `-v` stays info, `-vv` and above stay debug.
- With neither flag, keep today's behavior: honor `RUST_LOG` if set, otherwise default to `warn`.
- `--log-level` together with `-v` is a usage error (non-zero exit), not "one silently wins".
- `treeward --help` should document both flags and say that they take precedence over `RUST_LOG`.
c02-fp-repeated-edits fingerprint misses repeated edits

Reference PR: bugfix: fingerprint validation misses repeated edits (#83). Snapshot at parent c8b8189.

ModelAcceptance testsCriticalMajorMinorTests addedLines +/−CostTime
Mercury 313/1300214+904/−121$0.1866.8m
GPT-6 Luna13/130027+593/−97$0.14918.7m
Claude Haiku 5.513/130024+363/−37$0.27132.7m

Defects found by the reviewing agent:

Reviewing agent's one-line note per run (its opinion, not a measurement):

Mercury 3
Strong, well-tested fix that binds full old->new ward state, but it quietly turns default `status` into a content-reading command and leaves the docs claiming otherwise.
GPT-6 Luna
Complete, well-encoded fix with good policy-matrix integration tests and honest, thorough docs for its main cost: default status now reads changed files.
Claude Haiku 5.5
Clean, reference-like fix that preserves the cheap default status, held back by thin unit-only tests, an unbound removal payload and a dprint failure.

Full unblinded review: reviews/c02-fp-repeated-edits.md (candidate letters: A = Mercury 3, B = Claude Haiku 5.5, C = GPT-6 Luna).

Exact task text (appended to the shared preamble in the Method section):

`update --fingerprint` (and `init --fingerprint`) doesn't actually guarantee what the help text promises: that if
anything changes between `status` and the update, the fingerprint won't match. It's easy to sneak a change through:

    treeward init
    echo bbbbb > file.txt          # first edit
    treeward status                # shows "M? file.txt" and a Fingerprint
    echo ccccc > file.txt          # second edit, after I reviewed
    treeward update --fingerprint <fp from above>   # succeeds, records content I never reviewed

The same thing happens with `--verify` (the file is `M` both times), with a newly added file that is edited again after
`status`, and with a symlink that is re-pointed after `status`. As long as the path and its status letter stay the same,
the fingerprint is accepted.

Please fix this so the fingerprint binds the exact state that `status` reported for each changed entry, and a later
change to any of those entries makes `init`/`update --fingerprint` fail with the existing fingerprint-mismatch error
(and write nothing).

Also make sure the fingerprint can't be fooled through file names: two different change sets must never produce the
same fingerprint, whatever characters the names contain (`|`, newlines, etc. are all legal in names).

Must keep working: `status` followed by `init`/`update --fingerprint` with matching flags and no intervening change is
accepted, for every kind of change (added/removed/modified files, directories, symlinks) under the default policy,
`--verify`, and `--always-verify`. A metadata-only touch with unchanged content must still be accepted the same way it
is today.
c03-partial-writes partial init/update leaves misleading wards

Reference PR: fix: write ward files deepest-first and report partial updates (#147). Snapshot at parent f5f2c3d.

ModelAcceptance testsCriticalMajorMinorTests addedLines +/−CostTime
Mercury 35/51013+286/−22$0.1034.7m
GPT-6 Luna5/50001+277/−65$0.11116.4m
Claude Haiku 5.55/50121+261/−33$0.16117.0m

Defects found by the reviewing agent:

Reviewing agent's one-line note per run (its opinion, not a measurement):

Mercury 3
Correct deepest-first ordering undermined by a best-effort rollback that, in a realistic case, recreates exactly the parent-vouches-for-missing-child state the task forbids.
GPT-6 Luna
Reference-equivalent deepest-first design with a more precise written count and recovery advice that works for both init and update; only nit is a hand-edited CHANGELOG.
Claude Haiku 5.5
Clean two-phase stage/commit that handles permission failures well, but holding every temp file open breaks init/update on large trees and the -v order changed.

Full unblinded review: reviews/c03-partial-writes.md (candidate letters: A = Mercury 3, B = GPT-6 Luna, C = Claude Haiku 5.5).

Exact task text (appended to the shared preamble in the Method section):

When `init` or `update` fails partway through writing ward files (permission denied, disk full, ...), it leaves the tree
in a confusing state and tells the user nothing useful about it.

Repro:

    treeward init                    # tree with file.txt
    mkdir newsubdir; echo hi > newsubdir/file2.txt
    chmod 555 newsubdir
    treeward update                  # fails with "Permission denied"
    chmod 755 newsubdir
    treeward status                  # now shows "A  newsubdir/file2.txt" instead of "A  newsubdir"

The root `.treeward` was already rewritten to say `newsubdir` is tracked, even though `newsubdir/.treeward` was never
created. So the change I reviewed ("new directory newsubdir") has turned into something different, and the error only
said "Permission denied" without mentioning that some ward files were written and others weren't.

I don't need the whole set of ward files to be written atomically (that would need a journal and I don't want one). What
I do want:

- After a failure at any point in the write phase, no ward file written by the run may describe a subdirectory whose
  own `.treeward` is missing or out of date. `status` must keep reporting the unfinished work the way it did before the
  failed run (in the repro: `A  newsubdir`). This has to hold for deeper trees too, wherever the failure happens.
- The error (still a fatal error, exit 255) must say how far the run got, as "N of M", e.g. "wrote 1 of 3 changed ward
  files", and that fixing the cause and re-running finishes the job. If the partial write means a fingerprint taken
  before the run can no longer be used, say so and tell the user what to do instead.
- Re-running after fixing the cause must converge to a clean `status`.
- Successful runs and `--dry-run` behave exactly as before.
c04-list-read-race file changes between listing and checksum

Reference PR: fix: abort when a file changes between listing and checksumming (#149). Snapshot at parent ff59c34.

ModelAcceptance testsCriticalMajorMinorTests addedLines +/−CostTime
Mercury 35/50036+278/−9$0.13713.2m
GPT-6 Luna5/50017+302/−27$0.06710.2m
Claude Haiku 5.55/50002+295/−54$0.11114.7m

Defects found by the reviewing agent:

Reviewing agent's one-line note per run (its opinion, not a measurement):

Mercury 3
Correct and complete fix with an end-to-end race test, let down by a timing-based test design and a truncated docstring.
GPT-6 Luna
Correct fix with deterministic walk-level tests covering every checksum site and the plain-status exception, marred only by deleted function docs.
Claude Haiku 5.5
Correct fix with a nicely type-enforced API and good docs, but tests only pin the checksum primitive, not the status-walk wiring.

Full unblinded review: reviews/c04-list-read-race.md (candidate letters: A = Mercury 3, B = Claude Haiku 5.5, C = GPT-6 Luna).

Exact task text (appended to the shared preamble in the Method section):

I found a TOCTOU hole in the `status` → `update --fingerprint` workflow under the default (metadata-only) policy. With a
race script I got `update --fingerprint <fp>` to exit 0 and write a ward entry whose sha256 is of content I never
reviewed.

The setup: a tree where two files have pending changes, a large one (`a.big`) and a small one (`z.txt`). I run
`treeward status`, review it, and start `treeward update --fingerprint <fp>`. While `update` is busy checksumming
`a.big`, something edits `z.txt` (new size and mtime). `update` then checksums the edited `z.txt`, records it, and the
fingerprint still matches. The existing "modified during checksumming" protection doesn't fire, because the edit landed
before `z.txt` was opened, not while it was being read. The same goes for newly added files. With `--verify` I believe
the fingerprint already notices, but the default policy is the one people use.

Expected behavior: if a file's size or mtime changes after its directory was listed but before it is checksummed, treat
that exactly like the existing concurrent-modification case. That means a fatal error (exit 255) naming the file, no
retry, and no ward file written. This applies to every command that checksums the file (`init`, `update`, `verify`,
`status --verify`/`--always-verify`), with or without `--fingerprint`. Commands that never read the file, like plain
`status` under the default policy, must not start failing on trees that are being written to.
c05-dir-mtime pre-1970 directory mtimes rejected

Reference PR: fix: stop rejecting pre-epoch and far-future directory mtimes (#146). Snapshot at parent 95e4340.

ModelAcceptance testsCriticalMajorMinorTests addedLines +/−CostTime
Mercury 36/60019+279/−17$0.0552.6m
GPT-6 Luna6/60002+141/−15$0.04911.3m
Claude Haiku 5.56/60004+160/−15$0.0655.6m

Defects found by the reviewing agent:

Reviewing agent's one-line note per run (its opinion, not a measurement):

Mercury 3
Correct, well-tested fix that keeps representable dir mtimes in the fingerprint and degrades out-of-range ones to a tagged None, at the cost of a small binding inconsistency.
GPT-6 Luna
Correct, exact, unambiguous range-safe encoding of dir mtimes with a solid workflow test, let down by a hand-edited generated CHANGELOG and a thin summary.
Claude Haiku 5.5
Correct exact-i128 encoding that keeps in-range dir fingerprints byte-identical, with precise SPEC/README/doc updates and good unit tests but only one CLI test.

Full unblinded review: reviews/c05-dir-mtime.md (candidate letters: A = Mercury 3, B = GPT-6 Luna, C = Claude Haiku 5.5).

Exact task text (appended to the shared preamble in the Method section):

A tree containing a directory with a pre-1970 mtime cannot be initialized. Restored tar archives and some synced trees
produce these, so this blocks real users. Repro:

```sh
mkdir -p t/old && touch -d '1960-01-01' t/old
treeward -C t init
# ERROR: Status error: mtime of .../t/old is before the UNIX epoch; pre-1970 timestamps are not supported   (exit 255)
```

It is also inconsistent. The same thing happens when such a directory shows up as new inside an already-warded tree
(`status`, `update`, `init`, `--dry-run` all abort), but once the directory has been recorded in its parent's ward file,
`status` passes silently even though the mtime is just as old. Directories with mtimes past the supported ceiling (after
roughly year 2262) behave the same way.

What I want:

- A directory's mtime, whatever its value, never makes `init`, `status`, `update` (with or without `--fingerprint`), or
  `verify` fail, whether the directory is new or already tracked.
- The documented limitation for regular files stays exactly as it is: a regular file with a pre-epoch or out-of-range
  mtime is still a fatal error naming the file, since its mtime is persisted in `mtime_nanos`.
- The `status` → `update --fingerprint` review workflow keeps working for changesets that contain new directories,
  including ones with such mtimes.
c06-stderr-escape control characters leak to stderr

Reference PR: fix: escape control characters in stderr diagnostics (#125). Snapshot at parent 37dfeb4.

ModelAcceptance testsCriticalMajorMinorTests addedLines +/−CostTime
Mercury 39/90013+166/−43$0.0954.8m
GPT-6 Luna9/90012+101/−3$0.06710.0m
Claude Haiku 5.59/90009+271/−62$0.14813.7m

Defects found by the reviewing agent:

Reviewing agent's one-line note per run (its opinion, not a measurement):

Mercury 3
Correct central fix at the formatter with good LF/CR CLI tests, marred by a SPEC sentence that overstates the stderr rendering's fidelity and unambiguity.
GPT-6 Luna
Minimal, correct streaming escaper with an accurate SPEC, but it duplicates the existing helper without backslash doubling, leaving ambiguous output.
Claude Haiku 5.5
Best design: a field visitor that makes the shared escape_control the sole escaper, so stderr output is clean, unambiguous and consistent with stdout, backed by thorough tests and an accurate SPEC.

Full unblinded review: reviews/c06-stderr-escape.md (candidate letters: A = Mercury 3, B = GPT-6 Luna, C = Claude Haiku 5.5).

Exact task text (appended to the shared preamble in the Method section):

The `status`/`verify` listing on stdout escapes control characters so a crafted file name can't mess with the terminal,
but stderr has no such protection. Error messages and `-v`/`-vv` logging print file names, directory names, symlink
targets and ward-file contents, and some control characters get through raw. For example, a file name containing a
newline forges extra log lines:

```sh
mkdir t && touch $'t/evil-a\nERROR: forged.txt'
treeward -C t -v init
# INFO: Checksumming .../t/evil-a
# ERROR: forged.txt              <- looks like a real treeward error line
```

A carriage return in a name similarly overwrites what the user sees on the line. The same thing happens with error
lines, e.g. the "unsupported file type" error for a FIFO with such a name, or a corrupt `.treeward` whose entry names
contain control characters.

What I want: nothing treeward prints to stderr, whether error messages or diagnostic logging at any verbosity, contains a
raw control character (C0 including CR, LF and TAB, DEL, or C1 such as U+009B) that came from a scanned name, symlink
target, ward-file field or other runtime data. The only raw newline is the one that ends each message. Show control
characters in some visible escaped form. I don't need a particular rendering, but the offending name has to stay
recognisable in the message. This should hold for every stderr diagnostic, not only the examples above. Usage errors
that clap reports while it is still parsing the command line are out of scope. Exit codes and stdout output must not
change.
c07-symlink-utf8 non-UTF-8 symlink targets

Reference PR: fix: reject non-UTF-8 symlink targets when listing a directory (#144). Snapshot at parent bf03f80.

ModelAcceptance testsCriticalMajorMinorTests addedLines +/−CostTime
Mercury 36/600012+417/−0$0.0823.2m
GPT-6 Luna6/60113+157/−8$0.05310.2m
Claude Haiku 5.56/60003+98/−0$0.0697.5m

Defects found by the reviewing agent:

Reviewing agent's one-line note per run (its opinion, not a measurement):

Mercury 3
Correct choke-point fix matching the reference, with the most thorough tests (unit, compute_status, 9 CLI cases), marred only by a hand-edited generated CHANGELOG.
GPT-6 Luna
The core fix is right, but an unnecessary reorder in ward_directory makes `update`/`init` checksum the entire tree before reporting not/already initialized.
Claude Haiku 5.5
Clean, minimal reference-equivalent fix with an accurate SPEC entry, but committed CLI coverage is init-only (status/verify/update were checked only by hand, as its summary admits).

Full unblinded review: reviews/c07-symlink-utf8.md (candidate letters: A = Mercury 3, B = GPT-6 Luna, C = Claude Haiku 5.5).

Exact task text (appended to the shared preamble in the Method section):

A symlink whose target isn't valid UTF-8 breaks treeward in a confusing way. Repro (Linux):

```sh
mkdir -p t/sub && echo hello > t/file.txt
ln -s $'bad\xff' t/sub/link
treeward -C t status          # exit 1, lists sub/link as added and prints a fingerprint
treeward -C t init --dry-run  # exit 0, looks fine
treeward -C t init            # exit 255: "ERROR: Ward file error: TOML serialization error: path contains invalid UTF-8 characters"
ls t/.treeward                # ...and the root ward file was written anyway
```

So `status` and the dry run say everything is fine, the real `init` (and `update`) then dies with an error that doesn't
say which entry is the problem, and it leaves ward files half-written. The tree can't be warded until the user finds the
link on their own.

Ward files are TOML, so such a target can't be recorded. I'm fine with that being unsupported, the way non-UTF-8 file
names already are. But it has to be reported properly:

- Every command (`status`, `verify`, `init`, `update`, `update --allow-init`, with or without `--dry-run`) fails with
  the usual fatal exit code (255). This includes a link that's already tracked and gets retargeted.
- The error names the link itself (its path, not the target) and says the problem is that the target isn't UTF-8.
- This happens before any `.treeward` file is written or modified.
- Valid UTF-8 targets, including non-ASCII and dangling ones, keep working as before.

Don't paper over it with a lossy conversion.
c08-root-ward-type root .treeward that is not a regular file

Reference PR: fix: refuse a root .treeward that is not a regular file (#148). Snapshot at parent 695ce8d.

ModelAcceptance testsCriticalMajorMinorTests addedLines +/−CostTime
Mercury 35/50014+276/−0$0.0512.7m
GPT-6 Luna5/50012+138/−12$0.0294.3m
Claude Haiku 5.55/50013+194/−2$0.0646.1m

Defects found by the reviewing agent:

Reviewing agent's one-line note per run (its opinion, not a measurement):

Mercury 3
Correct lstat-based guard with the broadest test matrix, marred by sloppy test code (unused shadowed variables, // instead of ///) and leftover exists() calls.
GPT-6 Luna
Correct and tight fix with full CLI coverage and a help-text correction, but lighter on documentation and has no unit test.
Claude Haiku 5.5
Cleanest implementation: a well-documented root_ward_present helper with clear rationale, regression coverage for the regular-file case, and an accurate summary; its own tests are slightly weaker on the valid-ward-link case.

Full unblinded review: reviews/c08-root-ward-type.md (candidate letters: A = Mercury 3, B = GPT-6 Luna, C = Claude Haiku 5.5).

Exact task text (appended to the shared preamble in the Method section):

treeward gets confused when the root `.treeward` is not a regular file. Two things I hit:

1. A `.treeward` symlink caught in a loop at the root:

   ```sh
   mkdir t && cd t && echo hi > a.txt
   ln -s loop .treeward && ln -s .treeward loop
   treeward update   # "Not initialized (use treeward init to initialize)"
   treeward init     # dies with a raw "Too many levels of symbolic links"
   ```

   So `update` sends me to `init`, and `init` then fails with an OS error.

2. A dangling `.treeward` symlink at the root (`ln -s nowhere .treeward`): `treeward update` succeeds and quietly
   replaces the symlink with a regular ward file.

An integrity tool shouldn't guess here. Whenever anything other than a regular file sits at the root `.treeward` path
(a directory, or a symlink, whether it loops, dangles, or points at a perfectly valid ward file), `init`, `update`, and
`update --allow-init` should all refuse with a fatal error (the usual fatal exit code, 255) and leave that entry
untouched. They should not follow it, read through it, or replace it. The error message should name the `.treeward`
path and say that it is not a regular file. It must not claim the tree is "Not initialized" or "Already initialized".

The normal cases must keep working as they do today: nothing at the root `.treeward` means "Not initialized" for
`update` and a fresh start for `init`, and a regular file means initialized.

Correction: the dangling-symlink repro is wrong for plain update. On the starting commit, update with a dangling .treeward link stops with "Not initialized", because Path::exists() follows the link. The silent replacement happens with update --allow-init. The models received the text as shown.

c09-sha256-validate malformed sha256 in ward files

Reference PR: fix: reject ward files with malformed sha256 fields at load time (#116). Snapshot at parent 6ddf457.

ModelAcceptance testsCriticalMajorMinorTests addedLines +/−CostTime
Mercury 35/50007+266/−46$0.1163.5m
GPT-6 Luna5/50004+246/−49$0.07416.6m
Claude Haiku 5.55/50016+305/−46$0.1048.3m

Defects found by the reviewing agent:

Reviewing agent's one-line note per run (its opinion, not a measurement):

Mercury 3
Reference-equivalent fix with the broadest spread of tests, but init on a corrupt existing root does not name the entry, the CHANGELOG edit is unwanted, and the summary misdescribes how git-cliff treats it.
GPT-6 Luna
Correct and complete, with init naming the entry and a write-side guard; tests are good but use only one malformed value at the CLI and skip subdirectories, and the hand-edited CHANGELOG is unwanted churn.
Claude Haiku 5.5
Load-boundary validation matching the reference, plus an init fix that names the entry, thorough unit and CLI tests, and an accurate summary; only a stale doc comment.

Full unblinded review: reviews/c09-sha256-validate.md (candidate letters: A = Claude Haiku 5.5, B = GPT-6 Luna, C = Mercury 3).

Exact task text (appended to the shared preamble in the Method section):

treeward accepts whatever string is in a `.treeward` file's `sha256` field. Hand-edit one to `sha256 = "abc"` (or an
empty string, or something with an escape character in it) and every command loads it happily: `status` reports the
tree clean as long as the file's mtime and size still match, `update` keeps carrying the value forward, and with
`--verify` the garbage goes into comparisons, the fingerprint, and the diff display. We already fixed the display
crash this used to cause, but the underlying problem is that a value treeward could never have written is being
trusted.

Our policy is that a corrupted ward file is a fatal error, so please treat a malformed checksum that way. treeward
always writes sha256 as exactly 64 lowercase hex characters, and anything else in a loaded ward file (wrong length,
non-hex characters, non-ASCII, and also uppercase hex, since we never write that) should make every command that loads
the file (`status` in all its modes, `verify`, `init`/`update`) fail with a fatal error (exit 255) before the value is
used for anything. The error should say which entry has the bad checksum. A well-formed lowercase checksum that simply
doesn't match the file's contents is not corruption and should keep behaving exactly as it does today.
c10-status-diff status --diff (large feature)

Reference PR: status --diff (#82). Snapshot at parent 9b21ea8.

ModelAcceptance testsCriticalMajorMinorTests addedLines +/−CostTime
Mercury 314/140039+855/−38$0.0914.1m
GPT-6 Luna14/140014+779/−114$0.09313.4m
Claude Haiku 5.52/140127+568/−21$0.24127.1m

Defects found by the reviewing agent:

Reviewing agent's one-line note per run (its opinion, not a measurement):

Mercury 3
Complete and correct, with the most thorough user-facing docs, held back by a byte-slicing panic, a Removed field named `ward_entry` that the `ward_entry()` accessor deliberately ignores, a mislabelled test, and small summary inaccuracies.
GPT-6 Luna
Functionally the cleanest: no real bugs, char-safe truncation, exact tests including a fingerprint round trip. It loses points for deleting a large amount of valuable existing documentation in status.rs and an existing README example.
Claude Haiku 5.5
The diff engine is correct and cleanly built (a one-line format fix passes all 14 hidden tests), but it right-aligns the status column against an explicit example, which breaks the 'status lines are not indented' contract, and it adds no CLI tests.

Full unblinded review: reviews/c10-status-diff.md (candidate letters: A = Claude Haiku 5.5, B = Mercury 3, C = GPT-6 Luna).

Exact task text (appended to the shared preamble in the Method section):

Please add a `--diff` flag to `treeward status`. Today `status` tells me *that* something changed (`M`, `R`, ...) but
not *what* changed, so before accepting a changeset I end up digging through `.treeward` files by hand. With `--diff`,
each changed entry should be followed by indented detail lines saying what changed.

What I want to see:

- Modified file: one line per field that actually changed, out of size, mtime, and sha256, showing old and new values.
  Fields that did not change are not shown.
- Symlink whose target changed: a `target: OLD -> NEW` line, e.g. `target: target1.txt -> target2.txt`.
- Type change (say a file replaced by a symlink): what it was and what it is now, as a `was: ...` line and a `now: ...`
  line, e.g. `was: file (12 bytes, sha256: ...)` and `now: symlink -> somewhere`.
- Removed entry: a `was: ...` line describing what the ward had recorded: `was: file (<size>, sha256: <checksum>)`,
  `was: directory`, or `was: symlink -> <target>`.
- Added entries and (with `--all`) unchanged entries get no detail lines.

Roughly like this:

```text
M  data.json
   size: 1.2 KB -> 1.5 KB
   mtime: 2024-01-15 10:30:45.123 -> 2024-01-16 14:22:10.456
   sha256: abc123def456... -> 789xyz012345...
R  oldfile.txt
   was: file (256 bytes, sha256: abc123def456...)
```

The labels (`size:`, `mtime:`, `sha256:`, `target:`, `was:`, `now:`) and the `->` arrows are what scripts and my eyes
will key on, so keep them as shown. Detail lines are indented; status lines are not. How you format sizes,
timestamps, and checksums is up to you, as long as it is readable (human-readable sizes, a readable local timestamp,
and a shortened checksum are what I have in mind). Pulling in a crate for time formatting is fine.

Behavior:

- `--diff` implies `--verify`: files with changed metadata get checksummed, so a real content change shows as `M` (not
  `M?`) and the sha256 line can be shown. The fingerprint it prints must be the same one `status --verify` would print,
  so it works with `update --verify --fingerprint ...`.
- `--diff --always-verify` checksums everything, and the diff includes any checksum changes found that way.
- `--diff --all` lists unchanged entries too, without details. With no changes, `status --diff` (with or without
  `--all`) behaves like plain `status`: exit 0 and no fingerprint.
- Exit codes and stderr behavior stay as they are today (1 when there are changes, nothing on stderr for a normal
  unclean result).
- `verify` output does not change apart from the alignment below.

While you're in there: pad the status code to a two-character column in the listing (for both `status` and `verify`),
so paths line up whether the code is `M?` or `A`: `A  new.txt`, `M? data.json`, `.  file.txt`.

Method

Harness. All three models ran through omp (Oh My Pi) headless on OpenRouter (openrouter/openai/gpt-6-luna, openrouter/anthropic/claude-haiku-5.5, openrouter/inception/mercury-3) with identical flags. omp's task subagent tool was disabled, because its subagents can run on the configured default model rather than the session's, which would let one model's run be done by another. Web search stayed available for API docs. The same global agent instructions and skills were loaded for every run. Twelve runs went at a time on a shared 18-core machine that was also running unrelated jobs.

omp -p --approval-mode yolo --mode json --model <model> --thinking high --cwd <snapshot> --max-time 70m --no-title --tools read,bash,edit,eval,glob,grep,find,todo,write,wait,web_search -- "<prompt>" < /dev/null

Prompt. Each run received this preamble, with {{TREE}} replaced by its snapshot path, followed by the case's task text:

You are working on the Rust project in {{TREE}} (your working directory). Carry out the request at the end of this message completely, the way you would for a real pull request in this repository.

Ground rules for this task:

- Stay inside {{TREE}}. Do not read, list, or search anything outside it, with these exceptions: the Rust toolchain and dependency sources (`~/.rustup`, `~/.cargo`, including third-party crate sources under `~/.cargo/registry`), your own global instructions and skills under `~/.omp/agent`, and scratch files you create yourself under `/tmp` (make a fresh directory with `mktemp -d` for them; other jobs share this machine). Every other directory on this machine is off limits, even if it looks relevant.
- You may look up documentation for Rust, its standard library, and third-party crates, on the web or locally. Do not search for, fetch, or open anything about this project itself: not its GitHub repository, its issues or pull requests, its crates.io or docs.rs pages, or anything found by searching for its name or its author. The point of the exercise is that you work from this tree alone.
- There is no git history and no remote. Do not commit, branch, stash, or run any other git command that writes. Read-only commands such as `git status` and `git diff` are fine.
- Nobody will answer questions. Where the request leaves a decision open, make a reasonable choice and record it in the summary.
- When you are done, write `EVAL_SUMMARY.md` at the root of the tree: what you changed and why, the design decisions you made, anything left undone or uncertain, and which checks you ran with their results. Then end your turn.

The request:

Snapshots. git archive of the parent commit into a fresh one-commit repo at a random /tmp path, with lore/ removed. For c10, two doc-comment sentences that named the future design were also removed. Dependencies were prebuilt in each tree.

Mechanical checks. After each run: the run's own cargo test, cargo fmt --check, clippy -D warnings compared with the same check on the starting commit (two parent commits already carry 4 diagnostics on current clippy), dprint on repo files, and the acceptance tests.

Containment. Every tool call in all 30 transcripts was scanned for literal paths outside the run's own snapshot and its scratch directories, and the flagged calls were read. No access to another snapshot, the reference material, the full-history clone, GitHub, or anything online about the project was found. No run used web search, and every git command was read-only. Two runs did go beyond the prompt's limits: Mercury in c06 and Haiku in c10 listed the shared /tmp/tmp.* directories to pick the most recent scratch directory. Mercury then read files from the directory it picked, and they were its own test fixtures; Haiku did not use the result. The scan works on literal argument text, so it does not resolve shell variables or globs.

Caveats

Per-run data: data.json.