Opus 5 / 4.7 / 4.6 — Tooling Code Review Comparison Date : 2026-08-27 Prompt : “there have been a ton of commits to the tooling over the last 2 days. i think it’s all suspect so give it a highly skeptical review. also look for any places that may be causing long pauses with no apparent activity while running the app live” Reviewer : Claude Opus 4.6 (session jve-spec-kit-claude-12), comparing transcripts Coverage Quality of Findings Opus 4.6 — broadest surface, shallowest depth. 15 findings but most are observations about code structure (“8+ process spawns per Bash call”, “13 reading the same transcript”). The biggest claim — broken_model_gate.py spawning a 60s LLM subprocess on 20% of edits — was the headline, but there was no verification it actually blocks (it doesn’t — the gate is advisory-only by design). Editor findings are read-the-code extrapolations (busy_timeout, project_browser refresh, codec_probe dead code), not traced from live behavior. Opus 4.7 — fewest findings, tightest signal. Every app pause cites a specific commit or log line (fd81b5adf for the 832ms audio stall, b2fee2136 for the pool mismatch). Recommendations are ranked and actionable. But the tooling review is thin — “surprisingly solid overall” from the fork, four items, no deep investigation of the hook chain’s aggregate cost. Opus 5 — the only one that found a real bug (#1: run_gate.sh:127 empty-array crash under bash 3.2), verified it live, and traced its consequence (gate exits non-zero after already certifying the tree). Also the only one that read the TSO and found timestamped silences in the actual app session (5s and 7.5s import gaps). Then, when asked to opine, produced the structural diagnosis — 145 tooling vs 48 app commits, 33/54 hooks policing other hooks, the error-rate-rising-alongside-gate-count measurement — and mined 1,811 transcripts to build an empirical gate census with positive controls. The “proposed set” (57 → 14 hooks) is grounded in per-gate firing and retry rates, not opinion. Methodology 4.6 : fork-and-merge, three parallel forks for hooks/scripts/app. Efficient but each fork works in isolation — no cross-cutting insight. 4.7 : single fork for everything, then relayed findings. Fastest wall-clock. Trusted the fork’s “surprisingly solid” without challenging it. 5 : no forks, all inline. Spent the most bash calls but each one built on the previous — timing probe → TSO gap analysis → transcript census → positive-control verification. Caught its own broken sweeps twice and reported them as a finding about itself. Accuracy — Verified Against Code Each model’s factual claims were checked against the actual codebase by forked verification agents. Opus 4.6 (10 claims checked) 4.6 score: 4 confirmed, 2 inflated, 4 wrong. The headline finding — that the biggest pause source is broken_model_gate blocking for 60s — is factually wrong. The gate is advisory-only by its own design doc. Opus 4.7 (8 claims checked) 4.7 score: 6 confirmed, 1 inflated, 1 wrong. Opus 5 (8 claims checked) Opus 5 score: 3 confirmed, 1 partial, 2 inflated, 2 wrong. Accuracy Summary Overall Assessment 4.7 is the most accurate per-claim. 4.6 made the most claims and the most errors — including getting its headline finding wrong (advisory gate ≠ blocking gate). 5 found the highest-value bugs (the bash crash is real and consequential) but was sloppy on line numbers and counts. The pattern: 5 did the best thinking, 4.7 did the most careful citing, 4.6 did the most volume with the least verification. The gap is largest on the “highly skeptical” part of the prompt — 4.6 and 4.7 both accepted the tooling as mostly sound; 5 measured the ratio and said the tooling itself is the problem. Only 5 went beyond the review into structural diagnosis: the gate-count-rising-alongside-error-rate measurement, the transcript census, and the concrete proposal for what to keep and what to delete. submitted by /u/joefilmmaker
Originally posted by u/joefilmmaker on r/ClaudeCode
You must log in or # to comment.