TraceHound / Benchmark
Tied with four of five commercial reviewers on the blind test, ahead of one. Every loss is on this page.
Aggregate results for TraceHound v2 (pool-verify-v1 on Nemotron 3.5 Super), copied from the committed benchmark report. Medium+ findings; low-severity findings are ignored, and duplicates count as neither true nor false positives.
Source: docs/bench/results.md · decisions 067–071 · b0f1bc4 · the five recorded reviewers are shown as Reviewer A–E here; their names are in results.md
Blind test
F by reviewer
- TraceHound v2
- TraceHound v1
- Other reviewers
- Tied with v2
F on the blind test under bench-judge-v1, highest first: Reviewer A 73.2%, Reviewer B 66.7%, TraceHound v1 58.3%, TraceHound v2 57.1%, Reviewer C 50.0%, Reviewer D 46.2%, Reviewer E 36.7%.
- Reviewer A
- Reviewer B
- TraceHound v1v1
- TraceHound v2v2
- Reviewer C
- Reviewer D
- Reviewer E
Blind test: Cal.com, v2
| Reviewer A | Reviewer B | TraceHound v1pool-verify-v1, 3 Super, run 1 | TraceHound v2pool-verify-v1, 3.5, run 4 | Reviewer C | Reviewer D | Reviewer E | review-diff-v13.5, run 4 · secondary, not ranked | pool-raw3.5, run 4 · secondary, not ranked | |
|---|---|---|---|---|---|---|---|---|---|
| Commentsposted, Medium+ | 23 | 21 | 30 | 30 | 24 | 49 | 31 | 30 | 34 |
| True positivesgolden findings matched | 15 | 13 | 14 | 12 | 10 | 15 | 9 | 12 | 13 |
| False positivescomments matching none | 3 | 3 | 11 | 7 | 7 | 27 | 17 | 6 | 9 |
| Missedgolden findings not found | 8 | 10 | 9 | 11 | 13 | 8 | 14 | 11 | 10 |
| PrecisionTP / judged comments | 83.3%15/18 | 81.3%13/16 | 56.0%14/25 | 63.2%12/19 | 58.8%10/17 | 35.7%15/42 | 34.6%9/26 | 66.7%12/18 | 59.1%13/22 |
| RecallTP / golden findings | 65.2%15/23 | 56.5%13/23 | 60.9%14/23 | 52.2%12/23 | 43.5%10/23 | 65.2%15/23 | 39.1%9/23 | 52.2%12/23 | 56.5%13/23 |
| Fbench-judge-v1, scorer of record | 73.2% | 66.7% | 58.3% | 57.1% | 50.0% | 46.2% | 36.7% | 58.5% | 57.8% |
| Fbench-judge-x1 | 73.2% | 66.7% | 58.3% | 58.5% | 55.0% | 50.8% | 37.5% | 60.0% | 60.5% |
10 blind pull requests, n = 23: the second blind look at Cal.com, pre-registered (069). P and R are bench-judge-v1 counts. 11 of v2's 30 comments were judged duplicates. On this run the verifier did not raise F over review-diff-v1 alone (57.1% vs 58.5%). Secondary rows are stages of the same pipeline and are not ranked.
Cal.com: v2 minus each reviewer, 95% intervals
| Reviewer A | Reviewer B | TraceHound v13 Super pool-verify-v1 (run 1) | Reviewer C | Reviewer D | Reviewer E | |
|---|---|---|---|---|---|---|
| v2 minus reviewerbench-judge-v1, F points [95%] | −16.0[−30.9, +3.5] | −9.5[−30.2, +16.7] | −1.2[−16.0, +12.2] | +7.1[−15.8, +36.8] | +11.0[−6.2, +27.7] | +20.4[+11.8, +34.3] |
| v2 minus reviewerbench-judge-x1, F points [95%] | −14.6[−28.5, +4.4] | −8.1[−27.9, +17.5] | +0.2[−13.4, +13.1] | +3.5[−17.1, +31.2] | +7.7[−8.3, +20.5] | +21.0[+1.4, +37.0] |
| Verdictthe rule's sentence | statistically tied with Reviewer A | statistically tied with Reviewer B | statistically tied with 3 Super | statistically tied with Reviewer C | statistically tied with Reviewer D | ahead of Reviewer E |
F(v2) − F(row) in points, PR-level paired bootstrap, 10,000 resamples. Under both judges v2 is statistically tied with Reviewer A, Reviewer B, Reviewer C, Reviewer D and 3 Super, and ahead of Reviewer E. Only 10 pull requests, so these intervals are probably too narrow.
Dev and blind sets together
F by reviewer
- bench-judge-v1 (scorer of record)
- bench-judge-x1
F with dev and blind sets together, in rank order: Reviewer A 66.3% (v1), 65.9% (x1); Reviewer B 53.8% (v1), 58.8% (x1); TraceHound v2 52.2% (v1), 55.4% (x1); Reviewer C 49.5% (v1), 55.1% (x1); Reviewer E 40.0% (v1), 45.0% (x1); Reviewer D 35.2% (v1), 42.0% (x1).
- Reviewer A
- Reviewer B
- TraceHound v2v2
- Reviewer C
- Reviewer E
- Reviewer D
All 50 PRs, bench-judge-v1 (scorer of record)
| Reviewer A | Reviewer B | TraceHound v2dev run 7r + Cal.com run 4 (pre-registered) | TraceHound v2dev run 8 + Cal.com run 4 (second dev draw) | Reviewer C | Reviewer E | Reviewer D | TraceHound v13 Super pool-verify-v1, row (i) (not ranked) | |
|---|---|---|---|---|---|---|---|---|
| Rankpoint estimate | 1 | 2 | 3 | 3 | 4 | 5 | 6 | – |
| True positivesgolden findings matched | 56 | 42 | 47 | 46 | 45 | 41 | 43 | 40 |
| False positivescomments matching none | 16 | 17 | 36 | 35 | 40 | 67 | 104 | 57 |
| Missedgolden findings not found | 41 | 55 | 50 | 51 | 52 | 56 | 54 | 57 |
| PrecisionTP / judged comments | 77.8%56/72 | 71.2%42/59 | 56.6%47/83 | 56.8%46/81 | 52.9%45/85 | 38.0%41/108 | 29.3%43/147 | 41.2%40/97 |
| RecallTP / golden findings | 57.7%56/97 | 43.3%42/97 | 48.5%47/97 | 47.4%46/97 | 46.4%45/97 | 42.3%41/97 | 44.3%43/97 | 41.2%40/97 |
| Fharmonic mean of P and R | 66.3% | 53.8% | 52.2% | 51.7% | 49.5% | 40.0% | 35.2% | 41.2% |
| v2 minus reviewerF points [95% interval] | −14.0[−22.6, −4.7] | −1.6[−12.6, +10.7] | – | – | +2.8[−7.1, +13.2] | +12.2[+3.8, +20.6] | +17.0[+6.0, +28.0] | +11.0[+1.8, +20.1] |
Dev + Cal.com, n = 97; 40 of the 50 pull requests are the dev set v2 was tuned on. Ranks are point estimates. The last column is v2 (dev run 7r + Cal.com run 4) minus that row, with its 95% interval: behind Reviewer A, statistically tied with Reviewer B and Reviewer C, ahead of Reviewer D and Reviewer E, and ahead of v1 (3 Super) under this judge.
All 50 PRs, bench-judge-x1 (report only)
| Reviewer A | Reviewer B | TraceHound v2dev run 7r + Cal.com run 4 (pre-registered) | TraceHound v2dev run 8 + Cal.com run 4 (second dev draw) | Reviewer C | Reviewer E | Reviewer D | TraceHound v13 Super pool-verify-v1, row (i) (not ranked) | |
|---|---|---|---|---|---|---|---|---|
| Rankpoint estimate | 1 | 2 | 3 | 3 | 4 | 5 | 6 | – |
| True positivesgolden findings matched | 56 | 45 | 49 | 50 | 49 | 45 | 50 | 45 |
| False positivescomments matching none | 17 | 11 | 31 | 30 | 32 | 58 | 91 | 45 |
| Missedgolden findings not found | 41 | 52 | 48 | 47 | 48 | 52 | 47 | 52 |
| PrecisionTP / judged comments | 76.7%56/73 | 80.4%45/56 | 61.3%49/80 | 62.5%50/80 | 60.5%49/81 | 43.7%45/103 | 35.5%50/141 | 50.0%45/90 |
| RecallTP / golden findings | 57.7%56/97 | 46.4%45/97 | 50.5%49/97 | 51.5%50/97 | 50.5%49/97 | 46.4%45/97 | 51.5%50/97 | 46.4%45/97 |
| Fharmonic mean of P and R | 65.9% | 58.8% | 55.4% | 56.5% | 55.1% | 45.0% | 42.0% | 48.1% |
| v2 minus reviewerF points [95% interval] | −10.5[−19.8, −0.3] | −3.5[−13.7, +8.3] | – | – | +0.3[−9.5, +10.7] | +10.4[+1.1, +19.2] | +13.4[+1.2, +25.6] | +7.2[−3.0, +17.4] |
The same rows under the second judge. The same sentences hold, except that against v1 (3 Super) v2 is statistically tied under this judge. With dev run 8 in place of 7r, the Reviewer A interval includes 0 under this judge.
Dev set
Dev set (40 PRs, n = 74), mean per model
| row | judge | 3 Super, mean of 5: P / R / F | 3.5 Super, mean of 2: P / R / F | Ultra, mean of 2 (report only): P / R / F |
|---|---|---|---|---|
| pool-verify-v1 | v1 | 40.5 / 44.3 / 42.3 | 54.8 / 46.6 / 50.4 | 39.0 / 59.5 / 47.1 |
| pool-verify-v1 | x1 | 50.9 / 51.9 / 51.3 | 60.5 / 50.7 / 55.1 | 46.5 / 66.9 / 54.9 |
| pool-raw | v1 | 30.0 / 52.7 / 38.2 | 48.0 / 61.5 / 53.9 | 32.7 / 65.5 / 43.6 |
| pool-raw | x1 | 37.3 / 63.8 / 47.0 | 49.2 / 62.2 / 54.9 | 39.5 / 73.6 / 51.4 |
The set every variant was tuned on. 3.5 − 3 Super on pool-verify-v1: v1 +8.1 [+1.4, +14.4], x1 +3.9 [−4.2, +11.1]. Ultra is report only (free NVIDIA API, not Token Factory, not the product); its only interval excluding 0 is pool-raw under v1, Ultra − 3.5 = −10.2 [−17.7, −3.2]. Five 3 Super runs spread 35.6–48.0 F. 3.5 fails to return valid JSON on about 7 of 40 pull requests per run.
Precision levers tried on dev
| decision | idea | bar | result (F / P on dev) | verdict |
|---|---|---|---|---|
| 056 verify-v1 | A second Super pass gives each review-diff-v1 finding a confidence of 1–5; findings below a threshold are dropped. | The threshold with the highest dev F, ties to the higher; no bar against the baseline. | Threshold 1, which keeps every finding: 32.2% / 25.6%. Thresholds 2–5: F 32.1, 31.2, 29.2, 29.1%; P 29.5, 30.0, 31.7, 44.4%. | No gain: the rule chose the threshold that removes nothing. |
| 057 review-ctx-v1 | The reviewer also sees repository context: the changed files and git grep references. | None pre-registered; compared with review-diff-v1 (32.2% / 25.6%). | 31.8% / 36.2% | F did not move. Used later only as a second source for 059's pool. |
| 058 verify-ctx-v1 | A verifier that sees repository context answers keep or drop for each review-diff-v1 finding. | F ≥ 35.0% | 32.8% / 38.9% | Loss; not used on its own. |
| 059 pool-verify-v1 | Pool review-diff-v1's findings with review-ctx-v1's ctx-only findings, then filter with verify-ctx-v1. | F ≥ 35.0% | Run 1: 35.6% / 36.1%. 5-run mean (064): 42.3% / 40.5%. | Adopted, at a 0.6-point margin on run 1. |
| 064 vote | Keep only findings that recur across five independent runs (best variant: vote-k3of5). | v1 F ≥ the baseline's 5-run mean + 5.0 points; v1 P above the baseline's mean; x1 F ≥ the baseline's x1 mean. | vote-k3of5: 47.8% / 53.3%, against the baseline's 42.3% / 40.5%; x1 F 50.4%, against 51.3%. | Loss: fails the x1 condition. |
| 065 evidence verifier + gate | B: a verifier that must quote the failing code, then a deterministic gate that checks the quote. A: the gate alone, on pool-verify-v1. | A: mean P above the baseline's, and mean F at least the baseline's under v1 and x1. B: mean v1 F +3.0 points, mean v1 P +8.0 points, mean x1 F at least the baseline's. | 5-run means: A 41.6% / 40.3%; B 38.3% / 37.2%; baseline 42.3% / 40.5%. | Loss: A and B both fail every condition. |
| 067 3.5 Super | Run every stage of pool-verify-v1 on Nemotron 3.5 Super instead of 3 Super. | Mean over the completed runs: v1 F ≥ 45.28%, v1 P ≥ 40.52%, x1 F ≥ 51.29%. | 2-run mean (7r, 8): 50.4% / 54.8%; x1 F 55.1%. | Adopted (068), on 2 runs (the plan's fallback: the cap stopped run 9). |
| 070 recall-v3 | Two lenses per chunk (A: correctness, reliability, security, performance, API; B: tests, docs, typos, maintainability, completeness) plus a missed-issue pass; arm S on 3 Super, arm V on 3.5. | P009: v1 precision ≥ the model's pool-raw dev mean − 10.0 points (S 20.0, V 38.0); pick the qualifying arm with the higher recall. | S 21.8% / 14.0%; V 31.4% / 20.9%. | Loss: both arms below P009's precision floor. Measured partly hampered by truncated replies and the $0.12 per-PR cap. |
| 071 Nemotron 3 Ultra | Run every stage of pool-verify-v1 on Nemotron 3 Ultra via the free NVIDIA API. | None (report only); compared with the 3 Super and 3.5 Super means. | 2-run mean (10, 11): 47.1% / 39.0%; pool-raw 43.6% / 32.7%, against 3.5's 53.9% (−10.2 [−17.7, −3.2]). | Report only, not adopted: no interval-backed gain; worse than 3.5 on pool-raw. |
| levers v2: per-PR cap | Keep at most k findings per pull request (k = 1–5), ranked by agreement or by order, replayed on saved dev runs of 3 Super, 3.5 and Ultra. | Report only; a cell counts only if its 95% interval excludes 0. | 3.5 verified, uncapped 50.4%: k = 1–5 F 39.8 to 50.9% (agreement), 33.7 to 51.5% (order). Pool-raw: 4 cells above 0 (3 Super k = 4–5, 3.5 k = 5). | No cap on the verified pipeline improves F with an interval excluding 0; small gains only on the unverified pool at k = 4–5. Hypothesis, not adopted. |
| levers v2: verifier on/off | Verified (pool-verify-v1) minus raw (pool-raw) F on the same 9 saved dev runs. | Report only; interval rule as above. | Interval above 0 in 2 of 9 runs (3 Super 2r +10.9, Ultra 11 +6.7). 3.5: 7r −1.6 [−10.0, +5.5], 8 −5.4 [−14.7, +3.3]. | Inconclusive on 3.5: an open question, not a finding. On ReviewBench raw scores higher in both models. |
| levers v2: 2-of-3 vote | Keep 3 Super verified findings that recur in 2 of 3 runs. | Report only; vote F against the mean of its three runs. | Runs 4, 5, 6: 38.4% / 37.7% (mean 42.6; −4.2 [−10.5, +1.9]). Runs 1, 2r, 4: 48.9% / 56.1% (mean 42.5; +6.3 [−0.3, +12.6]). | No gain established. |
Every idea tried after the diff-only baseline. The levers v2 rows replay saved dev records at $0 and are report only. Run-to-run spread (35.6–48.0 F over five 3 Super runs at temperature 0) is larger than most effects measured.
Other runs
ReviewBench test runs (25 PRs, one round, ReviewBench's own judge)
| # | config | grounded F1 | G P | G R | augmented F1 | A P | findings |
|---|---|---|---|---|---|---|---|
| 331 | pool-raw-x2 · 3 Superpre-registered winner | 21.3 | 76.3 | 12.4 | 30.6 | 50.6 | 222 |
| 333 | pool-raw-missed · 3 Super | 20.4 | 64.5 | 12.1 | 28.9 | 43.9 | 194 |
| 328 | pool-raw · 3 Super | 16.0 | 70.6 | 9.0 | 23.0 | 52.5 | 124 |
| 330 | pool-verify-v1 · 3 Super | 13.9 | 79.5 | 7.6 | 22.3 | 67.2 | 99 |
| 329 | pool-raw · 3.5 | 13.5 | 56.3 | 7.7 | 19.0 | 44.0 | 86 |
| 332 | pool-verify-v1 · 3.5= the product | 12.2 | 69.2 | 6.7 | 17.6 | 57.2 | 60 |
Test runs, not a leaderboard submission, and no leaderboard rank. There is no final run: it would cost about $107 with VAT against a $43.52 balance. Six configurations on the same 25 pull requests carry selection bias. 3.5 lost to 3 Super here in both pipelines; the grounded metric rewards volume, which the verifier cuts.
v1 history: every Cal.com run (3 Super)
| Cal.com run | bench-judge-v1: F, point-estimate rank | bench-judge-x1: F, point-estimate rank |
|---|---|---|
| 1headline | 58.3%, 3rd | 58.3%, 3rd |
| 2 | 51.3%, 3rdn = 21, one JUDGE_FAILED | 57.1%, 3rd |
| 3 | 52.8%, 3rd | 47.3%, 5thbelow Reviewer C 55.0 and Reviewer D 50.8 |
v1 (pool-verify-v1 on 3 Super, 062) is superseded by v2. Point-estimate ranks among six reviewers; for run 1 no gap to another tool was outside its 95% interval. The full v1 page is results.md at e91a268.
Cost
Dev cost per pull request, pipeline only
| pipeline per pull request | n | input tokens median | output tokens median | output tokens mean | cost median | cost mean | cost max |
|---|---|---|---|---|---|---|---|
| 3 Superruns 1, 2r, 4, 5, 6 | 200 | 41,393 | 19,688 | 27,546 | $0.0307 | $0.0430 | $0.2724 |
| 3.5 Superruns 7r, 8 | 80 | 49,357 | 50,064 | 62,471 | $0.0626 | $0.0743 | $0.3031 |
| 3.5 Superruns 7r, 8 and the 23 PRs of run 9 | 103 | 50,151 | 52,510 | 63,497 | $0.0642 | $0.0756 | $0.3397 |
Judges excluded, every attempt counted, pre-VAT, third-party prices. The product costs about $0.06–0.07 per pull request on 3.5, about 2 minutes wall time median.
