Reviewing Every Coding Agent Edit with a 3B Decision Model: Ryan Yannelli

I wanted to know if a small model could review every edit a coding agent makes, before the edit lands, without making the agent wait.

Claude Code, Codex, and Cursor can all run a hook before a file is written. The hook can let the edit through, add a note for the agent, or send the edit back with a reason. A hook runs on every edit, so the reviewer behind it has to be fast.

I built three things:

A dataset of 12,000 JavaScript and TypeScript snippets labeled for 10 code review flaws.
Four small models fine-tuned on that dataset.
Agent hooks that send each edit to the best of those models, on a hosted endpoint at code-review.ryanyannelli.com.

Then I gave three coding agents the same task with and without the hooks and compared the code they wrote.

Why a decision model?

A chat model reviews code by writing the review, one token at a time. A decision model reads the code once and scores each possible option directly. It returns probabilities and generates no text.

That lets one request ask several questions at once. The model I ended up using is a fine-tune of Liquid AI’s d1-3B. It reads the snippet once and answers 13 questions as branches of that one forward pass. “Does this code have dead code?” comes back as P(yes) 0.03. “How mergeable is it?” comes back as a distribution over five levels.

Probabilities also let you set a cutoff per rule. A rule you trust can block at 0.5, and a rule that misfires on your codebase can flag at 0.9 instead.

The dataset

JS/TS Code Review Decisions is on Hugging Face under CC BY 4.0. It has 3,000 tasks, and each task has four variants: one clean and three with flaws. That’s 12,000 snippets and 27,450 line-level findings.

Each snippet is labeled for 10 flaws:

Flaw	What counts
Unclear naming	Single letters outside short callbacks, names like data/temp/obj, abbreviations a new teammate couldn’t expand
Noise comments	Any comment other than a tool directive like `// @ts-check`
Dead code	Commented-out code, unused variables or imports, unreachable branches, leftover `console.log`
Inconsistent style	camelCase next to snake_case, async/await mixed with `.then`, `var` next to `const`
Over-abstraction	An interface with one implementation, a factory for one case, wrappers that only forward
Overly clever code	Nested ternaries, one reduce doing three jobs, bitwise tricks
Duplicated logic	The same logic two or more times with small variations
Error handling problem	Empty catch, a catch that logs and carries on, errors turned into `null` with the cause lost
Type safety hole	`any`, non-null `!`, `as` casts on unchecked values, `@ts-ignore`
Unsafe input	Request bodies, query params, env vars, or API responses used unchecked; string-built SQL, `innerHTML`, `eval`

Language models wrote the snippets in three rounds of 1,000 tasks. A reviewer model read each snippet without seeing the planned labels and recorded what it found. Snippets that failed went back for repair and another review. A separate model then audited all 27,450 findings and accepted 98.3% of them.

In the second and third rounds, each flawed variant is a small edit of its task’s clean variant. The planted flaws are the main difference between the four versions of a task.

The noise comments rule changed after the first round. Comments that explained why used to be exempt; now any comment counts. The first-round snippets had to be brought under the new rule: a model added 349 comment findings to 220 flawed snippets, and comments were removed from 84 clean snippets so they stayed clean.

The dataset also has a decisions config. It poses each snippet as the same 13 typed questions: the 10 flaws, whether the code has any flaw, which flaw matters most, and how mergeable it is (five levels, from “Should not merge.” to “Exemplary. Merge as is.”). Those rows replay against llama.cpp’s /v1/systemone endpoint or any API with the same shape, and score_decisions.py scores the results.

Four models

I fine-tuned four models on the same 9,600 training snippets and scored them on the same 1,200 test snippets (15,600 decisions). The dataset keeps all four variants of a task in one split, so no test task was seen in training.

Model	Params	Macro F1	Accuracy	Clean judged clean
d1-3B	3.1B	0.917	0.940	95.0%
d1-omni-600M	587M	0.771	0.882	82.7%
GLiNER2.5-Decide	486M	0.687	0.863	68.3%
NanoJev	596M	0.452	0.831	0.0%
d1-3B base model, zero-shot	3.1B	0.220	0.726	95.3%
TF-IDF + logistic regression	-	0.502	0.830	30.7%

Macro F1 is the F1 score for each of the 10 flaws at a 0.5 cutoff, averaged. Clean judged clean is the share of the 300 clean test snippets that the model says have no flaw.

For a hook, the last column matters as much as F1. NanoJev scored 0.831 accuracy and judged 0 of the 300 clean snippets clean, so it would flag every edit. Accuracy hides that because most of the 15,600 decisions are “no”: a baseline that never reads the code scores 0.766.

The base d1-3B judged 95.3% of clean snippets clean but found few of the flaws (macro F1 0.220). Fine-tuning took the same 3.1B model from 0.220 to 0.917. That run took 36 minutes on one H100: LoRA at rank 32, one epoch, 1,200 steps of 8 snippets.

d1-3B is weakest on unsafe input (F1 0.845), duplicated logic (0.867), and type safety holes (0.879). Quality is the hardest question at 68.4% accuracy, and 80% of its misses are one level off.

Speed

Median time for one snippet with all 13 questions, one snippet at a time:

Hardware	Runtime	p50
One H100 SXM	PyTorch, bf16	30.5 ms
One L40S	llama.cpp, Q8_0 GGUF	221 ms
16 Neoverse V3 CPU cores	llama.cpp, Q8_0 GGUF	6.2 s
16 Neoverse V3 CPU cores	PyTorch, fp32	8.9 s

6 to 9 seconds is too long to sit in front of every edit, so the hooks call a GPU.

The Q8_0 GGUF scores the same as the PyTorch model on the full test split. Q4_K_M loses 0.004 macro F1 and changes 0.8% of top answers. The GGUF builds run in llama.cpp and Ollama if you’d rather run the model yourself.

How the hooks work

The hook runs locally next to the agent on Node.js. The model runs on a hosted GPU.

When you submit a prompt, the hook keeps its first 300 characters as the task for the review and sends a warm-up request to the GPU.
Before the agent writes a JavaScript or TypeScript file (JSX and TSX included), the hook applies the edit in memory and takes the changed lines plus 15 lines on each side, in up to three 80-line windows.
The model reviews those windows. The hook also reviews the same range as it was before the edit and reports a rule only when it was under the threshold before and over it after, so an edit isn’t blamed for code that was already there.
Your rules decide what happens. Each of the 10 rules is set to off, flag, or block, with a default threshold of 0.5.

Flag adds a note for the agent and the edit goes through. Block rejects the edit and sends the reason back to the agent. By default unsafe input blocks, noise comments is off, and the other eight rules flag.

The block message tells the agent that the reviewer is a small model and can be wrong. If a finding doesn’t apply, the agent is told not to rewrite the code to get past it and to tell you which rule fired. That keeps an agent from rewriting correct code to satisfy a false positive.

Failures never block. If the endpoint is down or a review runs past its 45-second limit, the edit goes through and the error is written to ~/.code-review-hooks/hook.log.

The hosted endpoint

With the GPU on, a review takes about 0.21 seconds per edit. The GPU shuts off after 60 idle seconds, so it costs $0 while nobody is using it. The warm-up request on each prompt wakes it before the agent’s first edit.

To install for Claude Code:

curl -fsSL 'https://code-review.ryanyannelli.com/install.sh?agent=claude' | sh

Replace claude with codex or cursor for the other agents. The installer backs up any config it touches, and the sandbox builds the same command with your own rules. crh changes settings later:

crh status                  current rules
crh set unsafe_input block  one rule
crh threshold 0.6           all rules
crh off | on                pause all checks
crh uninstall all           every agent, every file

The endpoint is free for a limited time, US only, with no uptime guarantee. If it goes away, your edits go through unreviewed.

Testing it on three agents

I gave three coding agents the same task twice, once with the hooks and once without: Claude Sonnet 5.5, Claude Opus 5.5, and GPT-6.1-Sol in Codex. Each run started from its own copy of a small TypeScript service, with effort set to low.

The task was a customer CSV import split across 7 modules. I wrote the spec so each of the 10 rules has a place to fire:

Unsafe input: environment variables, a file name from the request, SQL inserts, an HTML email, and a company lookup API
Error handling: retries and wrapped errors
Type safety: parsed JSON and API responses
Overly clever code: a CSV parser with quoted fields and line breaks
Duplicated logic: email and hostname checks needed in three modules
Naming, comments, dead code, style, and over-abstraction: every file

The hooks used a threshold of 0.9 with all 10 rules on, including noise comments. Unsafe input blocked, and the other 9 rules flagged. Both arms got the same prompt, which told the agent to write files with its file editing tools so every edit went through the hook.

I scored each run with 68 hidden tests the agents never saw. They cover every status code, path rule, retry, and escaping case in the spec. To check the tests themselves, I broke a working reference build 6 ways:

A prefix check that lets ../imports-other/ through
SQL built from strings
HTML without escaping
No retry on a 5xx
An error rethrown without its cause
An error message returned in a 500

Each broken build failed at least one test.

What happened

Run	Hidden tests	Flags	Casts left	Time	Cost
Sonnet 5.5, hooks off	68/68	-	6	102 s	$0.32
Sonnet 5.5, hooks on	68/68	3	6	85 s	$0.28
Opus 5.5, hooks off	68/68	-	8	123 s	$0.58
Opus 5.5, hooks on	68/68	5	0	149 s	$0.73
GPT-6.1-Sol, hooks off	68/68	-	2	181 s	9k tokens
GPT-6.1-Sol, hooks on	68/68	1	0	161 s	8k tokens

Casts left counts any, as casts, non-null !, and @ts-ignore in each run’s source files. Codex reports output tokens instead of a dollar cost.

Every run passed all 68 hidden tests and the type check, with or without the hooks.

The hooks made the difference in type safety. With the hooks on, Opus went from 8 casts to 0 and GPT-6.1-Sol went from 2 to 0.

Opus fixed all 5 of its flags:

config.ts: replaced the as casts on feature flags with a type guard
fetchCompanyProfile.ts: replaced a cast on the API response with in checks
saveCustomers.ts: renamed n to last
validation.ts: removed two JSDoc comments
handleImport.test.ts: rewrote a test the reviewer marked as overly clever

GPT-6.1-Sol wrote the feature in two patches. The hook flagged a type safety hole in fetchCompanyProfile.ts on the first one, and the second patch rewrote that file without the cast.

Sonnet got 3 flags and kept its code as written. One of them was dead code at 0.99 in a CSV parser with nothing unused, and the flag message tells the agent to skip findings that don’t apply.

No edit was blocked. Unsafe input never reached 0.9.

The median review took under 0.9 seconds per edit for the Claude agents and about 2 seconds per Codex patch. Opus spent 26 seconds and $0.15 more on its fixes. Sonnet and GPT-6.1-Sol finished faster with the hooks on; with one run per agent, I can’t separate that from normal variation.