Putting Small AI to Work Safely: Re:catch's Review-Routing Experiment

Code behavior described here reflects the September 23, 2026 checkout. Later shadow experiments are identified separately.

Our first review-routing rule treated an executable instruction document as ordinary prose and skipped specialist review. The fix was not a bigger model. The file path already told us it was a shared contract; we had put that fact in the wrong place.

The router runs before a pull request (PR) is published for review. Jev does not run our test-driven development cycle or decide whether the work is complete. The failures below show why that boundary matters.¹

The question and structure were inspired by Naia’s article about its Jev development-process experiment, published under CC BY-NC-SA. The wording, diagrams, and measurements here were independently developed from Re:catch’s code and experiment records.

What are recatch-tdd and Jev?

recatch-tdd is a team-specific development checklist and toolkit for taking Re:catch frontend and backend work through the same sequence: define policy and requirements, plan the work, write a failing test, make it pass, then finish the documentation, checks, and reviews. Its purpose is to keep the sequence and quality bar consistent even when the person or AI coding tool changes.²

Jev is TypeSafe’s System One judgment model. It understands natural-language text, but instead of generating free-form prose it returns structured judgments and likelihoods that code can consume. In this experiment we asked for yes/no likelihoods, an impact score, and a preferred review lane. “Small AI” describes the narrow scope of the task we gave it, not a claim about the model’s parameter count.³

The problem: reviewing everything is expensive, but skipping reviews by intuition is risky

Giving a security-configuration change and a one-line copy edit the same combination of specialist reviews is simple to operate. It also makes expensive reviewers reread changes that do not need their expertise. If people instead remove reviews by gut feel, they may miss an important risk. We wanted a consistent way to recommend the appropriate review path, not an AI system that decides whether a change is good enough to pass.

The first experiment used 20 real merged PRs: 12 backend and 8 frontend. This small convenience sample was weighted toward recent work and does not represent all development.⁴ Running all three specialist reviews on every one of the 12 backend PRs would have meant an upper bound of 36 review executions. The router selected 12. That is 67% fewer than the hypothetical always-run-every-lane upper bound, not a measured reduction from actual historical review activity.⁵

Cost alone was not the reason to adopt it. Real changes exposed a question that defined security risk too narrowly and a code rule that skipped an important kind of document. Those concrete, correctable failures mattered more than the headline percentage.

Where Jev enters the development process

The core recatch-tdd process stays intact: verify a problem with a failing test, make the test pass, improve the structure, and complete the required checks. Jev does not choose the next test or decide whether the tests are sufficient. It appears only after the code and tests are done, when specialist review paths are assigned.

In plain terms, the flow is:

• Finish the code, documentation, and tests.
• Gather the diff showing what changed.
• Let ordinary code check facts such as file locations.
• Ask Jev for small signals only where interpreting context matters.
• Let ordinary code apply the final review-routing rules.
• Run the assigned review path; when it selects specialists, they examine the change independently, and a person assesses their findings.
• If Jev or its provider fails, continue the existing checks and PR procedure.

Figure 1 — Code checks file paths and chooses the review path; Jev supplies contextual signals, not proof. skip-review and coderabbit-only add no local specialist reviewer. On a narrow screen, tap the figure to open a full-screen view and swipe to explore it; without JavaScript, swipe the inline figure sideways.

At the time described here, this flow was connected to the Re:catch backend and the Athena standalone backend, but not the Re:catch frontend. Frontend changes have a different distribution, so the proposed next step was a shadow evaluation: compare recommendations without triggering additional specialist reviews yet.¹

Here fail-open means that a broken advisory tool does not block the original procedure. If a helper going down prevented the entire PR from being prepared, the helper would effectively own the development process. The implementation records a short reason for failure, falls back to an assignment based on facts such as file paths, and continues. It does not waive the existing checks or independent review.⁶

Three answer formats; code makes the final decision

One Jev request carries the change title, paths, and diff as shared state. Each question sees that state, but not another question’s criteria or answer.

The answers use three forms:

• Noul returns the likelihood of “yes” for a question such as “Does this change an authorization boundary?”
• Score expresses a degree, such as whether an incident would stay within one file or reach another system.
• Choice selects one preferred first review lane from predefined options.

These answers are inputs to a recommendation, not the final policy. Ordinary code makes the final assignment. For example, a file path that creates a shared contract calls for a broader review regardless of Jev’s answer. Security and production-data signals take precedence over a signal that a change appears to be prose. This is code-owned policy: predictable code owns and executes the decision rules.⁷

We do not read confidence as “the probability of being right.” It is closer to a measure of how concentrated an answer is among the available choices. A confident answer can still be wrong when the question is badly framed or the input is incomplete. The code uses this value only in a limited rule for adding stricter reviews.⁸

Jev’s output therefore proves neither a defect nor completion. Evidence that work is actually done must come from independently checkable sources such as test results and separate reviews.⁶

Questions and rules we got wrong

The first failure was the auth_boundary question. Initially it asked only whether “who can do what” had changed. But changing the error code, denial reason, or log identifier visible to a rejected caller can also reveal information to an attacker. The model saw the whole change and selected security review, while the narrowly worded yes/no question returned a low signal. The question, not simply the model, failed to represent the policy. We expanded it to ask what an unauthorized user can observe.⁹

The second failure was the prose_only rule. A document can look like ordinary prose while actually specifying instructions an AI must follow or a contract shared across repositories. The original rule treated such files as copy edits and skipped specialist review. Asking AI to be smarter was not the fix: the file path already provided a reliable distinction. Ordinary code now recognizes those locations as important contracts.⁹

An earlier skill-catalog experiment, before the router was wired into recatch-tdd, exposed another mistake: we placed the duplication catalog inside one question’s criteria, assuming the other questions could see it. They could not. Moving the catalog into shared state changed the answer distribution. This was not a measurement of recatch-tdd’s runtime performance.¹⁰

We adopted review routing and rejected an AI blocking gate

We tested two ideas around the same time:

• A review-lane router that recommends which specialist should review a change first.
• A claim gate that would block publication if an AI claimed to have run checks it had not run.

We adopted the router as advisory, not as proof that every needed specialist will be called. Code applies the final routing rules, and a provider failure leaves the existing PR procedure intact. A mistaken assignment can still miss a specialist; we need to measure that risk.

We rejected the claim gate as a blocker. Among 889 sessions inspected, we found zero instances of the specific false-completion claim pattern we were looking for. Without knowing how often that target failure occurs, we could not calibrate a blocking threshold. At a threshold of 0.5, it wrongly blocked 4 of 12 normal messages. Five identical-input runs spanned 0.08, and rephrasing one question moved its score from 0.60 to 0.80. We could not use that as a publication blocker.¹¹

For a fact as concrete as whether a command ran, first check the tool record. A semantic model is unnecessary when there is a simpler, more reliable source of evidence.

Figure 2 — The initial PR sample exposed a narrow security question and a rule that skipped shared instructions. The router was adopted and the blocking claim gate rejected. A later 30-change frontend shadow sample is discussed below. On a narrow screen, tap the figure to open a full-screen view and swipe to explore it; without JavaScript, swipe the inline figure sideways.

Checking whether Korean inputs led to the same assignments

Jev’s documentation says accuracy is highest in English and advises testing other languages, including Korean, on actual content before relying on them. Many Re:catch PR titles and descriptions are Korean, so we needed a separate Korean-language evaluation.

We discarded the first English control. Replacing Korean fragments with English out of context had made the English sentences malformed. Comparing that broken translation with Korean could not isolate language effects from translation errors. For the corrected version, we translated entire titles and complete changed lines in context, rebuilding malformed lines as whole lines. Machine translation can still shift nuance, so we did not treat it as a certified translation.¹²

We also repeated one Korean input to see how much answers varied without changing languages. The largest signal difference was 0.05, and the final review lanes stayed the same. One extra run on one input is not a general stability estimate. In particular, we could not repeat the cases whose preferred first lane differed, so we could not attribute those differences to Korean.¹²

All 12 corrected Korean-English pairs ended with the same set of final review lanes after ordinary code applied its rules. The preferred first lane matched in 10 of 12 pairs. This is not Korean-language accuracy. It is an observation that this small sample produced the same final lane sets under the code’s policy.¹²

We also downgraded the status of the human labels. Labels assigned after seeing a model’s first output cannot serve as blind ground truth. We called them post-hoc policy adjudications and used them only to inspect where human expectations and system output diverged, not to calculate accuracy.

Cost estimates, not savings

The corrected Korean evaluation made 25 requests with 313,151 input tokens. At the provider’s then-published rate, that was an estimated $0.013, excluding the discarded first English control.¹³ Routing one PR used roughly 10k input tokens, or $0.0004 at that rate; diff size and pricing change the figure.¹⁴

The default model was jev-1.13.0, but an environment setting could select another model. It was not version-pinned.¹⁵ We have not measured reviewer time saved, unnecessary reviews triggered, or specialist reviews missed in production.

A follow-up: should incomplete input always trigger more review?

After the initial article was drafted, we ran separate shadow evaluations on 30 real backend changes and 30 frontend changes. The single-call router missed part of the diff in 8/30 backend and 11/30 frontend cases. The samples differ in change size and time period. Neither a pooled rate nor a workload-wide truncation rate follows from them.¹⁶

For the backend sample, we applied a proposed rule—“always add cross-provider review when the input is incomplete”—to the same 30 results. It repaired zero missed high-stakes cases and added stricter review to three ordinary changes. Under the preassigned labels, ordinary over-strict assignments rose from 2/20 to 5/20. That compares assignments, not reviewer hours or defects found. We rejected automatic escalation by this rule on this evidence.¹⁶

In the separate frontend sample, 6 of 10 high-stakes changes received the exact specialist lane expected, while 9 of 10 received at least one strict lane. Those are different criteria. With 11 incomplete inputs as well, the frontend stayed in shadow evaluation rather than changing actual reviewer assignments.¹⁶

A separate multi-pass experiment split diffs into multiple inputs and reconstructed the original content for all 60 backend and frontend cases in an offline check. That proves input reconstruction, not better review selection. The quality comparison was not completed, and this experimental path was not adopted for the default router or real review dispatch.¹⁷

Limits of the implementation

First, long diffs are truncated. The tool sends at most 6,000 characters per file and 60,000 characters in total to Jev. Code changes skip-review to coderabbit-only when input is incomplete and emits a separate signal that a stricter review might be warranted. This cannot recover the missing content. In a real-change sample, adding cross-provider review to every incomplete input repaired no missed high-stakes case, so we did not adopt that rule.¹⁸¹⁶

Second, model-version checking was weak. An environment variable could override the default model. If a valid response reported a different model, the router set its output model field to null but still selected review lanes from that response. It did not report a mismatch reason or switch to deterministic fallback. A version-sensitive run should reject the response and use the fallback instead; that guard had not been implemented.¹⁵

Third, the data boundary needed to be clearer. The title, file paths, and actual diff were sent to the external TypeSafe API. The contract kept raw credential values out of output, telemetry, and evidence. The usage collector did not yet have a routing-specific event. The router had no dedicated result store and printed results to standard output. We could not infer from this code whether a higher-level agent, session, or logging layer retained that output. Sensitive-file and diff-transmission policies needed separate design.¹⁹

Fourth, the evaluation sample was small. Paired Korean-English outputs did not tell us the false-positive or false-negative rates on the full workload. To make a claim about reallocating review cost, we would also need to record actual reviewer time, valid defects found by extra reviews, and review paths that should have been selected but were missed.

Fifth, the frontend was not wired into actual review dispatch. A separate shadow sample missed some expected exact specialist lanes and included substantial input truncation. Frontend work has a different mix of UI changes, generated code, and copy edits; we should not change actual assignments until the outcomes and costs are better understood.¹¹⁶

One boundary remains: Jev does not choose the next development task, prove that tests passed, or declare work complete. Those would be different systems requiring their own definitions and evidence, not natural extensions of this router.

Next: observe → shadow → calibrate → expand

The order of improvements is about building evidence safely, not accumulating features.

Observe. With user consent, store an opt-in routing receipt without storing the diff. Record the model used, whether input was truncated, which signals and code rules selected the reviews, and why a fallback occurred. Exclude sensitive files before transmission or require separate approval.

Shadow. In the frontend, produce a recommendation without automatically invoking specialist reviewers. Compare it against the reviews selected by the current process. Keep the existing rule that incomplete input cannot result in skip-review.

Calibrate. Define what counts as an unnecessary review and a missed necessary review in real work. Agreement between the current process and the model is not enough; measure whether reviews found valid defects and how much time they took. Treat confidence as concentration of a response, not as correctness.

Expand. Only connect frontend dispatch if both the cost and the risk of missed reviews are acceptable. For a mismatched model version, a sensitive file, or truncated input, return to deterministic rules. Even then, Jev’s output is not evidence and does not eliminate independent review.

Conclusion

We kept the review router and discarded the claim gate. The latter wrongly blocked 4 of 12 normal messages at the tested threshold. Later, automatically adding a review for every incomplete backend diff repaired no missed high-stakes case in our sample and added three reviews of ordinary changes. Whether the router saves reviewer time or catches more valid defects remains unmeasured.¹⁶

---

Measurement notes and public sources

Footnotes

1. Source: a comparison of the recatch-tdd PR workflow with the live Re:catch backend, Athena standalone backend, and Re:catch frontend packs. Limit: this applies to the checkout examined when the article was written. The absent frontend wiring was checked using the same search with positive controls in the backend packs. ↩ ↩² ↩³

2. Source: the introduction and quick summary in the recatch-tdd README. Limit: this describes Re:catch’s team-specific process, not a universal definition of TDD or a product feature. ↩

3. Public sources: TypeSafe System One, TypeSafe Models, Noul, Score, and Choice. Limit: Jev is a System One model, not an LLM; it consumes natural-language text and returns structured judgments and likelihoods. “Small AI” describes the limited task assigned here, not the model’s parameter count. Provider documentation defines its API and model properties but does not independently validate Re:catch’s routing quality. ↩

4. Source: initial review-gates experiment records. Limit: the 20 merged PRs were a convenience sample; 12 backend and 8 frontend PRs do not represent the distribution of all changes. ↩

5. Source: the initial experiment’s lane counts for 12 backend PRs. Limit: 36 is the upper bound if three specialist reviews ran on every PR, and 12 is the router’s selection count. The 67% figure is not a reduction from observed historical operations. ↩

6. Source: the Jev client, lane router, and backend PR workflow’s failure-handling and deterministic fallback contract. Limit: continuing after provider failure does not bypass existing checks or publication approval. ↩ ↩²

7. Source: the router’s question definitions, path rules, lane-composition function, and behavior tests. Limit: thresholds are current policy values, not universally calibrated numbers. ↩

8. Public sources: TypeSafe Confidence, Noul, Score, and Choice. Limit: provider documentation defines the API meaning, not Re:catch’s routing quality. The 0.80 threshold is the current promotion rule in Re:catch code. ↩

9. Source: the initial router sample and its v1→v2 corrections. Limit: path labels are also heuristics, so they were not treated as ground truth for model accuracy. ↩ ↩²

10. Source: an earlier internal managed-skill consolidation experiment: 160→132 skills, 63,225→54,034 characters of instructions, approximately 3,468 tokens saved per call, 642 requests, and estimated judgment cost of $0.31. Limit: token and cost figures were estimated from input length, not measured provider usage; this was not a recatch-tdd runtime experiment. ↩

11. Source: the claim-gate sample, paired trials, mutation trials, and repeatability trials. Limit: zero target failures among 889 sessions covered only the specific command-claim pattern; 12 normal messages and five repetitions are small samples. ↩

12. Source: corrected paired Korean-English routing evaluation. Limit: matching final lanes in 12/12 cases and preferred first lane in 10/12 do not measure accuracy. The maximum 0.05 same-input signal difference came from one repeat of one Korean input. ↩ ↩² ↩³

13. Source: provider usage for the corrected evaluation: 25 requests and 313,151 input tokens, multiplied by the then-published $0.042 per million token price for approximately $0.013. Limit: calls from the discarded first English control are excluded. ↩

14. Source: the approximate request size from the initial router experiment and the then-published provider price. Limit: about 10k input tokens and $0.0004 per PR depend on diff size, questions, and current model pricing. ↩

15. Source: the client and router code and TypeSafe Models. Limit: the default was jev-1.13.0, but an environment override meant it was not pinned. A response-model mismatch produced model: null while routing still used the validated answers; it was not a failover. Provider prices and aliases can change. ↩ ↩²

16. Source: separately annotated shadow samples of 30 merged backend PRs and 30 frontend PRs, plus a comparison of an alternative policy on the same backend results. Limit: these are convenience samples with different size and time distributions. Three extra backend ordinary assignments are a proxy for reviewer load, not measured false positives, defect yield, or billed cost. Frontend exact-lane 6/10 and any-strict 9/10 are distinct metrics. No automatic specialist dispatch was performed. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

17. Source: an offline comparison of recombined diff chunks with fixed source patches for 30 backend and 30 frontend changes. Limit: 60/60 demonstrates input reconstruction, not Jev judgment accuracy or actual reviewer outcomes. The separate quality evaluation was not completed, so the multi-pass path was not adopted as the default. ↩

18. Source: the diff collector’s input_complete computation and the lane-composition branch for incomplete input. Limit: the 6,000-per-file and 60,000-total limits are characters, not tokens or context limits. Incomplete input avoids skip-review but does not automatically add a specialist reviewer. ↩

19. Source: the collector schema, data-capture boundary, router standard output, and the recatch-tdd credential contract. Limit: usage telemetry held session, skill, and judgment events but no routing event. The router had no separate storage path and printed its output, but this does not establish whether higher-level agent, session, or logging systems retained it. Receipts and sensitive-file policies were proposals at the time. ↩

Close ×