Pull request reviewer
Every team that switches on an automated reviewer learns the same lesson in week one: the problem is not finding things to say, it is saying nothing when there is nothing worth saying. A reviewer that comments on every PR gets muted like every linter before it.
The job on one screen
| Runs when | A pull request is opened or updated. |
|---|---|
| Reads | The diff, the files it touches, the test results, and the project conventions. |
| Decides | Which findings are real defects, and which are opinions it should keep to itself. |
| Produces | Inline comments with a reason and a suggested fix, or approval with no comment. |
| Stops when | Findings posted, or nothing found worth a human's attention. |
| Tools it needs | Repository API, test runner, linter output. |
| Autonomy to start at | Approve. It does the work and stops before the irreversible step. |
Optional: load a real model
Run it
The routing rules
These are the exact buckets the demo above scores against. Change the wording of a ticket and you can watch the confidence move.
| Bucket | Signals it looks for | What happens next |
|---|---|---|
| Security defect | sql, concatenation, injection, eval, exec, secret, token, password | Blocking comment with the specific line, the attack, and the parameterized fix. |
| Correctness risk | retry, no maximum, backoff, race, null, off by one, unbounded, timeout | Blocking comment. Retries without limits, unbounded loops, and missing error paths belong here. |
| Missing test | no test, without a test, untested, feature flag, off path, coverage | Non-blocking comment naming the untested path. Suggest, never demand. |
| Style | rename, format, imports, whitespace, naming, reorder, comment typo | Say nothing. The formatter owns this, and a reviewer that argues about imports gets ignored on the day it finds a real bug. |
| Below the bar | no clear signal | Anything it cannot categorise gets no comment at all. Silence is the correct default for a reviewer that is not sure. |
Where this one goes wrong
Confident nonsense
A reviewer that describes a bug that is not there costs more trust than it can earn back. Require it to quote the line and state the failing input.
Reviewing the whole file
Comments on unchanged code annoy everyone. Scope strictly to the diff.
Blocking on taste
If it can block merges, it will eventually block one on an opinion. Make only two categories blocking.
How you would know it is working
| Measure | Why that one |
|---|---|
| Comments resolved as valid | The only quality measure. Under 50% and developers stop reading it. |
| Defects found before review | Compare against what humans caught afterwards. |
| Comments per PR | Should be under two on average. More is noise, not thoroughness. |
Earning more rope
| Assist | Post findings as a summary comment, non-blocking. |
| Approve | Inline comments, still non-blocking, humans resolve. Right place for most teams. |
| Auto | Blocking only on the security category, with an override that requires a written reason. |
Related: Flaky test hunter · Testing agents · Dependency upgrades · all agent jobs · Agent Lab home
Free from AI School - no signup, everything runs in your browser.