How an in-house review bot cut our PR merge time in half

Engineering work and experiments by Lat Ware, Member of the Technical Staff

When Lat Ware started building what he named the Pedantic Review Bot, a pull request at Vapi took an average of over 130 hours to go from open to merged.

That number changed how Lat thought about automated review. Vapi already paid for two AI review tools, and neither caught what his early prompts caught on Vapi’s codebase.

“I was competing with review times measured in days, so I could let the bot spend 20 minutes on a review, which the other products could not do.”

— Lat Ware

Building in-house let him weigh the cost of each review against the issues it found. Pedantic Review Bot now reviews every pull request Vapi engineers open, and the average time from open to merge has fallen in half, to 65 hours.

How a review works

The bot runs when an eligible pull request opens, including drafts. Authors can tag the bot for another pass. It skips commits it has already reviewed, so duplicate events don’t trigger a second paid review.

Each inline finding starts with a bold, one-sentence summary and a severity marker, from green for a nitpick to red for a blocker.

A second bot, Pedantic Architecture Review Bot, looks for design problems. Lat built it after a design flaw got past the original bot. “It was a design problem that no single line of code revealed,” he says. It uses almost the same prompt as the main bot, asking for “possible high-level design pattern problems” instead of “possible problems,” and needs a higher effort level to work. The architecture bot writes a report and keeps inline comments for findings tied to a specific location. A complexity filter limits when it runs: between Sep 2 and Oct 2, 2026, 763 architecture workflow runs produced 33 reports.

How it got its name

The first version left so many comments that Lat named it Pedantic Review Bot. Many of them concerned problems that already existed in the code a pull request touched.

“People complained that its reviews read like careless AI output, and they had a point.”

— Lat Ware

Lat revised the prompt repeatedly to make comments shorter and easier to act on. The complaints faded as engineers read the reviews closely and found that the findings were accurate. Fixing existing problems in the code being changed became part of the team’s maintenance work. “We can’t fix everything at once, so the bot limits that work to the code we are changing right now,” he says.

Making the bot prove its claims

The bot doesn’t report an objection until it has checked it. The analysis job checks out the pull request, and before reporting a finding, the agent can trace callers, run the tests it suspects will fail, or write a small reproduction.

Analysis and posting run as separate jobs. The analysis session has its Git credentials stripped and writes its findings to a JSON file. A second job validates that file and posts the comments without running anything the agent wrote. It can comment, but it can’t approve or push; engineers decide what to act on.

A month of findings

Between Sep 2 and Oct 2, 2026, the bot reviewed 1,694 pull requests and raised 363 blocker findings across 276 of them. More than half of those blockers involved builds, tests, CI, or deployment. Others concerned runtime behavior and billing.

Several of the test findings were problems a green check would hide. The bot flagged a change that replaced test assertions with Storybook stories the test runner never collects, and a benchmark that reported success without running a single test.

In one traffic-splitting change, the architecture bot caught two writes in the wrong order. The backend switched traffic to the new version first, and only then applied “publish at 0%.” For that window, a release meant to get no traffic got all of it.

101 versions of the prompt

Lat has experimented with nearly 200 prompts for the bot. In one documented experiment, he tested 101 versions of the opening instruction on a single pull request and kept everything else identical: the context about the codebase, the diff, the scoping rules, and the JSON output format.

They produced 803 raw findings, which grouped into 57 distinct issues. An AI judge checked each issue against the code without knowing how many runs had reported it: 32 were judged real, 24 false, and one unverifiable. No single run found more than 10 of the 32 real issues.

The report initially named a more aggressive prompt as the winner. Lat rejected that conclusion because the scoring didn’t penalize false positives enough. A prompt that reports more problems scores higher, but every extra claim, true or false, is one more thing an engineer has to check. Once false positives were weighed the way Lat intended, the original, ultra-simple prompt won.

This is the prompt he says performed best across his experiments:

Lat’s winning prompt — emoji shown for readability

Please aggressively review [PR link] for possible problems.
Post what you find as inline comments.
Don't sacrifice on making the problem easy to solve.
Give a one-sentence summary in bold at the top of each comment.
Assign a severity rating from the range 🟢 🟡 🟠 🔴,
with 🟢 being nitpicks and 🔴 being blockers.

The original prompt used emoji shortcodes.

On this pull request, instructions about what to inspect and how to report it performed better than reviewer personas. Spelling out the exact severity tokens also made the formatting more consistent.

Several cheap runs beat one expensive run

Since no single run found everything, Lat wanted to know how best to spend a fixed budget. He also wanted to compare models on Vapi’s own code rather than rely on published results. “Those benchmarks are measured on tasks the providers choose, and those tasks can look very different from our codebase,” he says.

For this experiment, he refactored a production service that contained a known blocker. The refactor moved a termination guard out of the service’s entrypoint, but a repository lint test still required the guard there, so the change failed the lint gate. The runs that caught it ran the scanner or lint suite against the code; the diff alone gave no hint of the rule.

He ran code-review and design-review configurations against this refactor repeatedly, at different effort levels. Findings were checked against the source and weighted by severity, so a nitpick counted less than a serious defect.

In both comparisons, three extra-high runs found more than one maximum-effort run, for the same cost or less. Each extra run repeats more of what earlier runs found, so the gains shrink.

The known blocker was much harder to find. Only eight of 194 code-review runs caught the lint failure. Two of 195 design-review runs did.

What counts as in scope

Several GPT-6 configurations reported almost nothing on the unchanged prompt. Their reports said that problems already present in the parent commit were outside the review’s scope. Lat added one sentence:

Include errors already present in the parent.

In the code-review experiment, GPT-6 Astra’s average rose from 0.3 to 24.2 severity-weighted points per run. The other models never received that instruction, so their scores aren’t a fair comparison. The sentence can also overcorrect: in the design experiment, every point GPT-6.1 Sol earned concerned a pre-existing problem, and none concerned the pull request’s own design.

A useful finding says whether the pull request introduced the problem or whether it already existed. An engineer may want to fix either, but needs that distinction to decide whether to merge.

What we still need to measure

Each prompt in the 101-version experiment ran once, on one pull request, and an AI did the judging. The model comparison also used a single pull request, so both need repeating on others before the results can be generalized. Some architecture reports in the monthly data were deliberate re-runs on pull requests merged months earlier, so they don’t count as catches before merge.

The termination-guard result shows a run can be mostly right and still miss the one finding that blocks the merge. An evaluation with deliberately seeded defects would test for that directly.

The activity report also doesn’t yet track which findings engineers acted on or dismissed. Those outcomes would show how often the bot’s comments lead to useful changes. For now, Lat reports one production outcome.

What we’d tell a team building their own

The models in these results will be out of date within months, and Lat plans to rerun the experiments as new ones ship. These lessons should last longer:

• Your review queue sets your latency budget. If humans are taking days, the bot can take minutes as an improvement.
• Make the reviewer run code instead of simply reading.
• Keep the agent that reads untrusted code away from the token that posts comments.
• Count false positives as a cost. In Lat’s case this led to the simple prompt winning.
• Try several cheaper runs before paying for maximum effort.
• Tell the reviewer whether pre-existing problems are in scope.

“I make decisions from data, so I enjoyed being able to run experiments that showed me how well these tools perform for us. Experiments are where the truth comes out.”

— Lat Ware