AI Code Reviewer
Build a GitHub/GitLab bot that reviews PRs with explicit rules, grounded context, and quality tracking.
Design & Review Strategy
Choose model, define review rules from team history, and design the PR comment format.
Why
An AI reviewer's value is in the rules, not the model. Without explicit rules (max function length, banned APIs, security patterns), the LLM gives generic 'consider...' comments that developers ignore. Precise rules plus grounded context (diff + related code) produce actionable reviews that get resolved, not dismissed.
When
Before connecting the bot. Spend 1 week defining 10-20 rules from your team's actual review comment history.
How
1. Mine your review history
- Export 6 months of PR review comments from GitHub/GitLab
- Cluster by theme: security, performance, naming, error handling, tests
- The top 10-20 recurring themes become your rules
2. Write rules as structured checks
- Each rule: { id, description, severity, check(diff, context) -> Finding[] }
- Example: no-console-log -> flag any added console.log in src/
- Mix static rules (regex/AST) with LLM-assisted rules (semantic)
3. Choose the model
- gpt-4o-mini for classification (cheap, fast, good enough for rule checks)
- Claude for nuanced review (security reasoning, architecture feedback)
- Cache rule results; only call the LLM for rules that need semantic judgment
4. Design the comment format
- One comment per finding, posted as a reviewable line comment (not a wall of text)
- Format: [Rule: id] severity icon + 1-sentence problem + suggested fix
- Group findings by file; cap at 5 comments per PR to avoid noise
5. Define the review trigger
- On PR open + on new push (re-review the diff)
- Skip drafts and bot-authored PRs (renovate, dependabot)
- Respect .ai-reviewer.yml for per-repo rule overrides
Example
// Structured rule
const rules = [
{
id: "no-console-log",
description: "No console.log in production code",
severity: "warn",
check(diff) {
return diff.addedLines
.filter(l => l.code.match(/console.log(/))
.map(l => ({ file: l.file, line: l.number,
message: "Remove console.log or use the logger" }));
},
},
{
id: "unhandled-async",
description: "Async calls without try/catch or .catch",
severity: "error",
async check(diff, context) {
// LLM-assisted: ask Claude to find unhandled awaits
return await claude.findUnhandledAwaits(diff, context);
},
},
];
// .ai-reviewer.yml (per-repo override)
rules:
no-console-log: off # this repo is a CLI, logging is fine
unhandled-async: error
max_comments_per_pr: 5
Common Problems
- ⚠No rules — the LLM posts generic 'consider adding a comment' on every PR, developers dismiss them within a week
- ⚠Posting a wall of text instead of line comments — developers cannot action a 20-line essay, they skim and miss real issues
- ⚠Reviewing bot PRs (dependabot) — the bot posts on the dependency bot's PRs, creating infinite comment loops
Best Practices
- ✓Mine 6 months of your team's review comments to define rules — the bot should enforce your team's standards, not generic ones
- ✓Post findings as reviewable line comments capped at 5 per PR — signal-to-noise ratio determines whether developers read the bot
- ✓Skip draft PRs and bot-authored PRs in the webhook handler to avoid noise and comment loops
Resources for this stage
Tools
Templates
Implementation & PR Integration
Build the webhook, diff parsing, and comment posting with rate limits and grounding.
Why
The webhook loop is where bots fail — posting on every push spams PRs, posting without throttling hits API limits, and posting ungrounded comments (not tied to specific lines) trains developers to ignore the bot. Batch reviews per PR and only comment on changed lines.
When
After rules produce good findings on sample diffs. Connect the webhook and run on a test repo before pointing at the main repo.
How
1. Set up the webhook receiver
- GitHub App (preferred over bot account for fine-grained permissions)
- Webhook: pull_request (opened, synchronize) + pull_request_review
- Verify webhook signature with X-Hub-Signature-256
2. Parse the diff
- GET /repos/:owner/:repo/pulls/:number/files (paginated)
- For each file: extract added lines with line numbers
- Skip generated files (*.min.js, *.lock, dist/) via .gitattributes
3. Run rules and collect findings
- Run static rules first (fast, no LLM cost)
- Batch LLM-assisted rules: one prompt with all relevant diffs + context
- Ground each finding to a specific (file, line) from the diff
4. Post reviewable comments
- POST /repos/.../pulls/:number/comments with position (line in diff)
- Use the GitHub Review API (POST /reviews) to batch comments in one review
- Mark the review as COMMENT (not REQUEST_CHANGES) unless an error-severity rule fires
5. Handle rate limits and retries
- GitHub: 5000 req/hour for apps; queue reviews if approaching limit
- LLM: cache findings by (rule_id, diff_hash) to skip re-review on re-push
- Idempotency: delete the bot's previous comments before posting new ones
Example
// Webhook handler (GitHub App)
app.webhooks.on("pull_request.opened", async ({ payload }) => {
if (payload.pull_request.draft) return; // skip drafts
if (payload.pull_request.user.login.includes("[bot]")) return; // skip bots
const diff = await github.getDiff(payload); // paginated files
const findings = [];
for (const rule of rules) {
findings.push(...await rule.check(diff, { repo: payload.repository }));
}
const top = findings.slice(0, 5); // cap at 5
// Post as a single review with line comments
await github.rest.pulls.createReview({
owner, repo, pull_number: pr.number,
event: findings.some(f => f.severity === "error") ? "REQUEST_CHANGES" : "COMMENT",
comments: top.map(f => ({ path: f.file, line: f.line, body:
`[${f.ruleId}] ${f.severity === "error" ? "🚫" : "⚠️"} ${f.message}` })),
});
});
Common Problems
- ⚠Posting on every push without dedup — a 10-commit PR gets 10 identical reviews, spamming the timeline
- ⚠Ungrounded comments (no line position) — posted as a general PR comment, developers cannot resolve them inline
- ⚠Hitting the GitHub API rate limit mid-review — the bot silently fails and the PR gets no review at all
Best Practices
- ✓Cache findings by (rule_id, diff_hash) and delete old bot comments before posting — makes re-pushes idempotent
- ✓Use the GitHub Review API to batch all comments into one review event, not individual comment posts
- ✓Queue reviews when approaching the API rate limit and process from a queue (BullMQ + Redis) instead of failing silently
Deployment & Tuning
Deploy as a serverless function or container, track review quality, and tune rules weekly.
Why
AI reviewers degrade silently — a model update changes comment style, or a new codebase pattern triggers false positives. Without tracking 'comments resolved vs dismissed' and a weekly rule review, the bot's signal-to-noise ratio drops until developers disable it.
When
When the bot posts on real PRs. Track quality metrics from day 1 and tune rules weekly based on the data.
How
1. Deploy the webhook receiver
- Cloud Run or Fly.io: container with the GitHub App, scales to zero between PRs
- Set min-instances = 0 (cost: pay per review), max = 5 (handle PR open storms)
- Store the GitHub App private key in the secret manager
2. Track review quality metrics
- For each posted comment: store { rule_id, pr, posted_at, resolved_at, dismissed }
- resolved = developer fixed the issue; dismissed = developer clicked "resolve" without fixing
- Dashboard: resolve rate per rule, dismiss rate per rule, total comments/week
3. Tune rules weekly
- Review the dashboard: rules with > 50% dismiss rate are noisy — tighten or remove
- Rules with 0 findings in a month may be stale — check if the codebase moved on
- Add new rules from recent manual review comments not covered by existing rules
4. Handle model updates safely
- Pin the model version (gpt-4o-2024-08-06) so an update does not silently change behavior
- On model upgrade: re-run the bot on 50 historical PRs and diff the findings
- Roll back if the new model produces > 20% more dismisses
5. Gather developer feedback
- Add a thumbs-up / thumbs-down reaction to each bot comment
- Monthly survey: "is the bot helpful?" 1-5 scale
- If average drops below 3, pause the bot and do a full rule review
Example
// Deploy on Cloud Run (scales to zero)
// Dockerfile -> gcloud run deploy ai-reviewer --source . --min-instances 0 --max-instances 5
// Quality tracking (Postgres)
await db.comments.insert({
rule_id, pr_number, repo, posted_at: new Date(),
status: "open", // -> "resolved" | "dismissed"
});
// Weekly quality report (cron job)
const rules = await db.query(`
SELECT rule_id,
AVG(CASE WHEN status = 'resolved' THEN 1 ELSE 0 END) AS resolve_rate,
AVG(CASE WHEN status = 'dismissed' THEN 1 ELSE 0 END) AS dismiss_rate
FROM comments WHERE posted_at > now() - interval '7 days'
GROUP BY rule_id
`);
// Alert Slack if any rule's dismiss_rate > 0.5 — too noisy, needs tuning
Common Problems
- ⚠No quality tracking — the bot posts for months, dismiss rate climbs to 80%, developers mute it and nobody notices
- ⚠Unpinned model — a silent model update changes comment tone, false positives spike, trust evaporates overnight
- ⚠No developer feedback channel — developers find the bot annoying but have no way to signal it, so they disable it silently
Best Practices
- ✓Track resolve vs dismiss rate per rule from day 1 — dismiss rate is the leading indicator of bot trust
- ✓Pin the model version and re-run on 50 historical PRs before upgrading — catch behavior changes before they hit production
- ✓Add a thumbs-down reaction to every comment and survey monthly — if helpfulness drops below 3/5, pause and review rules
Resources for this stage
Tools
Templates
Full Developer Lifecycle
A 14-stage developer roadmap from idea to maintenance — visualize dependencies and track progress.
Next Step