Pasi Huuhka on why building an automated PR reviewer is mostly normal event-driven software engineering, with the LLM as just one bounded stage in the pipeline. Pattern generalises to any AI automation with a clear start, process, and end.
Summary
The AI part of a PR review bot is surprisingly small. Most of the system is receiving events, gathering context, running contained workflows, parsing structured output, and posting results back. The key insight: let code own the workflow, let the model own the review judgment.
The architecture uses a manager/worker pattern - an event-driven manager layer (Azure Functions) handles triggering, dedup, state, and result posting, while isolated workers (Container Apps jobs) run the actual LLM review. Each review is a contained execution with its own failure boundary.
Inside the LLM run, a primary agent fans out to parallel specialist sub-agents (architecture, security, testing), then synthesises their structured findings. The Copilot SDK is a natural fit because it already provides file reading, code search, and command execution capabilities.
Key points
- Mostly control flow, not AI. The bot is normal event-driven software with an LLM in one stage. The manager decides when to review, what context to gather, which agent setup to use, how retries work.
- Coding harnesses fit naturally. The model needs to read files, inspect diffs, search the codebase, and optionally run commands. Copilot SDK, OpenCode, or even raw CLIs all work.
- Specialist sub-agents outperform one giant prompt. Fan out to parallel reviewers (architecture, security, testing) then synthesise. Copilot SDK’s custom agent support maps directly to this pattern.
- Structured output with severity is essential. Models get verbose. Add explicit severity levels (critical/high/medium/low) so your code can filter noise before posting to the PR.
- Skills are reusable across local and CI. Package domain guidance as skills that developers also use locally. Reduces drift between local agent usage and server-side automation.
- Same architecture powers a fix command. Swap the trigger and prompt - the manager, state, isolated execution, and callback flow all reuse.
The architecture
- Source system event arrives (PR created, updated, commented)
- Manager classifies whether work should start
- Manager gathers context, stores run state in DB
- Manager starts isolated worker run (container job or microVM)
- Worker runs LLM review with specialist sub-agents
- Worker returns structured findings
- Manager parses result, posts comments back to source system
- Manager updates run state
The LLM never decides when jobs start, how retries work, where state lives, or how comments are posted. That is all deterministic application logic.
Lessons worth keeping
- Treat the LLM like another API dependency. The tighter the guardrails, the better for bounded tasks.
- One giant agent is worse than a primary + parallel specialists pattern. Parallelism helps latency more than it hurts cost.
- Build automations on the same harness you use locally - faster feedback, easier debugging, less drift.
- The pattern generalises to issue triage, ticket classification, bug reproduction, documentation generation, and anything with a clear trigger and bounded reasoning.
Related
- L8 Principal’s Agentic Engineering Setup - another practical coding-agents-in-production perspective, same emphasis on architecture over model
- How to Automate AI Evals (Correctly) - structured output and bounded agent tasks from the Anthropic side
- I Built an LLM From Scratch - understanding what’s under the hood helps design better LLM pipelines
Sources
huuhka.net/building-your-own-pr-reviewer-with-coding-agents Full text: building-own-pr-reviewer-coding-agents