How we build Relevan with Claude Code
More than half the commits on Relevan’s main branch include a Claude co-author line. We designed the repo around the agent to make that work. This post walks through how we did it, with patterns other teams can adopt.
Our main workflow
We use Claude Code with a copy of Matt Pocock’s engineering skills in .claude/skills, pinned to a specific upstream commit. Our CLAUDE.md defines the primary workflow:
It starts with /grill-with-docs. The agent interviews us about the idea, asking one question at a time, and records each decision in the glossary and plan. The learning loop plan preserves our answers verbatim, including decisions like “staff queue is not realistic.” That matters because a future session shouldn’t have to revisit decisions we’ve already made.
From there, /to-spec and /to-tickets translate the plan into GitHub issues. Each ticket represents a thin vertical slice of work and identifies the tickets it depends on.
/implement picks up one ticket and follows a test-first approach at the seams: the public boundaries we’ve agreed to test ahead of time. Once implementation is complete, /review-diff runs two fresh sub-agents against the coding standards and original spec.
After review, /intent-check compares the finished work with the original plan. We cover it in more detail below.
The final step is /retro, and it’s easy to overlook. Its purpose is to improve the environment the agent works in. For example, we discovered that /to-tickets was creating tickets without linking them to a parent issue when working from a plan file. We fixed it by adding a section to our tracker documentation. That single change improved every subsequent session.
Pattern 1: Hooks that enforce rules
Instructions in CLAUDE.md tell the agent what it should do. Hooks let us enforce what it can do.
For example, we only create Git worktrees under .claude/worktrees/. A Claude Code PreToolUse hook enforces that constraint:
When a requested path falls outside the allowed directory, the hook returns a deny:
jq -n --arg allowed "$allowed" '{
hookSpecificOutput: {
hookEventName: "PreToolUse",
permissionDecision: "deny",
permissionDecisionReason: ("Create git worktrees only under " + $allowed
+ "/<name>. Use EnterWorktree, or give git worktree add a path there.")
}
}'The denial message is just as important as the restriction itself. Claude reads the reason, so we make it an actionable instruction that explains how to proceed. With that instruction, the agent can correct its next call and continue on its own.
We use two additional hooks for code quality. One formats and lints files immediately after they’re edited. The other runs linting across every changed package before the agent finishes.
Pattern 2: Skills for decisions that require judgment
Not every rule can be enforced with a simple hook. Some require context and judgment.
Database migrations are a good example. A hook can detect that a migration is being generated, but it can’t reliably determine whether the schema change is safe. That’s where our schema-namespace-check skill comes in.
The skill walks the agent through identifying breaking changes, including dropped columns and new NOT NULL columns without defaults. Breaking changes can be isolated in their own schema namespace, preventing them from blocking other developers.
We also use a hook to make sure the agent loads the skill before generating a migration:
if (/sql:generate/.test(command) && !/\bSCHEMA_CHECKED=1\b/.test(command)) {
// deny: "Load the schema-namespace-check skill and classify the change
// before you generate a migration. Then run the command again with
// SCHEMA_CHECKED=1 at the start."
}The distinction is simple: hooks enforce the requirement, while skills provide the process for making the decision.
Multiple agents, one pipeline
We don’t limit ourselves to running one agent at a time.
We also use Capy , where a Captain agent coordinates a crew of agent threads. Each thread gets its own machine and produces its own PR.
This post is one example. A Captain and three crew threads handled the writing, rendering, and review of this article, and we edited the final draft by hand.
Regardless of how many agents are working, every PR follows the same validation pipeline once it’s ready for review:
- Automated review. Capy reviews every push automatically. The agent responsible for the PR triages each finding.
- CI and deployment validation. CI runs formatting, linting, and tests. If a PR changes more than documentation, we deploy the affected packages to an isolated dev stack for that PR and run our Playwright end-to-end tests against it.
- Post-merge validation. After merging, main deploys to dev and runs the full end-to-end suite. Production deployment only happens after those tests pass.
The goal is to make the validation process consistent regardless of whether a change was written by a person, an agent, or a combination of both.
Our customers’ agents get the same treatment
We apply many of these principles to the agents that interact with Relevan, too.
Our API exposes four skills at /skills/<name>/SKILL.md: setup, ranking signals, mapping, and feedback. Our llms.txt indexes each skill, and every search index gets its own skill document , generated from its live mapping.
The documents reference one another, and we have a test that fails whenever the installable bundle drifts from its source.
We also test this experience from the perspective of a customer’s agent.
In October, we gave fresh agents access to only our public documentation and API. Their task was to build and tune a product search index. They uncovered six issues.
The most significant was a hidden monthly quota that caused tuning to stop halfway through. We addressed it by documenting the quota in llms.txt and adding an X-Search-Quota-Remaining header to every search response on capped plans.
Testing the product this way exposed problems that our internal development process hadn’t caught. It also gave us a better understanding of what an agent actually needs to integrate with Relevan successfully.
Checking the work against the original intent
Code review is effective at evaluating what’s directly in front of it. It catches bugs, examines implementation details, and challenges architectural or design decisions. What it doesn’t easily catch is whether the completed work still fulfills the original plan. Individual PRs can be technically correct while the overall effort gradually drifts from its intended outcome.
To address that, we added /intent-check to our main workflow. It runs after implementation and review, comparing the code against the original plan and its associated tickets. The plan defines the intent, the tickets define the claims, and the code and recorded results provide the evidence. A claim in a PR description can point to evidence, but it isn’t evidence by itself.
For every acceptance criterion, the agent assigns one of three verdicts: met, partial, or not met. It evaluates each criterion against four questions:
- Could the test actually fail?
- Did the code run against the data specified by the criterion?
- Is the test excluded from the default test run?
- Does the implementation fulfill the intent of the plan, or does it only satisfy the literal wording of the ticket?
It also evaluates the plan itself, including whether its exit criteria are actually achievable.
Findings are posted as comments on the parent issue and each affected ticket. The agent labels each one a potential gap, and a person decides which ones are real problems.
We ran /intent-check against the first two steps of our learning loop plan after all associated PRs had passed review. It uncovered two issues that had gone unnoticed.
First, the plan contained a success criterion that was mathematically impossible to achieve. The allowed set of variants could produce at most 28 counted passes, while reaching a pass required 29.
Second, the only coverage run used human grades as its evaluator. In effect, it was comparing human grades against themselves, making it impossible for the evaluation to identify a false pass.
Neither issue would have been caught by reviewing the individual PRs. The code in each one was correct. The problem was that the implementation, taken together, couldn’t deliver what the plan promised.
What’s next
The following are planned improvements.
- Evals as PR gates. We already use our open evaluation projects, decision-model-eval (results) and shots-to-success , to inform tuning decisions. The next step is running these evaluations automatically on PRs that modify ranking behavior or agent-facing documentation.
- The learning loop. Today, we train per-index ranking models offline using customer feedback. Plan 0043 introduces scheduled training runs, offline evaluation gates for each tuning change, and validation against live traffic. For lower-traffic indexes, an expert review will serve as the final check before a change goes live.
What we’d recommend to other teams
- Write hook denial messages as instructions. The agent reads them, so give it a clear path forward.
- Pair hooks with skills when enforcing a rule requires judgment or additional context.
- Give every automated review loop a maximum number of rounds and a clear handoff to a person.
- Check the finished work against the original plan as well as your coding standards. Code review tells you whether the code is good. An intent check tells you whether it does what you set out to build.
- Run retrospectives and improve the environment the agent operates in. Fixing the workflow often has more lasting value than fixing a single agent mistake.
The biggest takeaway is that good results from coding agents depend on the engineering systems around the model. Instructions, constraints, feedback loops, and validation gates are what make its work reliable.
If you’re building an agent that searches, start with our llms.txt. The relevan-setup skill can walk your agent through building its first index.
You can also try our free tier or book a call to talk through your setup.