Running Our Docs Like a Newsroom

If you’ve ever written product documentation, you know the drill, and sometimes the pain, of trying to understand what your dev team has changed, translating that into content your customers can understand, and getting it reviewed and published before that new code is released. And now, in the world of coding agents, new features and bug fixes can ship faster than ever before — far more than any one writer can keep up with on their own.

Luckily, with LLMs and WarmHub, we’ve been able to assemble our own agentic technical writing team: a writer named Nora, a fact-checker named John, and an editor named Wiley. Three agents, three very different jobs, and one rule none of them can break: nothing they produce touches the docs site until a human reviews and merges it.

Here’s how the team works, what it’s caught, and what you should know if you’re thinking about using AI for documentation.

It starts with a finding

One thing to know up front: our docs are docs-as-code. We build the doc site ourselves with Astro, and every page is a file in GitHub — which is why, throughout this story, “open a pull request” is literally how our documentation changes.

Every time a code PR merges, two documentation review skills we created — Doc Accuracy and Doc Content — read the merged code diff. The accuracy review skill evaluates whether the change made any page wrong, or shipped something the docs don’t cover: it maps what the diff actually touched — CLI commands, API routes, SDK surfaces — to the pages that document them, and compares those pages against the source of truth rather than against memory. The content review skill reads the same change with fresh eyes, the way a first-time reader would: is this understandable, well-structured, on-brand? Is there jargon nobody defined, a page that only makes sense if you’ve already read three others?

Every concern either skill raises becomes a finding — a durable record with its own identity and lifecycle, written to a product-docs repo in WarmHub. Findings carry a priority, because a wrong command in the quickstart is not the same as a sentence that could read a little smoother — and priority decides the route: urgent findings ride the next daily batch, while the low-priority ones queue up for a weekly pass.

Most findings shouldn’t become work. Some are wrong, some are trivial, and some describe a problem that got fixed an hour later. Reviewer models are enthusiastic, and if you turned every flag into a task, you’d bury a writer, human or otherwise, in plausible-sounding busywork.

A finding is a hypothesis, not a work order. Most of what a reviewer model flags should never become a pull request.

So most of the system’s cleverness isn’t in the writing at all. It’s in cheaply figuring out which hypotheses deserve a writer’s time before anyone writes a word.

Meet the team

Borrowing the oldest quality-control rule in publishing, we organized our team like a newsroom: one person writes the story, and someone else decides whether it runs.

And yes — an editor named Wiley plus a fact-checker named John puts John Wiley on the masthead: the two-century-old publishing house. It started as a joke and stuck, because it helps everyone remember who does what.

Wiley, our editor

Wiley’s job is protecting Nora’s time. Three models — deliberately drawn from three different model families — look at every finding and weigh in: is it real, is it worth acting on, is it already fixed? None of them answers yes or no. Each returns a subjective logic opinion: belief, doubt, and uncertainty as explicit numbers. Our Veritas component fuses the three opinions into one consensus verdict with a confidence attached. If a model times out, it counts as pure uncertainty rather than quietly disappearing from the vote. And an evaluator that keeps getting it wrong loses influence over time.

To make sense of Wiley’s numbers, you need to know one design choice: we capture findings per PR, at merge time, when the diff is small and the context is freshest — rather than batching the review up later. That tradeoff buys completeness: nothing slips through because it merged at 4:55 on a Friday. What it costs is shelf life. The code keeps moving after every capture, so a finding raised at noon is often resolved by a subsequent PR before the evening batch even runs, or superseded by a bigger change to the same page.

So over eight recent weeks, the review skills raised roughly 2,400 findings — and by the time each one reached the front of the queue, only a few hundred still needed a writer. That’s the price of eager capture, and Wiley exists to pay it in fractions of a cent instead of in a person’s afternoons. Because the human alternative was never reading 2,400 findings — it’s the sifting itself: figuring out which PRs impact user-facing docs and validating through manual testing. That triage is exactly the drudge work a docs team never has hours for, and it’s the part we automated first.

Nora, our writer

Nora isn’t one long-running agent — she’s five small jobs, each with its own trigger, sharing one rule: workflow state lives in GitHub, the platform everyone already trusts, and the findings — along with everything the team has learned about them — live in WarmHub, enriching every decision she makes.

The heart of it is a pair of daily batches. Each evening (to capture the day’s work) and early morning (to capture our overnight coding agents’ work), Nora pulls the open findings from WarmHub, bundles them by their respective doc site page — one PR fixes one page — and re-checks every finding against the current code before writing anything: is this still true, right now? Findings that were already fixed, or whose page was deleted, or that an in-flight dev PR is about to cover, get dropped here — with the reason recorded in WarmHub. For the findings that survive, Nora creates a GitHub issue referencing the findings, creates or updates the documentation, and opens a pull request.

Then, the same two skills that watch code merges — Doc Accuracy and Doc Content — run again on every PR Nora opens, this time before merge. Same reviewers, different moment: on a code merge, they’re hunting for what will need to be documented; on Nora’s PR, they’re checking what she wrote — her claims against the current source, her prose through a first-time reader’s eyes — and any finding drives a fix inside the same PR wherever possible — and anything still unresolved when the PR merges flows back into the findings queue rather than getting lost. Her PRs also get a third skill the code merges don’t need: Doc Example, which takes every piece of runnable example data her pages ship — repository names, shapes, references — and resolves it against a live WarmHub backend. Prose review can’t catch a quickstart whose example data doesn’t actually exist; only checking the real backend can. And none of this is special treatment for Nora: the same skills review every docs PR, human-written ones included. She’ll revise up to three rounds, and if still unsuccessful, her review cycle stops. After her early morning run, she’ll post a daily digest to Slack so a human can see the whole operation at a glance, including everything she chose not to do.

Beyond her daily work, she runs a weekly cleanup sweep for the slow drift no single merge causes — low-priority findings, broken links, stale pages. Her fifth job is grading what merged, and it deserves its own section — mostly because of who’s not doing the grading.

John, our fact-checker

Between “worth doing” and “shipped” sits the failure that kills trust in generated docs faster than anything else: text that reads beautifully and is completely wrong. You might know these as hallucinations. John exists for exactly this. He’s deterministic — he reads the actual source tree and cites file and line, never model memory. Every behavioral claim in a finding — the exact error code, the CLI flag, the default value, the reason string — gets checked against the code, and a finding the source contradicts is dropped, with the contradiction on record. Over those same eight weeks, John killed 83 findings that had already made it past triage — 83 pages that would have shipped reading fluently and being wrong. Then he gives Nora’s finished drafts the same treatment: every internal link has to point to a page that will actually exist in the build.

But wait, are some of these doing the same job?

If you’re keeping score, that’s three agents looking at every finding, and it can sound like redundant checking. It isn’t — because no two of those checks ask the same question, and no two happen at the same moment. Wiley judges whether a finding deserves a writer at all, and his verdict is durable knowledge — it lives in WarmHub next to the finding, where anything downstream can read it. Nora’s re-check is a gate, not a judgment: right before she writes, she re-asks whether the finding is still true against the current code — not because Wiley was wrong, but because his verdict is only as fresh as his last look, and as we all know, code keeps moving. A drop here has to show its work: the re-check must quote the actual doc text it read, or the drop doesn’t count. John doesn’t weigh worth at all; he verifies specific claims — flags, defaults, error codes — against specific lines. Same pipeline, different questions, each asked at the one moment it can actually be answered.

And Wiley’s verdicts earn their keep twice. Every finding he rejects as invalid or not worth acting on is also a data point about the review that raised it — a running, countable evaluation of the Doc Accuracy and Doc Content skills themselves. If a skill starts over-firing on some category of change, that shows up in Wiley’s reject rate long before a human would have spotted the pattern in a pile of tickets.

Nobody grades their own homework

After a fix merges, it gets judged. A separate three-model panel, again spanning model families, scores every merged fix on two axes: resolution — did it actually address the finding? — decided by majority vote, and comprehension — could a reader actually use this page? — where any single model can veto. A clean verdict closes the loop. A gap or a clarity problem opens a real follow-up PR — and only after the gap is re-confirmed against the source; an unconfirmed gap is held for a human instead. And when the panel can’t agree, nothing gets force-shipped — the case is flagged for a human in the digest.

The writer, the judge of what's worth writing, and the judge of what got written are three separate systems. Collapse any two of them, and that's where the self-serving errors hide.

The one job we keep human

Every path through this system ends the same way: a pull request that a person reviews and merges. There is no lane where generated text reaches our docs site without someone approving it. The team’s job is to make that person’s attention count — findings arrive pre-triaged, pre-verified, and pre-fact-checked, one page per PR — never to spend that attention for them. And whenever the system isn’t sure, it holds: grounding conflicts, unsettled evaluations, and pages a human is actively working on all park in a needs-attention state that a person clears.

So why keep a human on every merge, if the machines already triage, fact-check, review, and grade? The human review isn’t there to catch typos. It’s how the system earns trust in a form we can count. Every decision a reviewer makes is also a measurement. A merge is a graded yes. A PR closed without merging is wasted attention, and we count it. A revision request tells us a finding reached a person that shouldn’t have. That record, not our confidence, is what justifies delegating more.

And we’re honest about the direction: the goal is to shrink that involvement down to judgment points, not to zero. A human should be needed when the blocker genuinely requires judgment: a knowledge gap only an expert can fill, a claim the code can’t settle, a page two teams see differently. A human should not be needed because two models disagreed, or because the world changed and something needs a retry — those have mechanical answers. Everything in this post — the deterministic fact-checks, the independent grading panel, the counted states — exists to make that handoff safe to expand: as a category of fixes builds a clean record, it earns lighter review, and the attention we do spend goes where judgment was always the actual job.

The number we watch hardest isn’t in that strip: how many PRs a human closed without merging. That’s the purest measure of wasted reviewer time — work the whole pipeline failed to kill before it reached a person. Over the series it’s about one PR in thirteen, and every one of them is counted, because that count is what tells us where the triage needs to get smarter. Read it the other way, though: twelve times out of thirteen, the PR that reached a human was worth their merge — and it showed up within roughly a day of the code change that made it necessary, queued to publish when the code is released — the docs land with the change, not behind it.

The whole flow, end to end

A finding’s whole life: born when code merges, judged in WarmHub, re-checked and fact-checked at writing time — with every drop recorded — then re-reviewed on the PR, merged by a human, and graded by a panel whose bad verdicts loop back as follow-up PRs.

What if your docs don’t live in GitHub?

The shape still transfers. If your docs live in a hosted content platform instead of docs-as-code, you’d swap the PR for your platform’s own unit of reviewable change — a draft, a revision, an API write behind an approval — and give your writer agent access to the content backend. What doesn’t change is everything upstream: the findings, the triage, the fact-checks, and the grades live in WarmHub either way, and carry across whatever system you choose.

If you’re building your own

If you’re a technical writer reading this and feeling a little uneasy: the team didn’t replace our judgment, it replaced the chasing. The diffs get read, the claims get checked, and the drafts show up pre-verified — and today, a person still decides, every single time, what runs. We want that decision to be needed less often and to be easier every time it is needed. That’s not needing writers less — it’s moving our time from chasing diffs to the judgment calls that were always the actual job. Documentation is a promise you re-make with every merge. We just stopped trying to keep that promise by hand.

So now, if you’re wondering whether this post was itself a product of that flow — not quite. Nora only writes documentation, and she’d have dropped this finding as out of scope. But it was written the way the team works: a human — that’s me — drafting with an AI collaborator, every behavioral claim checked against the actual source before it shipped, revised through rounds of review — and a human deciding, at the end, that it runs.