Development journal — May 2026. This post walks through how we designed the pipeline, including the approaches we tried and ruled out. For a cleaner summary of what shipped, see Accord Book's Docs Pipeline: arc42, ADRs, and PR-as-Publication.

Most teams do not have an architecture documentation problem. They have an architecture documentation drift problem.

The code changes. The diagrams do not. The ADRs stop halfway through the migration. The README still describes the system you had two quarters ago. Then someone decides AI will fix this by pointing a model at the repository and asking it to “summarize the architecture.”

That works right up until the moment the system starts acting like a build agent instead of a documentation agent.

How to generate docs from code without turning AI into a build agent

For CTOs, that boundary matters. There is a big difference between an AI system that can read a repository safely and one that expects to clone it, install dependencies, execute arbitrary code, run tests, infer behavior from runtime side effects, and then publish its own conclusions as truth.

The more interesting technical question is this:

How do you generate architecture docs from a live codebase without turning the documentation pipeline into a privileged execution surface?

That is the design problem we have been working through in Accord Book’s documentation subsystem. The answer is not “less automation.” It is better boundaries.

If you want the shorter governance version of this argument, see AI Proposes, Humans Publish. This post goes deeper into the technical architecture behind that boundary.

Practical rule: if your documentation pipeline needs npm install, a pile of secrets, and a lucky moon phase before it can explain your architecture, you may have built a CI job wearing fake glasses.

The wrong mental model: treat documentation as a build artifact

A lot of AI-for-docs systems quietly inherit a CI mindset:

  1. clone the repository,
  2. install the project,
  3. run whatever is needed to understand it,
  4. ask an LLM to summarize the result,
  5. write the docs back.

That sounds practical, but it bundles together several different risk classes:

  • security risk: arbitrary customer code execution,
  • operational risk: slow, flaky, environment-dependent runs,
  • analysis ambiguity: generated output depends on runtime state and setup quality,
  • trust risk: draft model output is easily mistaken for approved architecture truth.

For brownfield systems, this is especially dangerous. Mature repositories often have partial setup docs, stale scripts, hidden assumptions, and environment-specific behavior. If your documentation pipeline depends on successful execution, then your documentation quality becomes hostage to the repo’s operability.

That is backwards.

A documentation system should be able to extract useful structure from an imperfect codebase without requiring the codebase to be runnable in the first place.

A safer design: read-only analysis, bounded evidence, explicit publication

The design we prefer has three distinct phases:

  1. safe repository analysis
  2. bounded synthesis
  3. human-gated publication

Those phases sound procedural, but they are really trust boundaries.

flowchart LR
  Repo[Repository snapshot] --> Index[Bounded indexing]
  Index --> Evidence[Evidence pack]
  Evidence --> Draft[LLM draft synthesis]
  Draft --> PR[Reviewable PR]
  PR --> Merge[Human merge]
  Merge --> Truth[Published truth]

1) Safe repository analysis

Start from the assumption that the repository is untrusted execution input.

That means the analysis side should be optimized for:

  • read-only access,
  • deterministic bounds,
  • reproducible snapshots,
  • zero dependency on package install or successful runtime startup.

In practice, that leads to a few architectural choices.

Use snapshotting, not live working directories

If documentation is generated from whatever happens to be in a mutable working copy, you have two problems immediately:

  • the result is hard to reproduce,
  • the model is reading a state that may never actually ship.

A better approach is to analyze a fixed repository snapshot at a known commit. That turns “what did the model look at?” from a mystery into a concrete answer.

For engineering leadership, this is more important than it sounds. A useful architecture document is not just a summary. It is a statement about a specific version of the system.

Treat the repository like data, not like software

The moment your documentation pipeline starts installing dependencies or invoking project scripts, you have crossed into an entirely different threat model.

A safer pattern is:

  • mirror the repository,
  • materialize a detached snapshot,
  • walk the tree read-only,
  • classify files heuristically,
  • build bounded evidence packs from selected excerpts.

That gets you a surprisingly large share of the architectural signal:

  • service boundaries,
  • data model shape,
  • module wiring,
  • deployment surfaces,
  • integration points,
  • asynchronous processing patterns,
  • documentation already present in the repo.

And crucially, it does so without requiring the system to execute.

Bound the indexing step aggressively

This is where a lot of AI docs systems fail quietly. They claim to analyze the repository, but what they really do is dump a huge and poorly curated context into a model and hope the model figures out what matters.

The better pattern is to constrain the analysis front end before the LLM sees anything.

That means capping things like:

  • recursion depth,
  • number of files,
  • bytes per file,
  • total bytes read,
  • allowed file types,
  • excluded directories such as build artifacts and dependency trees.

Then add lightweight categorization so the system can distinguish likely signals from noise:

  • manifests,
  • wiring and entry points,
  • API surfaces,
  • data model files,
  • async worker code,
  • deployment config,
  • tests,
  • existing docs.

This does two things at once:

  1. it lowers cost and latency,
  2. it improves synthesis quality because the model receives architecture-shaped evidence instead of raw repository entropy.

That is a subtle but important CTO-level point: the quality win comes as much from pre-LLM information architecture as from model capability.

Comparison: build-agent docs vs read-only docs

Question Build-agent approach Read-only documentation approach
What does it assume? The repo can be installed and executed The repo can be inspected as data
Failure mode Environment/setup flakiness blocks docs Analysis may be partial, but still usable
Security posture Higher execution risk Lower execution risk
Reproducibility Depends on runtime environment Anchored to a commit snapshot
Publication path Temptation to auto-write Natural fit for review and merge

Documentation generation is not one prompt

Once the analysis step is bounded, the next temptation is to ask the model for “the architecture document.”

That is usually the wrong abstraction.

For non-trivial systems, a better pattern is staged composition:

  • build an evidence pack,
  • generate section-level outputs,
  • require lightweight source attribution,
  • keep each synthesis step narrow enough that failures are diagnosable.

This matters because architecture docs are not just long-form prose. They are a structured set of claims about:

  • purpose,
  • constraints,
  • boundaries,
  • runtime flows,
  • deployment shape,
  • cross-cutting concepts,
  • decisions,
  • risks.

If the model gets one of those wrong, you want to know which section failed and what evidence it saw, not just that “the generated architecture doc feels off.”

In Accord Book, the more robust pattern has been to synthesize documentation in explicit document shapes rather than a monolithic blob. That keeps the generation pipeline more inspectable and makes targeted review possible.

If your team uses established architecture documentation conventions such as arc42 or short-form Architecture Decision Records (ADRs), the payoff is even bigger: the model can aim for known document shapes instead of inventing a structure every time.

What the pipeline actually looks like in practice

Here is what Accord Book's documentation generator actually does. It produces 10 arc42 sections and an ADR set. Each section is synthesised in two phases, then run through a deterministic canonicalisation step before a final structural guard decides whether to open the PR.

The seven stages, mapped to the code that owns them:

Stage Responsibility
Index Walk the repo worktree, classify every file by architectural category
Evidence pack Per section, filter indexed files by category and produce bounded ## FILE: evidence
Draft One LLM call per section, given its filtered evidence pack and a section-specific prompt
Merge Reconcile a code-authoritative draft and a docs-authoritative draft into one merged section
Canonicalise Post-LLM deterministic pass: rewrite the ## Sources block to real paths, drop unresolvable citations
Final guard Structural defects block the PR; path warnings log only
PR Open the GitHub pull request

The flow in full:

flowchart TD
  Repo[Repository snapshot] --> Index[File index + classify]
  Index --> Pack[Section evidence pack<br/>bounded FILE: excerpts]
  Pack --> Draft[LLM draft<br/>one call per section]
  Draft --> Merge[LLM merge<br/>code-draft + docs-draft reconciliation]
  Merge --> Canon[Deterministic canonicaliser<br/>exact match → suffix → basename]
  Canon --> Guard{Final guard}
  Guard -->|Structural defect| Fail[Job failed<br/>no PR]
  Guard -->|Path warnings only| PR[GitHub PR opened]
  PR --> Review[Human review + merge]
  Review --> Truth[Published .q_context/arc42/]

The boundary between the LLM stages and the deterministic stage is the design decision this whole section is about.

Why the evidence pack is the real quality lever

Each arc42 section gets a different slice of the repo. Section 07 (runtime infrastructure) sees deploy/, Caddyfile, docker-compose and systemd manifests. Section 05 (solution strategy) sees API controllers, module wiring, and provider bindings. Section 03 (system scope) sees the manifest, README, and entry points.

Giving the LLM a different, purpose-built evidence pack per section is strictly better than giving it the entire repository.

The quality gain is not from limiting context — it is from shaping context. An LLM asked to describe “the runtime infrastructure” produces a better answer when its evidence pack contains only infrastructure-shaped files than when it must first locate infrastructure files inside a repository-wide dump. The categorisation work you do before the LLM call pays dividends in every call after it.

The validation trap: what we tried and what worked

This is the part of the story most teams will not tell you, because it requires admitting seven iterations.

The underlying problem: the LLM cites file paths in a ## Sources block at the end of each section. Some of those paths do not exist. The LLM hallucinates a src/modules/ prefix, drops a directory, or invents a leaf. If fabricated paths land in the published .q_context/arc42/ files, downstream agents trust them and produce wrong answers. So real paths are a hard requirement.

The instinctive solution was to validate them. The instinctive solution was wrong.

v4 — the baseline worth protecting

Single LLM call per section, no path validation. Content was good: substantive, evidence-grounded, well-shaped sections. Path hallucinations happened occasionally and silently. This is the version every later iteration was measured against.

The key word is measured against. Before adding any validation, we had a content quality baseline. That measurement became essential for diagnosing what each subsequent change actually broke.

v5 — strict validation + QA critic + targeted repair

Three changes in one go:

  1. Deterministic source-path validation inside the draft and merge loops. If the LLM cited a path not in the section's evidence pack, the section failed, and the LLM was given a retry with an error message listing the allowed paths.
  2. An optional whole-document QA critic — a second LLM pass over the merged output, flagging contradictions between sections.
  3. Targeted repair — re-running any section the critic flagged.

Result: sections 02 and 10 improved. Sections 03, 06, 09, and 11 regressed — in different ways across runs. Quality oscillated because the critic's flags were not idempotent and repairs sometimes broke content the critic had not flagged.

Lesson 1 — An LLM grading another LLM's whole-document output is not a stable feedback loop. The critic introduces variance proportional to its own creativity. If you cannot express the failure as a deterministic predicate over the output, do not wire a critic into a repair pipeline.

v5.1 — rollback of QA critic, path validation kept

Dropped the critic and the repair loop. Kept deterministic path validation.

Still bad. Section 07 (runtime infrastructure) was missing Caddy and ngrok entirely, even though deploy/proxy/Caddyfile was in the repository. The path validation had nothing to do with this; something upstream was wrong.

v6 — diagnose the indexing gap

Two separate problems had been conflated:

  1. The file indexer did not classify Caddyfile, nginx.conf, traefik.yml, ngrok.yml, or deploy/proxy/* paths as deployment. They fell through to unknown and were excluded from section 07's evidence pack entirely.
  2. Section 07's prompt did not say “if you see proxy or tunnel evidence, describe it.”

Fixing the indexer to recognise common infrastructure file conventions (Caddyfile, nginx, Traefik, ngrok) is the kind of change that is generic and defensible. Any customer repository might use those tools. The fix was not Accord Book-specific; it was a gap in the classifier's understanding of common deployment conventions.

Result: section 07 finally had Caddy and ngrok. Sections 03, 09, and 11 were gutted — short, stripped, most content missing.

v6 post-mortem: path validation was strangling content

The smoking gun was in the retry logic. Draft-phase path validation passed only the section's own evidence pack as the allowed-paths set. When the LLM legitimately wanted to cite a file from another section's evidence — section 03 referencing a controller that lived in section 05's pack — validation failed. After three primary retries and three fallback retries, the LLM produced the shortest possible section that passed the check.

Lesson 2 — LLMs respond to repeated rejection by truncating, not by correcting. If your retry message says “you cited X which is invalid, try again” and X is genuinely good content, the model produces a shorter document on each retry rather than a more correct one. This is a content-quality death spiral.

The retry loop that was supposed to improve path correctness was silently destroying everything else.

v7 — decouple path correctness from the generation loop entirely

Three changes, in the opposite direction from every previous attempt:

1. Remove path constraints from the LLM retry loop. The LLM is no longer penalised for citing real-but-out-of-pack paths. The section prompt explicitly says: “you may cite any repository path; a deterministic post-processing step will canonicalise.”

2. Add a deterministic canonicaliser. After each merged section is produced, a post-processor walks the ## Sources block and attempts three resolution tiers against the full repo index:

Tier 1 — Exact match
  cited path exists verbatim in the index → keep as-is

Tier 2 — Longest unique suffix
  the tail of the cited path uniquely identifies a real path
  e.g. “src/modules/mcp/mcp.controller.ts”
       → “services/api/src/mcp/mcp.controller.ts”

Tier 3 — Unique basename
  the leaf filename is unique across the entire repo
  last resort when the directory is completely wrong

Unresolvable citations are dropped from the Sources block. All resolutions are logged.

3. Demote path mismatches from errors to warnings in the final guard. Structural defects (missing H1, section too short, deny-listed terms) still fail the job and block the PR. Path issues that survive canonicalisation produce a WARN log line and the PR is created anyway.

Result: best version across all seven iterations — 7 of 10 sections rated GOOD, zero BROKEN, zero GUTTED. The infrastructure fixes from v6 held. Sections 03, 09, and 11 returned to their v4 depth. Section 08's long-missing schedule_executions concept appeared for the first time.

Lesson 3 — Deterministic post-processing is strictly better than LLM self-correction for syntactic constraints. Path correctness is syntactic. Run a string-matching pass, not a retry loop.

Lesson 4 — For LLM-generated documentation, “warnings, not errors” is the right failure mode for most quality dimensions. The cost of a missing citation in a doc is low; the cost of failing the whole job and leaving stale docs in production is high.

Section quality across the iteration arc

Section v4 v5 v6 v7
§02 System context GOOD GOOD GOOD GOOD
§03 System scope GOOD REGRESSED GUTTED GOOD
§05 Solution strategy GOOD GOOD GOOD GOOD
§06 Async processing GOOD REGRESSED GOOD GOOD
§07 Runtime infrastructure MISSING CADDY MISSING CADDY GOOD GOOD
§08 Concepts GOOD GOOD GOOD GOOD+
§09 ADR overview GOOD REGRESSED GUTTED GOOD
§10 Quality requirements GOOD GOOD GOOD GOOD
§11 Risks GOOD REGRESSED GUTTED GOOD

The v5 and v6 regressions were entirely caused by the validation machinery. v7 removed that machinery and the quality returned to baseline, with §07 and §08 as genuine improvements from the indexer fix.

What is validated, and where

The current pipeline has four distinct validation phases. The key design principle is the last column: the LLM only retries on shapes it can plausibly fix in one more attempt. Path correctness cannot be fixed by retrying — the LLM does not know the repository any better on attempt two than on attempt one.

Phase What is checked Failure mode
Draft (per section) H1 matches title, minimum line count, ## Sources present, parse kind acceptable Retry the LLM (3 primary + 3 fallback)
Merge (per section) Same as draft + section-specific content invariants Retry the LLM
Canonicalise (per section, deterministic) Path citations against full repo index Rewrite or drop, log only
Final guard (whole document) Same as merge Structural → block PR; paths → warn + proceed

This is not the validation pipeline that feels natural to build first. It is the one that works.

The heuristics that survived seven iterations

These rules came from the iteration arc above — each one was learned, not guessed:

  1. The LLM is the unstable component. Treat its output like untrusted input from a system boundary. Validate, canonicalise, log warnings — do not try to make the LLM be careful.
  2. Deterministic beats LLM for any syntactic check. If you can write a regex or a string match for it, do that and skip the retry loop.
  3. Wide evidence plus tight prompt beats narrow evidence plus loose prompt. The LLM produces better content when it can see more of the codebase; the prompt is where you enforce scope and structure.
  4. Retry loops that fail on hard constraints destroy content. The LLM truncates rather than corrects. Retry on shape failures (parse error, missing section header) — not on content constraints.
  5. Build-failing checks should reject structural defects only. Path mismatches, citation gaps, and “this section could be deeper” are warnings. Missing H1, contradictions in the evidence, and parse failures are errors.

The counterintuitive summary: we tried to make the LLM care about path correctness. That made the docs worse. Then we stopped asking the LLM and fixed the paths ourselves. The docs got better on correctness and on content depth simultaneously, because the previous validation had been punishing good citations along with bad ones.

Brownfield reality: code and docs disagree

This is where architecture documentation stops being a summarization problem and becomes a reconciliation problem.

In real systems, there are usually at least three partially true stories available at once:

  1. what the code currently does,
  2. what the existing docs say it does,
  3. what the team believes it does.

A useful documentation generator needs to handle disagreement without pretending it can resolve every disagreement automatically.

Important: if the code and docs disagree, the most valuable output is often not a confident rewrite. It is a crisp note saying, “these two things diverge; a human should decide which one becomes truth.”

That kind of restraint is not a product weakness. It is what makes the system operationally credible.

Keep draft truth separate from published truth

The governance version of this argument is simple: AI should propose, humans should publish.

The technical version is more interesting.

If generated documentation is immediately available to downstream agents as authoritative context, then your synthesis pipeline is no longer a drafting tool. It is a control-plane mutation path.

That means a documentation bug can become an execution bug one step later.

The clean solution is to separate:

  • job state,
  • draft outputs,
  • published truth.

For a Git-native workflow, that typically means:

  • generation jobs track operational metadata,
  • AI writes proposed documentation changes to a branch,
  • a pull request becomes the review surface,
  • merge to the default branch becomes the publication event,
  • downstream tooling reads only the merged version.

This is also closely aligned with the broader concerns in the NIST AI RMF: Generative AI Profile, which is another reason it is worth treating publication as a separate engineering concern rather than a UI affordance.

The deeper architectural lesson: treat documentation as a derived system

There is a common trap in AI product design where generated docs are treated as an isolated feature.

In practice, they are better understood as a derived layer over canonical evidence.

That means the documentation subsystem should inherit good systems design habits from the rest of the platform:

  • canonical stores first,
  • derived artifacts second,
  • bounded transforms,
  • explicit lineage,
  • human-gated promotion for anything that becomes shared truth.

This is one reason the most useful documentation systems will not look like content tools. They will look like careful data pipelines.

That shift in perspective matters for CTOs because it changes what you optimize for.

If you think you are building a writing assistant, you focus on fluent prose.

If you think you are building a truth-adjacent synthesis layer, you focus on:

  • provenance,
  • boundedness,
  • reproducibility,
  • safe repository handling,
  • publication control,
  • compatibility with existing engineering workflows.

Those are very different product decisions.

Why this matters beyond documentation

The reason this topic is bigger than docs is that the same boundary problem is showing up all over AI systems.

Whenever AI is allowed to read operational state, summarize it, and feed it back into another system, you need to ask:

  • what is canonical,
  • what is derived,
  • what is draft,
  • what is published,
  • what is executable,
  • what is reviewable.

Documentation generation just makes the problem easy to see.

If you solve it well here, you usually end up with better instincts for adjacent systems too, including conflict detection and retrieval architecture.

Closing thought

The goal is not to make AI documentation less powerful. It is to make it powerful without quietly granting it the powers of a build system, a deployment system, and a source-of-truth publisher all at once.

The safest and most useful pattern we have found is straightforward:

  • analyze code read-only,
  • keep evidence bounded,
  • synthesize docs as proposals,
  • publish only through an explicit human gate.

That gives you a system that is technical enough to be useful, constrained enough to be safe, and legible enough to earn trust.