Development journal — May 2026. This post covers the retrieval architecture exploration in depth. We've since published a tighter summary of what shipped: Accord Book's Retrieval Stack in Production: Three Lanes, No Graph on the Hot Path.

Vector search is one of the most useful primitives in modern AI systems. It is also one of the easiest to over-trust.

If your problem is “find semantically similar passages in a corpus,” embeddings can take you a long way.

If your problem is “reconstruct the current state of a messy, long-running project from many partial sources,” vector search is only the opening move.

Why production retrieval needs more than vector search

That distinction matters for CTOs because a lot of production retrieval failures are not failures of semantic similarity. They are failures of state reconstruction.

The system finds something related. It just does not find the right thing, the current thing, or the evidence needed to support the answer.

That is why serious retrieval systems end up needing more than one lane.

If you want the shorter version of the argument, see Why Hybrid Retrieval Beats Plain Vector Search. This post focuses on what changes when retrieval has to work reliably in production.

Small but important distinction: “semantically similar” and “operationally useful” overlap less than teams would like.

The production problem is not nearest-neighbor search

Teams do not usually ask retrieval systems for semantically related text. They ask for operationally useful answers.

Questions look more like:

  • what changed,
  • what is current,
  • what replaced the earlier decision,
  • what evidence supports this answer,
  • what else is connected that would not share the same wording.

Those are harder questions because project knowledge has at least four awkward properties.

1) It is scattered across formats

Relevant state may live across transcripts, uploads, notes, comments, and derived records.

2) It is relational

Two records may be tightly connected even if they use different language.

3) It is temporal

The most similar item is not always the most useful one. In fact, for project work, the best semantic match is often exactly the thing that has been superseded.

4) It needs evidence

A production answer is not just a sentence. It is usually a sentence plus support:

  • why the system believes it,
  • what source material backs it,
  • what else was considered.

That pushes retrieval beyond “top-k by cosine similarity.”

Why vector search alone plateaus

Pure vector retrieval has three common failure modes in project systems.

It finds related language, not authoritative state

A stale plan and the current plan may both be highly similar to the query. If the retriever does not know which one replaced the other, the result set can look plausible while still being operationally wrong.

It misses connected evidence with different wording

A relevant implementation artifact may not sound semantically close to the original question, even though it is linked to the same feature, entity, or project boundary.

It is weak at explanation

A vector score tells you why an embedding model liked a passage. It does not tell the user why this item matters in the larger structure of the project.

This is part of why the original Retrieval-Augmented Generation paper mattered so much: it reframed retrieval as part of the generation system rather than as a disconnected search utility.

Production retrieval architecture

flowchart LR
  Query --> Semantic[Semantic lane]
  Query --> Lexical[Lexical lane]
  Query --> Entity[Entity or structural lane]
  Semantic --> Merge[Merge candidates]
  Lexical --> Merge
  Entity --> Merge
  Merge --> Rerank[Reranking]
  Rerank --> Evidence[Evidence-rich result]

Multiple retrieval lanes are a strength, not a fallback

One of the biggest mindset shifts is that having more than one retrieval lane is not a sign the embedding model is weak. It is a sign the problem is real.

Lane 1: semantic search over canonical units

You still want a vector lane over the system’s canonical retrieval units. In project systems that often means searching over source chunks, extracted memory records, or both.

Lane 2: lexical recovery

A production system should also have a lexical path.

Why? Because reality is messy:

  • embeddings may be missing,
  • thresholds may be too strict,
  • a query may contain exact names or acronyms that should not be thrown away,
  • some high-value items are best recovered by literal overlap.

Lexical fallback is sometimes described as degraded mode. In production, it is better understood as resilience infrastructure.

Lane 3: structural expansion

Once you have a few strong candidates, the next useful step is often not “ask the vector index again.” It is “follow the structure around the candidate.”

That can mean expanding through:

  • shared entities,
  • project links,
  • relationship edges,
  • lineage between old and new state.

Lane 4: entity-centric retrieval

Some questions are better answered by starting from recognized entities than from top vector hits.

That is especially true when a query mentions a subsystem, stakeholder, or project object directly. In those cases, an entity-centric lane can pull in related context that would not necessarily win a semantic top-k race on wording alone.

If your retrieval stack is exploring richer interaction patterns, work like ColBERTv2 is a useful reminder that better retrieval quality often comes from smarter scoring structure, not just bigger indexes.

Vector-only vs production retrieval

Capability Vector-only retrieval Production retrieval
Semantic similarity Strong Strong
Exact-name recovery Weak to variable Improved via lexical lane
Current-state handling Weak Stronger with lineage/freshness signals
Structural context Limited Improved with graph/entity expansion
Explainability Minimal Better with evidence and path hints
Graceful degradation Often weak Stronger if lanes degrade independently

Reranking is where product quality really shows up

A lot of systems stop after candidate generation. Production systems cannot.

The reason is simple: the raw candidate set is not the final answer set.

Once multiple lanes contribute results, the system has to decide what should actually appear at the top. That is where reranking becomes the quality layer.

In project retrieval, useful reranking usually depends on more than similarity. It often needs to consider factors like:

  • confidence or truth weighting,
  • temporal reliability,
  • recency,
  • whether an item has been replaced,
  • whether an item is supported by structural context,
  • whether the query is asking for current state rather than historical context.

Typical ranking signals

Signal Why it matters
Similarity Gets you plausible starting points
Freshness Helps avoid stale state
Lineage Lets the system understand replacement, not just recency
Provenance Supports explanation and trust
Structural support Surfaces connected but differently-worded evidence

Current-state query rule: if the user is asking “what is current,” the retriever should not let a semantically elegant but stale artifact outrank a newer authoritative one just because the older prose is prettier.

Freshness is not a nice-to-have

In long-running systems, freshness is a first-class retrieval signal.

The hard part is that freshness is not just a timestamp sort.

The current state of a project is often represented through lineage, not just recency. A newer item may supersede an older one, but both remain relevant as evidence. A production retriever should understand that relationship instead of forcing the LLM to rediscover it from raw text.

This is where lineage-aware retrieval becomes valuable. A system that knows an older item has a successor can do things a vector-only retriever cannot do reliably:

  • down-rank stale states,
  • surface the replacement state,
  • still retain the old item as context when useful,
  • explain why the newer item is more authoritative.

Evidence matters as much as ranking

For CTO audiences, one of the most important retrieval design choices is whether the system returns only a top answer or also returns supporting evidence.

Systems that only produce a polished answer push too much burden downstream:

  • the model has to reconstruct support on its own,
  • the user has to trust a synthesis without seeing its basis,
  • debugging becomes harder when the answer is wrong.

A better pattern is to return:

  • ranked state-like items,
  • supporting evidence chunks,
  • optional path or relation hints,
  • retrieval statistics useful for observability.

That turns retrieval from a hidden dependency into a legible substrate for the answering layer.

This is one of the reasons hybrid retrieval tends to hold up better in production. It is not just better at finding things. It is better at assembling answerable context.

Retrieval systems should fail partially, not catastrophically

Another under-discussed production requirement is graceful degradation.

If one lane is unavailable, the system should still return the best result it can from the remaining lanes.

That sounds obvious, but it has architectural consequences:

  • lanes need clean boundaries,
  • merges need to tolerate empties,
  • rerankers need to score mixed-source candidates sensibly,
  • observability needs to record which lanes actually contributed.

This is operationally important because retrieval is often on the critical path for every higher-level AI behavior above it.

If the retrieval system fails hard whenever enrichment is absent, then every downstream tool becomes brittle.

That same legibility is valuable in adjacent systems too. For example, conflict detection works better when its inputs are evidence-rich and state-aware, not just semantically similar fragments.

The broader lesson: retrieval is a control surface

It is tempting to think of retrieval as infrastructure hidden underneath the product. In AI systems, it is closer to a control surface.

What retrieval chooses to surface determines:

  • what the model sees,
  • what the answer can plausibly say,
  • what evidence gets cited,
  • what state the user walks away believing.

That means retrieval quality is not just an ML metric issue. It is an architecture issue.

This is also why retrieval architecture belongs in the same strategic conversation as safe documentation generation: in both cases, the central issue is not raw model capability, but how the surrounding system curates what becomes trustworthy context.

Closing thought

Vector search is a powerful primitive. But for production project systems, it is only the first pass.

To get retrieval that people can actually rely on, you usually need a wider design:

  • semantic candidate generation,
  • lexical resilience,
  • structural expansion,
  • lineage-aware freshness,
  • reranking for current usefulness,
  • evidence returned alongside results.

In other words, not just similarity search, but retrieval architecture.