Development journal — May 2026. This post describes our benchmark methodology and the pricing questions we were working through. For where we landed, read Graph Off the Hot Path: What the Benchmarks Showed and Where Graph Now Fits.
If a graph-backed retrieval system is expensive to run, the first question is not "is graph cool?"
It is: does graph change the product enough to justify what it costs?
That is not just an architecture question. It is a pricing question.
For us, the concrete decision was whether a cheaper vector-only tier could stand on its own, or whether Neo4j had to remain in the base product because the quality gap was too large.
The setup
Our retrieval stack has three broad parts:
flowchart TD
Accord Book[User query] --> E[Embed query]
E --> VC[Vector search over chunks]
E --> VM[Vector search over memory items]
VM --> GX[Optional graph expansion]
VC --> RR[Merge and rerank]
GX --> RR
RR --> OUT[Ranked memory + evidence]
The graph lane is useful, but it is not free:
- operationally heavier than Postgres
- more expensive to host
- another failure domain to observe
- another system to tune and back up
That means the right benchmark is not "can graph help?" It is "how much product quality do we lose if we turn it off?"
The benchmark design
We use the same retrieval flow against the same dataset, then toggle only the graph expansion behavior:
- V0: vector-only control, with
k_graph_expansion=0 - V1: hybrid baseline, with graph expansion enabled
That is a small but important design choice. We are not comparing two different products. We are comparing two variants of the same retrieval system under the same harness.
Why we use LongMemEval
The benchmark runner operates against the LongMemEval dataset, which is good at stressing memory reconstruction across sessions, updates, and temporal context.
That makes it a reasonable fit for the retrieval side of the product, especially on questions where freshness and cross-session reconstruction matter.
It is also important to say what it does not measure:
- proactive conflict surfacing
- false-positive rates for alerts
- domain-specific architecture-decision conflicts
Those need a separate evaluation track.
The metrics that actually matter
Raw similarity is not enough. Retrieval quality is downstream quality.
These are the metrics we care about most:
| Metric | Why it matters |
|---|---|
session_recall@10 |
whether the correct historical session is even visible in the top results |
session_recall@30 |
broader recall when the first page is too strict |
llm_correct |
whether a judge model considers the retrieved context sufficient for the answer |
llm_evidence_sufficient |
whether the result set actually supports the answer, including abstention cases |
graph_contribution_pct |
how much of the useful final set came from graph expansion |
top_has_superseded / supersedes_signals_used |
whether the system is respecting freshness and update lineage |
Notice what is missing from that list: blind faith in string_match.
String matching is still logged, but it is not the quality metric of record. In memory-heavy systems, paraphrase and abstention behavior matter too much for literal substring checks to carry the decision.
Why LLM-as-a-judge is worth it here
This is one of the rare cases where LLM-as-a-judge is not hand-wavy evaluation theater.
It solves a real scoring problem.
If the system retrieves the right evidence but phrases the answer differently from the gold text, naive matching undercounts success. If the correct behavior is to abstain because the evidence is absent, naive matching often over-penalizes or misclassifies the result.
A competent judge model is much better at answering the question we actually care about:
Given this evidence set, was the system materially correct and properly grounded?
That makes llm_correct and llm_evidence_sufficient more useful for product decisions than raw text overlap.
The acceptance criteria connect directly to pricing
We use explicit thresholds rather than vague interpretation.
| Metric | V0 must be within | Why |
|---|---|---|
overall llm_correct |
5 percentage points of V1 | broad retrieval quality bar |
overall session_recall@10 |
10 percentage points of V1 | visible top-result quality |
llm_correct on knowledge-update |
8 percentage points of V1 | most relevant slice for freshness-sensitive behavior |
string_match |
15 percentage points of V1 | loose directional check only |
Those thresholds translate the benchmark into a business decision.
If V0 clears them, a vector-only base tier is defensible.
If it fails, we have three honest options:
- ship a stronger Postgres-only mitigation
- keep graph in the base tier and price accordingly
- narrow the base-tier promise so it is not sold as having the same memory moat
That is what a benchmark should do: reduce strategy arguments to explicit tradeoffs.
The runner structure
The harness itself matters because retrieval quality depends on more than one moment in the pipeline.
flowchart LR
D[Dataset entry] --> I[Ingest haystack sessions]
I --> PRE[Run retrieval PRE-curation]
PRE --> C1[Entity resolution]
C1 --> C2[Relation discovery]
C2 --> C3[Conflict discovery]
C3 --> C4[Cluster synthesis]
C4 --> POST[Run retrieval POST-curation]
POST --> R[Emit matrix rows and summaries]
Each run emits per-question, per-variant, per-stage artifacts. That lets us inspect not just "which variant won," but also where the pipeline helped or degraded.
This is especially useful when debugging misses like:
- seed miss
- edge miss
- ranking miss
- graph data missing
- timeout
Without that taxonomy, retrieval work tends to devolve into parameter superstition.
The graph question is really a freshness question
The most interesting slice is usually not generic semantic retrieval. It is knowledge-update behavior.
That is where graph expansion and supersedes-aware ranking have a chance to justify themselves:
- a stale memory may be semantically similar but wrong
- its successor may be phrased differently
- the right answer may require following lineage, not just nearest-neighbor similarity
If graph meaningfully improves those cases, it may deserve its cost. If it does not, carrying the operational burden is harder to justify.
What good benchmark reporting looks like
A useful report should answer these questions directly:
- Did V0 pass the shipping threshold?
- On which question types did it fail?
- How much of V1's win came from graph contribution versus better reranking?
- Did supersedes-aware signals materially improve freshness-sensitive answers?
- Was the latency cost acceptable?
That is much better than a generic "hybrid performed better" conclusion.
The deeper lesson
A graph database in retrieval architecture should not survive on intuition alone.
If it stays in the product, it should stay because it wins on the benchmark slices that matter to users and to pricing. If it leaves, it should leave because the measured gap was small enough that the simpler system was the better product trade.
That is the discipline we want more of in AI systems: not just building the advanced path, but proving when it earns the right to exist.