The first version of our conflict detector looked plausible on a whiteboard.

A new source record arrived. We fetched some recent memory with overlapping entities. We asked an LLM whether anything contradicted anything else. If the answer looked positive, we wrote directly to a user-facing insights feed.

It shipped fast. It also failed for exactly the reasons experienced engineers would expect: weak candidate selection, a vague problem statement, and no clean boundary between detector behavior and product delivery behavior.

This post is about the redesign.

The original sin: asking the model to do search and judgment

The old pipeline collapsed three different responsibilities into one prompt:

  1. find the potentially conflicting records
  2. interpret whether they actually conflict
  3. publish the result into a capped UI surface

That coupling made the whole system hard to reason about.

Failure mode What actually went wrong Why the old design hid it
Missed conflict candidate retrieval never found the right prior event looked like "the model missed it"
False negative correct pair was present, but adjudication failed mixed with retrieval misses
Suppressed alert delivery caps or dedupe blocked the UI event looked like no conflict existed
Weak benchmark result scoring observed insights, not detector internals detector quality and delivery policy were fused

The most important lesson was not "prompt better." It was: do not let a user-facing feed be the source of truth for a detection system.

What changed

We replaced the single-hop design with a two-stage pipeline:

flowchart LR
  SR[New source record] --> AT[Conflict atomization]
  AT --> CG[Stage 1: deterministic candidate generation]
  CG --> ADJ[Stage 2: LLM adjudication]
  ADJ --> CF[(conflict_findings)]
  CF --> DL[Delivery policy]
  DL --> INS[Insights / alerts / digest]

That architecture sounds heavier than the original version because it is. It is also much more honest.

Stage 1: deterministic candidate generation

The first stage should answer one question only:

Which prior records are plausible conflict candidates for this new event?

It should not answer whether a real conflict exists.

To make that possible, we introduced a structured claim layer: conflict_atoms.

Each atom captures fields that are useful for matching and later adjudication:

  • claim_type such as decision, constraint, request, or commitment
  • normalized subject and subject_key
  • state such as descoped, requested, approved, blocked, frozen
  • authority information
  • timestamps and validity windows
  • provenance back to the source record and memory item

That gave us something the previous system lacked: a representation shaped for the conflict problem instead of a loose pile of extracted prose.

Why atoms matter

Project conflicts are rarely pure contradictions.

A client asking for Apple Pay is not a factual contradiction. It becomes a conflict only if an earlier source explicitly descoped Apple Pay from the MVP. Likewise, a request to touch a subsystem is only problematic if that subsystem was previously frozen, assigned elsewhere, or ruled out for schedule reasons.

That means candidate generation has to look for conflict shapes, not just semantic similarity.

A few examples:

Prior atom New atom Likely shape
decision: billing = stripe_only_for_mvp request: add apple_pay_before_launch scope creep
constraint: sync_worker = frozen commitment: refactor sync worker this sprint constraint violation
assignment: sara owns migration assignment: alex owns same migration, same week resource conflict
status: vendor api unsupported decision: proceed as if supported decision drift

The actual retrieval stack is layered:

  1. hard project and time filters
  2. subject matching
  3. lexical or embedding similarity where useful
  4. hand-designed conflict heuristics
  5. recency and authority weighting

That is exactly the sort of deterministic scaffolding that LLM-heavy systems often skip too early.

Stage 2: focused adjudication

Once we have a plausible pair, then the model gets involved.

But now the prompt is narrower and better formed. The model is not asked to search the project. It is asked to adjudicate a packet that already contains:

  • the new source event
  • the prior source event
  • each side's extracted atom
  • timestamps
  • author roles
  • candidate reasons
  • relevant project context

That shift matters because the prompt becomes a classification task instead of an open-ended discovery task.

We also made the output strict and machine-readable.

{
  "has_conflict": true,
  "conflict_type": "scope_creep",
  "severity": "high",
  "confidence": 0.87,
  "is_supersession": false
}

The schema forces the model to separate several cases that are easy to blur together in natural language:

  • real conflict
  • clean clarification
  • authorized supersession
  • harmless scope addition
  • stale prior assumption

That distinction is where a lot of production trust is won or lost.

Durable findings were the real design upgrade

The highest-leverage change was not the JSON schema. It was persistence.

Every adjudication is stored as a durable conflict_finding, including negatives.

That gives the system a clean control plane:

  • detector truth lives in conflict_findings
  • delivery behavior lives in insight, digest, or notification policy

So if a high-severity item never appears in the UI, we can answer the question that previously mattered most in debugging:

Did the detector miss it, or did delivery suppress it?

Once you split those layers, benchmark design improves immediately.

Benchmarking became honest

A good evaluation pipeline should tell you where the miss happened.

After the redesign, the useful counters became things like these:

Metric Why it matters
atoms_count verifies the atomizer preserved conflict-relevant structure
candidate_pairs_count verifies Stage 1 actually generated work for Stage 2
candidate_contains_gold_pair pinpoints candidate-generation misses
llm_adjudications_count shows whether candidate pairs reached the model
findings_count detector output of record
suppressed_insights_count exposes delivery-policy side effects

That failure taxonomy is much more valuable than a single number at the insights layer.

Why this is a better product architecture

The redesign also forced us to narrow the product claim.

We do not describe the system as automatic contradiction prevention. We describe it as a conservative risk signal with provenance.

That framing fits the architecture:

  • candidate generation is high-recall and heuristic
  • adjudication is probabilistic
  • project reality changes over time
  • humans still decide whether a surfaced conflict is material

This is a healthier contract with the user than pretending the system has perfect project truth.

The engineering takeaway

There is a repeatable pattern here for LLM systems beyond conflict detection:

  1. use deterministic structure to create a reviewable candidate set
  2. use the model for bounded judgment, not open-world search
  3. persist detector output before any product-facing suppression logic
  4. benchmark the internal pipeline, not just the final UI artifact

If you skip those steps, you can still get demos. What you usually cannot get is a system you can tune with confidence.