Symptom-Understanding Retrieval: CTAS Lookup + SNOMED Knowledge Graph
Architectural Overview
The tradeoff case study 1 flagged as in-progress: a knowledge-graph grounding step for the classification pass. We built a SNOMED CT knowledge graph in Neo4j behind the same GraphContextProvider interface a hand-curated CTAS lookup table already used, evaluated both with two independently-designed eval efforts, and found that the more sophisticated retrieval backend didn't retrieve better by the metrics that measure retrieval — but that turned out not to be the thing that determined the end-to-end result.
A knowledge-graph retrieval idea was actually rejected once already: Sprint 15's original 'Graph RAG' proposal had no detailed scope beyond a one-line intent pointer and was scrapped for it. What replaced it was a validated two-track plan — ship a static, hand-curated lookup first, then justify a graph-based retrieval track separately and later, on its own evidence, not as a bet made in advance. Case study 1's tradeoff section named this explicitly: a knowledge-graph grounding step 'aimed at the classification step itself' was in active development, with the static two-pass orchestration as the layer it would plug into, not a replacement for it. This is that step, and the evaluation of whether it was worth building.
The Problem
v1's retrieval is alias/substring matching against a manually maintained table: no embeddings, no LLM extraction, just a literal string match against a curated symptom_triage_data.json. That has two structural ceilings. First, it can't generalize to how patients actually phrase things — a match only fires if the patient's words happen to contain one of the table's exact aliases. Second, the table itself caps what the system can ever recognize: every new complaint, synonym, or red flag has to be hand-added by someone who thought to add it. A structured medical vocabulary — SNOMED CT, the terminology standard Canadian health systems already use — was the candidate fix for both ceilings at once: a vastly larger, professionally maintained concept graph instead of a hand-rolled list.
v1 only recognizes a symptom if the patient's words happen to contain one of a curated list of aliases verbatim — no embeddings, no semantic step, nothing that bridges lay phrasing to clinical terms.
Every complaint, synonym, or red flag the system can ever recognize has to be manually authored first. The table cannot grow past what someone remembered to add.
The Architecture Strategy
v1 (Sprint 18) is a straightforward Symptom → RedFlag → FollowupQuestion static lookup sourced from the CTAS (Canadian Triage and Acuity Scale) reference tables, with an explicit, reviewed mapping from CTAS's 5 acuity levels down to the app's 4-level severity schema — a deliberate design decision, not an implicit rounding. It shipped behind a feature flag with zero behavior change to existing users. v2 (Sprint 19) keeps the same GraphContextProvider interface and the same hand-curated Symptom → RedFlag → FollowupQuestion content, but re-anchors that content onto a small curated set of SNOMED CT concepts instead of hand-listed alias strings, backed by a Neo4j graph seeded from the SNOMED CT Canadian Edition (RF2 release). Severity classification stays the LLM's job unconditionally in both versions — the graph only ever supplies context, never a verdict. Getting there required a real SNOMED CT Affiliate License (submitted to SNOMED International's MLDS, approved within 3 days) plus a separate Canada Health Infoway acquisition for the Canadian Edition specifically. And the build hit a real ceiling of its own: Neo4j AuraDB's free-tier 200,000-node cap, discovered empirically mid-load at roughly 14% of the originally planned single-root subtree — not documented anywhere in advance. The fix was a redesign to a multi-root, bounded-depth (4 levels) subset keyed off the 165 CTAS complaints, landing at 31,327 concepts, 126,816 descriptions, and 50,351 IS_A relationships — 79% of the free-tier cap, with all 154 curated anchor concepts backed by loaded data, up from 69 before the redesign.
def get_graph_provider() -> GraphContextProvider:
"""v1 and v2 sit behind the identical interface. Switching is an
env var, not a code change on either side, and each provider is
constructed once and cached, not rebuilt per request."""
provider_name = os.environ.get("GRAPH_RAG_PROVIDER", "off").lower()
if provider_name not in _provider_cache:
_provider_cache[provider_name] = _build_provider(provider_name)
return _provider_cache[provider_name]Alternatives Considered
The lowest-effort option: keep hand-adding aliases as gaps get noticed. Rejected as a strategy, not just a tactic — it doesn't fix the structural ceiling, it just raises it slowly, and every addition is unmaintained-vocabulary debt someone has to keep paying down.
Considered and not built this round. The v2 design explicitly targeted a graph traversal with precedence rules over a curated concept set, not a vector index — a scope decision this case study's own eval later surfaces as consequential: see case study 'Fair Retrieval Evaluation' for what that specific choice cost.
Lessons Learned
Retrieval-quality eval (Track A/B, full depth in case study 'Fair Retrieval Evaluation') and end-to-end triage-accuracy eval (a 27-vignette multi-turn ablation) were built by different work, at different times, measuring different things. Reading either one alone would have been misleading — the pairing is the actual finding, not a footnote.
Sprint 19's original scope named a deliverable — a short write-up of why both retrieval paths exist and how the switch works — that never shipped as its own file. This case study substantially fulfills that intent; better to say so directly than leave the gap silently unresolved.
The production default for GRAPH_RAG_PROVIDER has not been independently confirmed against the live deployed config as of this writing — the factory code defaults to off when the env var is unset, but that's a code-level default, not a verified statement about what's actually running. Given the results below, that's worth checking rather than assuming, not a footnote. Separately, and by design: full PART_OF/cross-symptom-cluster authoring beyond one seeded pilot cluster, and expanding the entity-linking precision test suite past its current 3 pilot anchors, are both explicitly deferred to a future symptom-understanding-improvement sprint — not fixed here, and not silently dropped either.
- End-to-end triage accuracy: neo4j (v2) 66.7% (18/27), off (no graph) 63.0% (17/27), static (v1) 51.9% (14/27) — v1 is the worst-performing leg of all three, including worse than no augmentation at all.
- v1's under-triage rate (37.0%, 10/27) is the worst of the three legs and concentrated in the highest-stakes bucket: 7 of 16 emergent-severity vignettes under-triaged, versus 4 of 16 for both other legs.
- Isolated retrieval quality: v1 scores 100% (20/20) hit-rate on its calibrated scenario set, v2 scores 35% (7/20) — full methodology, the vocabulary-neutral control that overturns the naive reading of this number, and the complete Track A/B tables live in case study 'Fair Retrieval Evaluation: Static Lookup vs. SNOMED Knowledge Graph'.
- Task 13: a purpose-built, Clean-Architecture eval harness driving 27 real Ontario CTAS vignettes through the actual multi-turn conversation agent, three legs (static / off / neo4j), scored on confusion-matrix accuracy, under-triage rate, and elicitation coverage — because the only existing vignette source is third-person exam narrative, not a chat conversation, and elicitation coverage specifically requires information that's disclosed only when asked.
- Track A/B: isolated single-turn retrieval-quality eval, run identically against both providers — see the dedicated case study for the full design, including a vocabulary-neutral control set and a stale-number correction caught mid-eval.