Fair Retrieval Evaluation: Static Lookup vs. SNOMED Knowledge Graph

The first retrieval comparison run had a real, if disclosed, bias baked in: its scenario messages were written using v1's own alias vocabulary, so a system that matches on that exact vocabulary was always going to look artificially strong. Disclosing a bias in code comments isn't the same as controlling for it — nothing in the original eval separated 'v1 retrieves well' from 'v1 retrieves well on text calibrated to v1.' A second problem sat underneath the architecture docs: v1 (flat alias table) and v2 (SNOMED knowledge graph) look structurally distinct, but reading their actual query code side by side shows both retrieve via the same operation — case-insensitive substring containment — just against different vocabularies. Nothing in the design documentation surfaces that; only the code does.

Before any comparison could be called fair, both providers needed a shared prerequisite fix: both originally used first-match-wins retrieval, so a longer, more specific match sitting later in the data could lose to an earlier, less-specific one purely by insertion order. Both were rebuilt to rank by match specificity instead. Only after that fix landed did the actual eval run: Track A is a deterministic retrieval hit-rate check, 20 isolated single-turn scenarios, identical scoring against both providers. Track B runs DeepEval's faithfulness, contextual precision, and contextual recall metrics (same judge model and methodology as case study 'Two-Track LLM Evaluation' — see that post for how those metrics work) against transcripts generated by replaying the same in-process agent used for the end-to-end eval, so no retrieval-context construction logic is duplicated between the two eval efforts. The change that actually mattered most: a second scenario set, LAY_SCENARIOS, was built specifically to avoid every one of v1's alias/name substrings — enforced programmatically against the real alias table, not by hand-checking — but it sat built and unexecuted for two weeks before this eval redesign finally ran it.

def score_hit(ctx, expected_complaint):
    """Selection accuracy: given the candidates, did the FINAL choice match?"""
    if expected_complaint is None:
        return not ctx.matched
    return ctx.matched and ctx.complaint_name == expected_complaint


def score_recall(candidates, expected_complaint):
    """Coverage: was the right concept ANYWHERE in the candidate set,
    regardless of which one got selected? Separates 'never found it'
    from 'found it, picked wrong' — collapsing these into one number
    is exactly what a naive hit-rate metric does."""
    if candidates is None:
        return None
    if expected_complaint is None:
        return len(candidates) == 0
    return expected_complaint in candidates