Three system deep-dives. Each one follows the same structure: what broke, how we approached it, what we traded away, and what we'll measure.
LLM Symptom Understanding
7 MIN READ
#AI-Agents#LLMTools#Groq#Evals
Two-Pass Tool Orchestration for Symptom Triage
Splitting LLM symptom triage into two forced passes: a tool-only severity classification, then a deterministic facility lookup, then a grounded response, so the model can never invent a facility name or commit to a severity before it has enough information — then proving it holds with two independent evaluation tracks: an online deterministic check against simulated live traffic, and an offline LLM-as-judge pass over a purpose-built dataset.
Filtering facilities by severity eligibility before ranking by distance, using a plain Haversine calculation over an in-process cache, fast enough for the inline triage path, with the ranked candidate list already shaped for a future travel-time upgrade.
Two-Tier Facility State: In-Process Cache + Redis Wait Times
Splitting facility state into two tiers that match their actual freshness and failure requirements: an in-process ETag cache for the rarely-changing facility directory, and a Redis cache-aside chain with a Supabase fallback for wait times that change every scrape cycle.
Event-Driven Fan-Out: S3 to Lambda to EventBridge to dbt
Decoupling a multi-source ingestion pipeline into independent Lambda stages using native S3 and EventBridge event triggers instead of a Step Functions orchestrator or a polling coordinator — so a failure in one processor never blocks the others, and a completion signal from any of three independent sources fans into one shared transform-integrity-test-live-health checkpoint.
An LLM's output can't be verified by reading the code that calls it, so Pass 2's generator output is checked by two independent tracks, each catching a different class of unfaithfulness: a zero-cost deterministic substring check running on every live response, and an offline LLM-as-judge pass scoring subtler fabrications a string match can't see.
The tradeoff case study 1 flagged as in-progress: a knowledge-graph grounding step for the classification pass. We built a SNOMED CT knowledge graph in Neo4j behind the same GraphContextProvider interface a hand-curated CTAS lookup table already used, evaluated both with two independently-designed eval efforts, and found that the more sophisticated retrieval backend didn't retrieve better by the metrics that measure retrieval — but that turned out not to be the thing that determined the end-to-end result.
Fair Retrieval Evaluation: Static Lookup vs. SNOMED Knowledge Graph
A retrieval comparison that looks fair on paper — same scenarios, same scoring, both providers — can still hide a confound. Building one for MediCoord's two symptom-understanding retrieval backends took a shared ranking fix, a split hit-rate/recall metric to tell coverage failures from ranking failures apart, a vocabulary-neutral control scenario set, and a caught-and-corrected stale number. The eval design process is as much the content here as the final numbers.
