Two-Track LLM Evaluation: Groundedness + Faithfulness

Two different failure classes need two different detection strategies for that generator-only faithfulness question. A wrong facility name is a simple, exact substitution — cheap to catch deterministically. A wrong address, or a rehab centre mischaracterized as an ER, is subtler unfaithfulness a substring match can't see, but running a judge-LLM call on every live production request adds a third round-trip's cost and latency to every triage response.

Two tracks, matched to what each failure class actually needs — not a generic RAGAS-style metric suite applied uniformly, but a cheap deterministic check for exact-fact grounding and an LLM-as-judge pass (DeepEval's FaithfulnessMetric, the generator-side metric RAGAS's vocabulary would call faithfulness) for the subtler cases only a semantic judge can catch. Track A (online, deterministic, zero marginal cost): check_facility_groundedness() runs on every Pass-2 response — exact facility-name substring match against the deterministic lookup result, no LLM involved. Track B (offline, LLM-as-judge): 100 requests replayed through the real /chat endpoint and saved as a purpose-built transcript dataset; 89 that resolved to a facility recommendation scored offline with DeepEval's FaithfulnessMetric (gpt-4o-mini judge — chosen for cost, bounded by a stated $5-credit constraint noted in the Sprint 17 discussion notes, not claimed as the best available judge) at a 0.7 pass threshold, against build_retrieval_context()'s facts-only restatement.

def build_retrieval_context(facility: dict) -> list[str]:
    """
    Pure factual restatement — name, address, distance only.
    Deliberately NOT the full grounding-message text (which also
    contains instructions like "use this exact name"): DeepEval's
    FaithfulnessMetric extracts "truths" from retrieval_context via
    its own LLM call, and instructional text would pollute that
    extraction.
    """
    return [
        f"{facility['name']} is located at {facility['address']}, "
        f"{facility['distanceKm']} km away."
    ]


def score_transcript(transcript: dict, metric) -> dict | None:
    facility = transcript.get("recommended_facility")
    if facility is None:
        return None

    test_case = LLMTestCase(
        input=transcript["message"],
        actual_output=transcript["response_text"],
        retrieval_context=build_retrieval_context(facility),
    )
    metric.measure(test_case)
    return {
        "message": transcript["message"],
        "score": metric.score,
        "success": metric.success,
        "reason": metric.reason,
    }


def summarize_scores(results: list[dict]) -> dict:
    if not results:
        return {"count": 0, "mean_score": 0.0, "pass_rate": 0.0}
    count = len(results)
    mean_score = sum(r["score"] for r in results) / count
    pass_rate = sum(1 for r in results if r["success"]) / count
    return {"count": count, "mean_score": mean_score, "pass_rate": pass_rate}