Two-Tier Facility State: In-Process Cache + Redis Wait Times
Architectural Overview
Splitting facility state into two tiers that match their actual freshness and failure requirements: an in-process ETag cache for the rarely-changing facility directory, and a Redis cache-aside chain with a Supabase fallback for wait times that change every scrape cycle.
Two very different pieces of facility data feed every triage decision: the facility directory itself (name, address, accepted severities, changes rarely) and current ER wait times (changes every scrape cycle). They have different freshness requirements, different failure tolerances, and, in the current implementation, genuinely different storage.
The Problem
Wait-time data goes stale fast and comes from unreliable external sources: a scheduled worker scrapes multiple public ER-wait sites every ~15 minutes, and any one of those sources, or Redis itself, can be down at read time. A routing decision can't just fail because a third-party wait-time site timed out; it has to degrade to something reasonable. Meanwhile the facility directory needs to survive being read on every single triage request without a database round trip each time, but a plain in-process cache doesn't survive a redeploy or a second server instance, which is a real limitation, not a hypothetical one.
Wait-time scrapers hit third-party sites that can go down, change their markup, or return stale numbers with no warning. The read path has to assume any single source can fail on any given request.
The facility directory cache lives in a single Python process's memory. It's fast, but it's also gone on restart and invisible to a second server instance. There's no cross-process consistency today.
The Architecture Strategy
We split facility state into two tiers that match their actual freshness and failure requirements. The facility directory lives as a module-level dict, populated from a Supabase query and stamped with a SHA-256 ETag over its sorted-key JSON serialization, cheap to read on every request, and the ETag lets callers detect 'nothing changed' without re-fetching. Wait times go through a proper cache-aside chain: read the Redis hash first, since that's what the scraper writes every ~15 minutes; on a Redis error or an empty hash (cold start, before the first scrape has ever run), fall back to a Supabase RPC and best-effort repopulate Redis from that result via a pipeline write, so the next read doesn't have to hit Supabase again. If both Redis and the Supabase fallback fail, the function returns an empty map rather than raising. Missing wait data is treated the same way missing hours data is treated elsewhere in the codebase: it always passes filters rather than blocking a routing decision.
_cache: dict[str, Any] = {"facilities": None, "etag": None}
def get_cached_facilities() -> tuple[list[dict] | None, str | None]:
return _cache["facilities"], _cache["etag"]
def set_cached_facilities(data: list[dict]) -> str:
serialized = json.dumps(data, sort_keys=True, default=str)
etag = f'"{hashlib.sha256(serialized.encode()).hexdigest()[:32]}"'
_cache["facilities"] = data
_cache["etag"] = etag
return etagdef get_wait_minutes_map() -> dict[str, int | None]:
try:
raw = redis_client.hgetall(REDIS_HASH_KEY)
if raw:
return {fid: json.loads(v).get("wait_minutes") for fid, v in raw.items()}
except Exception:
logger.warning("redis_unavailable_falling_back_to_supabase")
try:
rows = supabase_rpc("latest_wait_times", {})
except Exception:
logger.warning("wait_times_fallback_failed_returning_empty")
return {}
wait_map = {r["facility_id"]: r["wait_minutes"] for r in rows}
try:
pipe = redis_client.pipeline()
for r in rows:
pipe.hset(REDIS_HASH_KEY, r["facility_id"], json.dumps({
"wait_minutes": r["wait_minutes"], "raw_wait": r.get("raw_wait"),
}))
pipe.execute()
except Exception:
logger.warning("redis_populate_failed")
return wait_mapAlternatives Considered
Putting the facility directory in Redis too, alongside wait times, would remove the single-process limitation immediately. We didn't do that yet because the facility directory changes at a completely different rate than wait times: it's edited by hand or by an admin process, not scraped every 15 minutes. Adding a network hop to read data on every single triage request for something that rarely changes is the wrong trade when a single-process deployment is what's actually running today.
The tempting alternative to degrading to an empty wait-time map is to raise and block the routing decision until wait data is available. We rejected that because a routing decision without wait-time data is still strictly better than no routing decision at all. The same reasoning already applied to the facility hours filters, extended here for consistency rather than inventing a new failure convention for this one data source.
Lessons Learned
The first version of the Supabase fallback path let a Redis write failure during the best-effort repopulate step propagate up and mask a successful Supabase read. The caller would see an error even though it actually had good wait-time data in hand. Wrapping just that pipeline write in its own try/except, separate from the read path exception handling, fixed it.
Early versions of the facility cache had no ETag and downstream consumers re-serialized the full list on every poll to check whether anything had changed. Hashing the sorted-key JSON once at write time and comparing that string is far cheaper than re-diffing a list of facility dicts.
The facility-directory cache is explicitly a Phase 1 shortcut: it works for single-node deployment but doesn't survive horizontal scaling or process restarts. Every server instance would build its own independent view of the facility directory, with no invalidation signal between them. Wait times don't have that problem since Redis is already the shared store, but the two-tier split means the two data types have genuinely different consistency guarantees today, which is worth knowing if you're debugging why one field updated instantly and another didn't. What's next: moving the facility directory into the same shared store as wait times (Redis Cluster with AOF persistence) is the change that would let this run on more than one process. Priority-queue gating (routing emergent cases first, letting moderate/routine absorb remaining capacity) is also not backend logic yet; today that coordination is what Sandbox Mode visualizes on the frontend, not something this cache enforces server-side.
- 100% cache hit rate across 300 wait-time reads (300 redis_hit, 0 supabase_fallback) under a sustained read burst.
- Cache-aside mechanism confirmed correct end-to-end in an earlier verification pass: 1st read after a cold Redis hash triggered supabase_fallback and repopulated Redis, 2nd read correctly hit redis_hit.
- wait_times_cache_outcome_total: a Prometheus counter labeled by outcome (redis_hit, supabase_fallback, total_failure), incremented on the existing cache-aside branches inside get_wait_minutes_map(). Zero added latency, no new endpoint.
- 300 unauthenticated GET /facilities requests fired via a ThreadPoolExecutor-based Python script against the eval-project preview backend — not a load-testing tool, same approach as case study 2.
- Hit rate computed by diffing wait_times_cache_outcome_total before and after the burst, read directly off /metrics. Window: 2026-07-14, eval Supabase project.