Event-Driven Fan-Out: S3 to Lambda to EventBridge to dbt

A central orchestrator has to know about every source up front — adding or retiring a source means editing shared orchestration state, not just adding or removing an independent resource. And a failure in one source's processor shouldn't block or freeze the sources that are healthy.

S3 Object Created events, filtered by key prefix (raw/places/, raw/er-wait/, raw/open-data/), trigger the matching processor Lambda directly — no orchestrator in between. Each processor publishes a custom ProcessorComplete event (status: SUCCESS, source medicoord.pipeline) to the default EventBridge bus after a successful Supabase upsert. One DbtRunnerRule matches that event pattern regardless of which of the three processors fired it, fanning all three into a single shared dbt-runner that never needs to know which source triggered it. Once triggered, dbt-runner runs three phases, not one: Phase 1 dbt run (transforms facilities_clean), Phase 2 dbt test (13 automated data-quality tests, 13/13 passing — an integrity gate, not just a transform), Phase 3 the medi_db_health_check Supabase RPC (dead tuples, long-running queries, deadlocks — a live database health check, not a pipeline-internal check). IAM backs the whole chain with three least-privilege roles: ingestion (SSM read + S3 write + logs), processor (S3 read + logs + EventBridge publish scoped to the default bus), dbt-runner (logs only — Supabase access is HTTPS with env-injected credentials, no AWS resource access needed). The pipeline is defined across two separate CloudFormation stacks, not one. s3-buckets.yaml (plain CloudFormation) defines the landing bucket with native EventBridgeConfiguration (S3-to-EventBridge without the manual console toggle that was tried first and abandoned), a 30-day lifecycle expiry (raw files are transient — cost hygiene, not an afterthought), and a full PublicAccessBlockConfiguration (this bucket is internal-only, stated explicitly rather than left to default). template.yaml (AWS SAM) defines the compute layer: 3 IAM roles, 4 processing/ingestion Lambdas plus dbt-runner, EventBridge rules. The S3 stack exports BucketName/BucketArn via CloudFormation Export; the compute stack consumes them via !ImportValue in every IAM policy and EventBridge rule that touches the bucket — one resource, one source of truth, zero hardcoded ARNs or duplicated bucket names across stacks.

Outputs:
  BucketName:
    Description: Name of the raw ingestion bucket
    Value: !Ref MedicoordRawBucket
    Export:
      Name: medicoord-ingestion-bucket-name

  BucketArn:
    Description: ARN of the raw ingestion bucket
    Value: !GetAtt MedicoordRawBucket.Arn
    Export:
      Name: medicoord-ingestion-bucket-arn

# One rule covers all three processors — fires when any processor
# publishes ProcessorComplete with status SUCCESS. dbt-runner never
# knows which source triggered it.
DbtRunnerRule:
  Type: AWS::Events::Rule
  Properties:
    Name: medicoord-dbt-runner-rule
    EventPattern:
      source:
        - medicoord.pipeline
      detail-type:
        - ProcessorComplete
      detail:
        status:
          - SUCCESS
    State: ENABLED
    Targets:
      - Id: DbtRunnerTarget
        Arn: !GetAtt DbtRunner.Arn

# Cross-stack reference — no hardcoded bucket ARN
S3Write:
  PolicyDocument:
    Statement:
      - Effect: Allow
        Action: s3:PutObject
        Resource: !Sub
          - '${BucketArn}/*'
          - BucketArn: !ImportValue medicoord-ingestion-bucket-arn

logger.info("Phase 1 — dbt run")
# ... run facilities_clean transform ...

logger.info("Phase 2 — dbt test")
# ... 13 automated data-quality tests, fails loudly on bad data ...

logger.info("Phase 3 — DB health checks (RPC)")
# ... medi_db_health_check: dead tuples, long-running queries, deadlocks ...