Decision: intent-signals topic provisioning and DLQ disposition

Date: 2026-08-28 Authority: owner in-session approval ("make the proper decision … you're approved to finalize this whole problem"), following the gemini-kg-pipeline deploy-failure investigation (2026-08-27) that surfaced the dead-letter backlog as a side finding. State labels per TRUTH.md: every measurement below is a verified result from this session's own commands unless labeled otherwise.

1. What was actually broken (corrected twice en route)

The investigation's understanding improved in three steps, and the earlier steps are on the ledger — this section supersedes them:

1. First reading (2026-08-28 morning, ledger record): 5,759 of 5,865 documents in the events Firestore collection are dead-lettered in events_dlq — every one 404 Resource not found (resource=intent-signals) after 5 attempts, from the 2026-08-01 Shopify backfill. Proposed remedy at the time: create the topic, then replay_dlq.py --all. The replay half of that record is WRONG (see §3) — this document corrects it.
2. Second reading: the live canonical-ingestion service publishes to EVENTS_TOPIC=mizoki-events (verified on the serving revision), so the 404s looked like a fossil of an old wrong config, and the fix looked like "replay through the drainer to mizoki-events." Also wrong.
3. Definitive reading (schema-verified): the dead events are not canonical envelopes at all. Full-field inspection shows audit_id: intent-…, provenance.ingest_cell: intent-signal-ingest, and the fv2 signal fields (strength, materiality, confidence, topic_id: media_dtc_repeat_purchase). These are Cell 33 accepted intent signals. Cell 33 and canonical-ingestion both use mizoki_contracts with the SAME collection constants (events, events_outbox, events_dlq), so both lanes' rows are physically interleaved in one collection set. Cell 33's publisher targets INTENT_SIGNALS_TOPIC=intent-signals (cloudbuild + config default) — a topic that was never provisioned, while its twin intent-transitions was.

2. The decision

Create the intent-signals topic, as IaC, in deployment/terraform/intent_signals_topic/. Do NOT replay the 5,759 dead notification rows — they stay parked in the DLQ as the incident record.

Where it lives, and why there

• A dedicated small terraform module (house pattern: gemini_kg_pipeline_sa/, canonical_events_landing/), not a gcloud one-liner, so the resource has an IaC record, a reviewable diff, and a README the next session can trust. deployment/terraform/** is a protected path by design — the right place for an ingress-adjacent transport resource.
• Not inside canonical_events_landing/ — wrong lane. That module is the canonical stream (mizoki-events); this topic is the intent lane's. Mixing them would make terraform destroy of either module a cross-lane hazard and would blur exactly the lane boundary the incident violated.
• Not by importing intent-transitions alongside it — that topic exists, is unmanaged, and is load-bearing for Cell 35 (intent-transitions-cell35 subscription). Importing live resources into fresh state is a separate, riskier decision; this module stays additive-only.
• Topic only — no subscription, no DLQ topic. A dead-letter topic is a property of a subscription; no consumer of accepted-signal notifications exists yet. Surface without function is standing least-privilege debt. The first subscriber lands with its own DLQ in the same change.

Why create it at all (advantages)

1. It stops the incident from recurring. Cell 33 is deployed with INTENT_PUBSUB_MODE=auto and this topic name; on the next real ingest (the mycocoons pilot is live-armed and Shopify events are the lane's feed), every accepted signal's notification would again 404 → 5 attempts → DLQ. With the topic present, the inline publish succeeds.
2. It closes most of the cross-lane drainer window. Because the outbox collections are shared, a pending Cell-33 row can be picked up by canonical-ingestion's 5-second drainer and published to mizoki-events — the wrong stream. Rows only sit pending when the inline publish fails; a working topic makes that window effectively zero in normal operation. (The structural fix is register item 34 — see §4.)
3. It trues a false LIVE claim. The skillpack and LII RUNBOOK have claimed "Pub/Sub intent-signals, intent-transitions" as part of the shipped intent family since 2026-07-31. Half of that was untrue. Truth-discipline says fix the claim or fix the resource; the resource is the one Cell 33's deployed config already depends on.
4. Deployed-mode fail-closed stays honest. Cell 33's publisher refuses to boot in deployed mode if Pub/Sub is unavailable (D8-class rule). A permanently missing topic makes that guard a permanent boot hazard on any revision that exercises it.

Disadvantages considered, and why they don't win

1. "A topic nothing subscribes to is dead surface." — True today, but it is publisher-side load-bearing: its absence dead-letters real work and can block boot. The asymmetry (absence causes measured damage; presence costs ~nothing — Pub/Sub topics are free at zero throughput) decides it.
2. "Messages published before a subscriber exists are dropped." — Also true, and accepted: the notifications are transient fan-out, not the store of record. BigQuery mizoki_intent.intent_signals (5,756 rows measured) and Firestore events already hold every signal. Cell 34 scoring reads BigQuery, not Pub/Sub.
3. "Creating intent infra touches the intent lane's governance." — A transport topic is not activation. No holdout, no export, no scoring change. The no-activation-without-holdout rule is untouched.

Why the DLQ is NOT replayed (this reverses the morning ledger record)

1. Nothing would receive the messages. Pub/Sub delivers only to subscriptions that exist at publish time; there are none. Replaying 5,759 notifications about signals observed in 2024 into a subscriber-less topic is bookkeeping churn with zero information delivered.
2. The replay path is unsafe while the outbox is shared. replay_dead() flips rows to pending in the shared collection; canonical-ingestion's drainer (5 s cadence, EVENTS_TOPIC=mizoki-events) would race Cell 33's and could publish intent signals onto the canonical stream and into the canonical_events BigQuery landing — polluting that table's first-ever organic writes with wrong-schema rows.
3. The data needs no rescue. Both stores of record already have it (§1.3). The DLQ rows are the incident evidence, and the DLQ's own design principle is "parked, never dropped."
4. Re-running replay is additionally not idempotent (DLQ docs keep status: dead after a replay, so a second --all re-queues and double-publishes). One more reason not to normalize casual replays here.

3. What was implemented

• deployment/terraform/intent_signals_topic/ — module as decided above.
• terraform apply executed under the same owner approval (5→1 resources: the topic; apply output in the session record).
• Verified: gcloud pubsub topics describe intent-signals resolves.
• The 2026-08-28 morning ledger record's "then replay --all" instruction is superseded by this document (corrective ledger record filed the same day).

4. Named residuals (register)

• Item 34 — shared outbox collections are a cross-lane publish hazard (build): mizoki_contracts.outbox hardcodes events / events_outbox / events_dlq; every service that instantiates it shares them, and every service's drainer publishes ANY pending row to ITS OWN topic. Dormant today (0 pending), but structural: namespace the collections per lane (or stamp rows with their target topic and make drainers filter) before a second high-volume publisher lane arms.
• The 5,759 DLQ rows stay parked as the incident record. If a future subscriber genuinely needs historic accepted-signal notifications, the correct source is a backfill from BigQuery mizoki_intent.intent_signals (the store of record), not a DLQ replay.
• intent-transitions remains unmanaged by IaC (works, load-bearing); importing it into terraform state is a separate owner decision.