Runbook — Journey → KG lineage (rule 6 closure)
Scope: the operator tail of the 2026-08-10 owner directive to connect the
Gemini pipeline, journey determination, and KG input into one lineage.
Everything code-side landed on main (feat(kg): connect virtuoso journey
ingress to the KG; one edge store); the steps below need live GCP access and
CANNOT run from a Claude Code sandbox (gcloud/bq absent there).
Governing doc: .claude/memory/active/marketing-commerce-connectors.md
rule 6 — its measured table is the scoreboard. No claim that the lineage
spine is operational may be made until checks a–d pass with a citable
artifact.
What the code now does (context, not steps)
- Every inserted/updated event through
/api/v1/journey-events/virtuoso/{source}projects oneJourneyEventnode intokg_nodesand typed edges intokg_relationships(PERFORMED_BY → Customer,ACTED_ON → Campaign/AdGroup/Creative/Transaction). Disable:VIRTUOSO_KG_PROJECTION=0. - The journey KG sink writes edges ONLY to
kg_relationships, in the ledger shape Cell 3's GraphRAG edge sync reads.kg_edgesgains no new documents. - Raw email/ip/ua and free-text keyword/search_term/page never cross into the KG (pinned by test).
Operator steps, in order
- Confirm the deploy served. Merging the commit auto-deploys
gemini-kg-pipeline(path-triggered workflow).
bash
gcloud run services describe gemini-kg-pipeline --region=us-central1 \
--format='value(status.latestReadyRevisionName, status.conditions[0].status)'
latestReady == latestCreated, Ready=True.
- Migrate the 422 legacy edges (dry-run first, then apply):
bash
python3 scripts/migrate_kg_edges_to_kg_relationships.py # dry-run
python3 scripts/migrate_kg_edges_to_kg_relationships.py --limit 20 --apply
python3 scripts/migrate_kg_edges_to_kg_relationships.py --apply
Additive; kg_edges is never edited. Retiring kg_edges afterwards is a
separate deliberate step — do NOT delete it until rule 6a is re-measured.
- Backfill already-stored virtuoso events (same discipline):
bash
python3 scripts/backfill_virtuoso_journey_events_to_kg.py # dry-run
python3 scripts/backfill_virtuoso_journey_events_to_kg.py --apply
- Verify counts moved (a 200 from an async route is never proof — count):
bash
# JourneyEvent nodes should be > 0 for the first time
# (aggregation via Boss /api/v1/kg/stats or a Firestore console count)
Expected: kg_nodes gains type=JourneyEvent documents;
kg_relationships gains PERFORMED_BY / ACTED_ON edges; kg_edges
count stays frozen at 422.
-
Redeploy / re-sync Cell 3 so retrieval sees the new nodes+edges (rule 6b — the edge-sync code is on main; the serving revision must carry it). Note
MEMORY_SYNC_MAX_NODES(default 10,000) against the grown node count — raise it or the sync will legitimately drop the overflow and log the skipped count. -
Produce the rule 6d artifact: re-run
eval/graphrag/against the SAME frozen gold set, and treatgraph_coverageas real only where expectation sets are non-empty. Remember the endpoint is measured non-deterministic — one run per arm is not evidence; run several. -
Update the measured table in
.claude/memory/active/marketing-commerce-connectors.md(checks a–d) with the artifact id, and record the result viapython3 scripts/claude_memory.py record(ff-only sync first — see rule 02 on the theirs-merge splice).
Failure modes to expect
gcloud auth print-access-tokenexpiry mid-run: both scripts refresh proactively and retry once on 401 (house pattern from the dangling-edge backfill).- Edges whose Transaction target uses a REAL order id will show as skipped-dangling in Cell 3's sync until the order-node re-key lands — that is by design; the sync never invents endpoints.
- If
virtuoso_journey_eventsis empty, step 3 is a no-op — the collection only has documents if the virtuoso route received production traffic before projection wiring existed.