kg_edges Retirement Plan — writers-first migration design

Status: EXECUTED — closed 2026-08-24. All waves landed and Wave 7's delete ran (operator, 2026-08-20: 422 twins verified, 422 deleted, unmappable 0); kg_relationships is the sole edge store and the last open decision (D8 kg-proxy) closed 2026-08-24. One live hazard outlives the plan: its D8 row explains why the remaining "operator gcloud builds submit" line must NOT be run — see §3c. Body below is design-vintage (measured 2026-08-19, read-only pass at main be339abe6; line numbers verified against that tree) and is kept as the execution record, not as outstanding work. Register item 8 is the tracking row; marketing-commerce-connectors.md rule 6a carries the standing HOLD. Decision context: item 8a settled 2026-08-19 — writers-first sequencing, no unscoped --apply; the delete (scripts/retire_kg_edges.py, operator) is the LAST step. Register rows: docs/reports/MIGRATION_CLOSURE_REGISTER_2026-09-02.md MC-01…MC-06 (Wave 1, 2026-09-02) — closure tests in tests/test_migration_closure_register.py.

0. The corrected inventory (supersedes the "seven writers / five readers" map)

A wide multiline grep (collection("kg_edges") … .set|.add|.update|.delete within 250 chars) found what the recorded same-line grep missed: five writers use asyncio.to_thread(<ref>.document(x).set, payload) — .set with no parenthesis. Excluded as different collections: threat_kg_edges, mt_kg_edges, gemini_kg_edges, gdp_kg_edges, fl_kg_edges, fl_causal_kg_edges, agent_registry_kg_edges, mart_kg_edges_for_neo4j.

15 write sites (12 live in-service, 1 local-dev seeder, 2 delete sites), 12 read sites. Not one write site double-writes kg_relationships — every edge written to kg_edges is invisible to Cell 3 GraphRAG retrieval today. CORRECTED 2026-08-20: even this multiline grep missed the kg_projections cluster — a live writer that names its collection through a variable (config["collections"]["edges"]) — so the true totals are 16 write sites / 13+ read sites; see §0b. The lesson (variable-collection-name writes evade every literal grep) is recorded cross-repo in shared-memory.

Writers (all collection names hardcoded)

# Path:line Doc id Kind Note
1 miz-oki-adk-agents/boss/connector_gateway/kg_writer.py:141 auto-id (.add()) new edge non-idempotent; _safe() swallows failures (:170-174); dead Neo4j branch :142-153. (No services/ copy exists)
2 miz-oki-adk-agents/boss/boss_agent_core.py:33347 edge_{uuid4[:12]} new edge reached by unauthenticated routes — see Wave 2
3 miz-oki-adk-agents/boss/policy_engine_integration.py:960 edge.idempotency_key audit append uses from_id/to_id — unmappable by the migration script
4 miz-oki-adk-agents/boss/ads_causal_controller.py:1361 edge_{user}_{asset}_{ts} new edge paired reader :1434 range-queries top-level timestamp
5 miz-oki-adk-agents/boss/eshkg_signal_integration.py:1203 pre-existing id, .update() enrichment 404s if the doc's creator migrates first; paired reader :1480
6 miz-oki-adk-agents/boss/privacy_safe_joins_attribution.py:656 caller id or edge_{uuid4[:8]} new edge merge=True; caller-supplied shape
7 scripts/process_mycocoons_to_kg.py:448 deterministic new edge, batch easiest: already id/source/target/properties; flip relation→type (:160,203,299 + kg_events mirror :459)
8 miz-oki-adk-agents/boss/budget_reallocator_integration.py:604 budget_{channel} new edge no source id — cannot be a canonical relationship as written
9 miz-oki-adk-agents/boss/autonomous_budget_reallocation_mvp.py:620 roi_{aud}_{camp} new edge from_id/to_id convention
10 miz-oki-adk-agents/boss/edge_roi_inference_integration.py:531 cate_{impression} new edge
11 miz-oki-adk-agents/boss/creative_test_deploy_integration.py:818 exp_{experiment} new edge
12 miz-oki-adk-agents/boss/journey_scoring_autopilot_integration.py:404 journey_score_{user}_{ts} new edge no target id — reclassify, cannot be an edge
13 boss_agent_core.py:33359 — DELETE (delete_edge) reached by unauthenticated DELETE /api/v1/kg/edges/{id} (:38536)
14 boss_agent_core.py:33313,33317 — DELETE cascade (delete_node) query-then-delete on source_id/target_id
15 miz-oki-adk-agents/boss/local_dev/local_server.py:176 edge_{i:03d} local-dev seeder mock-guarded (:169); out of scope

Writers 1–6 and 8–14 all ship in the boss-agent-adk image (Dockerfile.v5:27 COPYs all of boss/; imports verified at boss_agent_core.py module level).

Readers

Boss get_edges (boss_agent_core.py:33573 — filters relation, needs type after the flip), /api/v1/kg/stats dual-ledger counts (:36157 + kg_controller:33508 — the deliberate row; decision needed: keep reporting 0 as the retirement receipt, or remove the key + update boss.ts:218,242 and two UI tests asserting kg_edges: 310), delete_node cascade (:33312), command-center BFF/adapter/browse/hook (proxy + copy-only changes; PREFERRED_COLUMNS already handles both type and relation), srpaldl-cells/kg-proxy/app.py:416 (needs source/target/type + a composite index; IAM-locked, no CI deploy path — manual gcloud builds submit from a clean git archive), plus paired readers eshkg:1480, autonomous_budget:582 (4-field composite query), ads_causal:1434, policy_shaped_decision_layer.py:259, temporal_kg_snapshots.py:272,277 (checksums change when the collection empties), change_detection:291 (already reads type). One orphan reader: budget_reallocator_integration.py:285 (uplift_{campaign} docs) — dead against live data (nothing writes those docs to kg_edges; note the writer in the SAME file at :604 writes budget_{channel} docs, a different id scheme). CORRECTION (measured 2026-08-20): the previously-listed second "orphan" knowledge_graph_brain_integration.py:759 (src.nodeId/dst.nodeId/rel) is NOT dead — it is the live reader of the kg_projections cluster (§0b), a writer the literal collection("kg_edges") grep missed. Some of the 5 firestore.indexes.json composites serve that cluster's live query_edges, not only the one true orphan — re-audit before treating them as removable.

0b. The kg_projections cluster — a LIVE writer the grep missed (added 2026-08-20)

Both the §0 inventory and the recorded same-line grep were built from a literal collection("kg_edges") search. kg_projections_integration.py writes through a variable collection name, so it was invisible to every prior pass:

Consequences for the retirement: 1. It is an additional live writer + two readers not in the §0 table. 2. Its nested src/dst/rel docs are unmappable by canonical_from_legacy() (flat-field only) → they ABORT the all-or-nothing retire, same class as writers 3/8/9/12. A dedicated nested→canonical mapper is required (src.nodeId→source, dst.nodeId→target, rel→type, rest→properties). 3. query_edges needs a composite index on the canonical mapping. 4. Operator verification (cannot close from the repo): whether the deployed boss-agent-adk env sets ENABLE_KG_PROJECTIONS=false. Code default is live; an env override to false makes the cluster dormant and drops it in priority. Confirm the served env before scheduling this cluster's migration.

Neither kg_projections_integration.py nor knowledge_graph_brain_integration.py is on the deploy-boss-agent-core.yml allowlist (they ship in the image but a reader-only PR to either alone merges-not-serving — bundle with an allowlisted file, per Wave 1).

1. Target decision: direct canonical shape (Target A)

The governance-preferred path (MAPPERS[source] → ingest_gate → service-canonical-ingestion) publishes to mizoki-events and stops — no subscriber projects to kg_relationships (register item 26; verified end-to-end in service-canonical-ingestion/main.py:194-234). Routing writers there today is silent edge loss. Therefore: writers repoint DIRECTLY to kg_relationships in the canonical_relationship() shape ({id, source, target, type, properties}, doc id = id, .set(merge=True); kg_sinks.py:124-151 is the reference; Cell 3's reader consumes exactly this, graphrag_service.py:526-614). Re-route onto governed ingestion when item 26 lands a projector — recorded follow-on, not this lane.

2. Dual-write: NO

(a) nothing double-writes today, so there is no window to preserve; (b) the Boss/UI reads flip with their writers; (c) a dual-write keeps kg_edges growing, which the all-or-nothing retire verify cannot tolerate; (d) writers 1–2 use random ids, so dual-writes cannot be reconciled by id. Instead: cutover per writer, then re-run scripts/migrate_kg_edges_to_kg_relationships.py (idempotent, additive, skip-existing — proven 2026-08-11: +422, 0 unmappable) as the sweep reconciler after each wave.

3. Waves (lowest blast radius first; deploy-coupled last)

Wave 7 stale-reference sweep list (measured 2026-08-20, post-5b)

A repo-wide re-measure (revised 2026-08-20 post-item-8a, after a parallel session landed writer 6, D6, D8 and the orphan reader) confirms no executing kg_edges Firestore site remains in the boss lane: what is left in *.py is comments, the migration/retire scripts that legitimately name the legacy collection, the intentional D7 dual-ledger count, and unrelated collections (gemini_kg_edges, gdp_kg_edges, fl_kg_edges, threat_kg_edges, mt_kg_edges, agent_registry_kg_edges). But the earlier inventory was scoped to *.py, so these non-executing references were never enumerated. None is a migration blocker; all become wrong once the collection is deleted, so the retire sweep must include them. SWEPT 2026-08-20 — all six flipped to kg_relationships and pinned in both directions by test_wave7_swept_reference_is_canonical (parametrized, 6 cases). Each safety claim below was re-verified before editing, not taken from this table:

Reference Why it is not live
boss/archive/boss_agent_merged_v5.0.0.py (3 collection() handles of 7 mentions) archive/ — superseded copy, not imported
~~boss/local_dev/local_server.py:176~~ RESOLVED on main (fb6411064) — seeder now writes kg_relationships. The remaining :519/:600 hits are a route name and an in-memory dict key, not Firestore handles
boss/lib/agents/knowledge/firestore-graph-store.ts:133, kg-ingest-client.ts:99 TypeScript inside a python:3.11-slim image (Dockerfile.v5); imported only by the lib/agents/index.ts barrel — never executes in the boss service
boss/lib/agents/mcp/server.ts:456 hardcoded stub (count: 0, "Connect to actual KG store") — 'kg_edges' is a response label, not a collection handle
miz-oki-command-center-ui/lib/firebase-admin.ts:84 (kgEdges: 'kg_edges') constant declared but never consumed (the other kgEdges hits are an unrelated useState in NeuralBrainVisualization.tsx). Path corrected 2026-08-20: an earlier row said command-center-ui/..., which does not exist — the narrower path would have read as absence
boss/config/budget_reallocation_config.yaml:263, config/kg_ingestion_config.yaml:388 inert — neither file has any Python loader (git grep -ln "<name>" -- '*.py' ⇒ empty), so no config-driven collection-name indirection exists here

Two near-misses worth recording so they are not re-investigated: the UI browse surface (/api/bff/kg/edges) proxies the boss API rather than reading Firestore, so it already rides wave 5a's migrated origin-scoped get_edges; and roi_edges / kg_roi_edges / abr_roi_edges / gdp_kg_edges are different collections, not kg_edges — out of scope for this retirement.

Per-writer tests ship with each wave: a source-literal assertion on the collection constant and a shape assertion on the emitted doc (rule 01). Waves 2a/2b/3/4 (both wave-4 clusters), 5a, 5b + writers 3/8/12, the wave-6 clean readers, (from the parallel session) writer 6, the D6 kg_projections cluster and D8, and the wave-7 stale-reference sweep are all pinned by tests/test_kg_write_perimeter.py (52 tests, green 2026-08-20). The sweep pin also guards the opposite direction: /api/v1/kg/stats' intentional D7 dual-ledger count and its local_dev mirror must NOT be swept, or the evidence the retirement is measured by disappears.

3c. Open decisions (owner rulings needed before Wave 5+ build)

The measurement surfaced choices that are NOT mechanical repoints. Each has a recommendation; a one-line ruling per item unblocks the build.

# Decision Recommendation
D1 get_edges open-surface scope (5a) Keep scoped to the CRUD type — do NOT broaden the public browse surface to all canonical edges
D2 Writer 3 transition_executed (no reader) BUILT 2026-08-20 — collapsed into kg_transitions; kg_edges write dropped (method now a documented no-op). See writers §.
D3 Writer 8 BUDGET_ALLOCATION (no source_id) BUILT 2026-08-20 — reclassified OUT; kg_edges write dropped, budget_allocations kept. See writers §.
D4 Writer 12 journey:score (no target_id) BUILT 2026-08-20 — reclassified OUT; kg_edges write dropped, autopilot_scores kept. See writers §.
D5 Writer 1 deterministic id FINAL 2026-08-20 — "edge_" + md5(f"{tenant_id}\|{source_id}\|{edge_type}\|{target_id}")[:16] (natural-key idempotent; md5 not Python hash(); drop both dead Neo4j branches). See 5b.
D6 kg_projections cluster (§0b) BUILT 2026-08-20 — confirmed LIVE; config→kg_relationships, writer+2 readers remapped canonical, 3 composite indexes declared, 5 perimeter tests
D7 /api/v1/kg/stats dual-ledger count DECIDED 2026-08-20 — keep reporting kg_edges count as retirement receipt (naturally 0 once drained); no code change needed, current dual-count code is correct
D8 kg-proxy CLOSED 2026-08-24 — the service is ALREADY DEPLOYED; do not run the completion step. Code was CODE-COMPLETE 2026-08-20 (reader flipped to kg_relationships with canonical field names; (source,type) composite index declared) and a deploy has since landed, so the plan's remaining "operator gcloud builds submit" line is spent. Running it now buys nothing and costs something: cloudbuild.yaml:46 carries --allow-unauthenticated and an unauth-curl smoke test, so a redeploy re-asserts the allUsers invoker binding. Ingress (internal-and-cloud-load-balancing) still holds the lock, so that is a standing least-privilege gap rather than an immediate public opening — but it is a hole that opens the moment ingress is ever loosened, and rule 04 is explicit that a later --no-allow-unauthenticated does not remove the binding. There is no CI path either (grep -rl "kg-proxy" .github/workflows/ = empty), so this file stays armed for whoever reads the plan literally next. The durable close is fixing the cloudbuild — dropping the flag and the unauth smoke test — not deploying again.

Once D1–D5 and D7 are ruled, the buildable order is: 5a boss-CRUD cluster (mechanical) ✅ → Wave 6 clean readers (bundled to deploy) ✅ → 5b writer 1 + writers 3/8/12 per ruling ✅ (all CODE-COMPLETE 2026-08-20, GATE-1 local) → 5c writer 6 ✅ (CODE-COMPLETE 2026-08-20) → kg_projections (D6) ✅ (CODE-COMPLETE 2026-08-20) → orphan reader budget_reallocator:285 ✅ (dead read removed 2026-08-20) → D7 ✅ (DECIDED: keep dual-count as retirement receipt) → D8 kg-proxy ✅ (CODE-COMPLETE 2026-08-20; deploy operator-only) → Wave 7 reconcile + retire (operator --apply sweep).

3b. Wave-4 operator index apply (kg_relationships) — GATE-2

Wave 4 declared four composite indexes in boss/firestore.indexes.json that no CI applies (verified: zero firestore.indexes refs under .github/workflows/, pinned by a test). Apply them to the boss Firestore database with the operator index-apply path (firebase deploy --only firestore:indexes, or gcloud firestore indexes composite create per index). Agents never run gcloud/firebase (AGENTS.md) — this is operator/GATE-2. Two tiers:

Tier 1 — RECOVERY (apply promptly; the reader is LIVE). The Touchpoint reader kg_time_decayed_uplift degrades to empty + ERROR-log until these land: - kg_relationships (type ASC, properties.timestamp ASC) — no-asset query - kg_relationships (type ASC, target ASC, properties.timestamp ASC) — by-asset query

Tier 2 — RESOLVED 2026-08-20; it was a code defect, not an activation gate. This tier previously said applying the reader-582 index "reactivates that budget input". That was false, and the doubt embedded in its own text ("confirm Firestore accepts the shape, or reorder the query first") was the correct instinct: Firestore does NOT accept it. The query was malformed — inequality on one field, first order_by on another — so it failed regardless of indexing, and no index applied or withheld ever gated it. The query is fixed; the index declaration below now describes the legal query.

Activation posture is unchanged and was never carried by the index: the module's public entrypoints default dry_run=True, create_decision yields PROPOSED/PENDING_APPROVAL rather than APPLIED, actuation is a separate explicit call, and no scheduler, cron, or terraform drives the module (measured 2026-08-20). Restoring a read leg is not arming an actuator. - kg_relationships (type ASC, target ASC, properties.confidence DESC) — reader 582, matching the corrected query. properties.delta is no longer an index field: delta ranking happens in-process. - kg_relationships (type ASC, properties.channel ASC) — reader 1480 channel path.

4. Preconditions for scripts/retire_kg_edges.py --apply (item 8b)

  1. Writer quiescence proof — the script snapshots then deletes (plan_retirement(), :70-78); a write between passes is deleted unverified. Run only after Wave 6, with the count stable across two passes.
  2. Zero unmappable — status re-measured 2026-08-20: - (a) from_id/to_id/edge_type (writer 3's PolicyEdge.to_dict(), confirmed from source history) — ✅ CLOSED: the mapper now reads these as canonical endpoints and type, and excludes them from the properties catch-all so they are not duplicated. edge_type had to join the type fallback too, or such rows would map as "RELATED" with the real type demoted — a quieter defect than the abort it replaces. 4 tests, 3 of them mutation-proved red against the prior mapper. - (b) single-endpoint docs (writers 8, 12) — OPEN BY DESIGN. An edge without both endpoints is not an edge; None is correct and retire failing closed on it is correct. Only an operator can prove the class is empty (a Firestore read). If it is non-empty, that is a decision — delete the orphans or exclude them by id — not a mapper change. - (c) nested src/dst/rel (kg_projections, §0b) — ✅ CLOSED, mapper reads src.nodeId/dst.nodeId/rel.

Note the twin check is by id, not by re-mapping: an unmapped doc simply never got a twin, which is exactly how an unmappable class becomes an abort. 3. Dual-ledger stats decision (reader table above). 4. Non-code cleanup the script does not touch — re-measured 2026-08-20: - miz-oki-adk-agents/boss/firestore.rules match /kg_edges/{edgeId} + doc comment — ✅ removed 2026-08-20 (post-delete). SOURCE-ONLY: no workflow applies the rules file, so the served ruleset changes only when an operator deploys rules. Removing it is also the safer resting state — with no match block, a recreated kg_edges is denied by default instead of writable. - 5 orphan kg_edges composite indexes — ✅ removed 2026-08-20. Also declaration-only (no workflow applies the index file, by design). - firebase-admin.ts:84 constant — ✅ swept (wave-7 sweep). - both inert config mentions — ✅ swept (wave-7 sweep). None of these blocks the delete; all become dead references after it. 5. The script authenticates via gcloud auth print-access-token — operator-only, unchanged.

← All docsView source on GitHub →