Incident Recovery Runbooks — Execution Action Classes (D6-3)

Scope: every registered actuator class in services/service-action-runner/execution_adapters/. Recovery order (OFFERING_MAP §D.6.3, enforced order): freeze class → revert via the rollback path → root-cause into the ledger → recertification before re-promotion. Machinery this runbook drives (all landed, test-pinned): the D6-2 freeze registry (execution_adapters/reconciliation.py — unreconciled write ⇒ UNRECONCILED_WRITE_CLASS_FREEZE ERROR alarm + class freeze), the §4.3 lifecycle (run_execution/run_rollback), per-provider kill switches (flags.py), and the D6-4 metric-contract pins (mizoki_contracts/metric_contracts.py). claim_label: built, pre-benchmark. No recovery drill has been executed against a real provider account (EXEC-1 is open); the first recorded drill is what closes D6-3's ACT half.


0. The four steps, generic (apply per class below)

  1. FREEZE. On an unreconciled write the D6-2 registry freezes the (provider, action_class) automatically and fires the alarm. To freeze manually (suspected bad class, provider incident), engage the provider kill switch — EXECUTION_ADAPTER_<PROVIDER>_KILL — via a config redeploy (never a code edit), or the global switch for a fleet stop. Verify: the next execution for the class refuses (ActionClassFrozen / kill_switch_engaged) BEFORE any transport call.
  2. REVERT. run_rollback(adapter, action, pre_state) per affected action — the freeze deliberately does NOT block the rollback path. Every rollback is verified by read-back (verify_rollback); a rollback without read-back proof is not a revert, it is a hope. Irreversible classes (see table) have no revert leg: containment is the kill switch plus provider-side remediation, which is why they are permanently approval-gated and can never reach Stage 4.
  3. ROOT-CAUSE INTO THE LEDGER. Record via python3 scripts/claude_memory.py record with the alarm line, the idempotency_key, pre/post-state evidence, and the fix. The FreezeRecord (reason, timestamp, idempotency_key) is the starting evidence; the audit chain holds the action lifecycle. A freeze cleared without a recorded cause is a defect, not a recovery.
  4. RECERTIFY BEFORE RE-PROMOTION. Clear the freeze only with an attributable operator identity (FREEZES.clear(provider, class, operator="<human>") — blank identities are refused). Stage-4 re-promotion (if the class ever held it) requires a fresh certification pack pinning current contract_fingerprint() values per consumed metric (D6-4) plus the five proofs incl. a recorded rollback drill — GOVERNANCE Art. 3.4; promotion is an owner decision, never part of this runbook.

1. Per-class register

Actuator class Reversible Revert leg Class-specific notes
meta_capi.conversion_upload NO none — a delivered conversion event cannot be un-sent Permanently sub-Stage-4 by registration. Containment: EXECUTION_ADAPTER_META_CAPI_KILL, then provider-side: shared event_id dedup (48h window) means a re-send is absorbed, and mis-sent values are corrected forward via value_restatement, never deleted
meta_capi.value_restatement yes restate the prior value (the adapter's own compensating action, read-back verified) The corrective tool for the row above — a restatement of a restatement is still forward-only truth
meta_ads.campaign_budget yes rollback to pre_state budget, read-back verified Bounds refuse unbounded increases; a frozen class blocks further budget writes while the revert stays open
meta_ads.campaign_status yes restore pre_state status Status flips are the lowest-risk revert; verify serving state, not the API ack
meta_ads.adset_bid yes restore pre_state bid
meta_ads.adset_audience yes restore pre_state audience config Audience reverts can lag provider-side; read-back until converged or escalate
google_ads.campaign_budget yes rollback to pre_state.amount_micros, read-back verified The drilled reference path in tests (test_rollback_path_stays_open_while_frozen)
google_ads.campaign_status yes restore pre_state status REMOVED is refused at execute time (irreversible state) — never "revert" by removing
google_ads.ad_group_bid yes restore pre_state bid
google_ads.keyword_bid yes restore pre_state bid
google_ads.keyword_status yes restore pre_state status
linkedin_ads.campaign_status yes restore pre_state status Wave-3, available: false — no tenant can hold credentials yet (EXEC-LI-1); a freeze here indicates catalog drift, treat as a finding
linkedin_ads.campaign_budget yes restore pre_state budget Same wave-3 caveat

2. What closes the D6-3 ACT half

One recorded recovery drill: seed a freeze on a reversible class against a real provider account (under EXEC-1's own governance pass), execute steps 1–4 end to end, and record the drill with its read-back evidence. Until then this runbook is implemented, the drill is blocked on EXEC-1's live engagement, and neither claims the other's status.

← All docsView source on GitHub →