Incident Recovery Runbooks — Execution Action Classes (D6-3)
Scope: every registered actuator class in services/service-action-runner/execution_adapters/.
Recovery order (OFFERING_MAP §D.6.3, enforced order): freeze class → revert via the
rollback path → root-cause into the ledger → recertification before re-promotion.
Machinery this runbook drives (all landed, test-pinned): the D6-2 freeze registry
(execution_adapters/reconciliation.py — unreconciled write ⇒ UNRECONCILED_WRITE_CLASS_FREEZE
ERROR alarm + class freeze), the §4.3 lifecycle (run_execution/run_rollback), per-provider
kill switches (flags.py), and the D6-4 metric-contract pins
(mizoki_contracts/metric_contracts.py).
claim_label: built, pre-benchmark. No recovery drill has been executed against a real
provider account (EXEC-1 is open); the first recorded drill is what closes D6-3's ACT half.
0. The four steps, generic (apply per class below)
- FREEZE. On an unreconciled write the D6-2 registry freezes the
(provider, action_class)automatically and fires the alarm. To freeze manually (suspected bad class, provider incident), engage the provider kill switch —EXECUTION_ADAPTER_<PROVIDER>_KILL— via a config redeploy (never a code edit), or the global switch for a fleet stop. Verify: the next execution for the class refuses (ActionClassFrozen/kill_switch_engaged) BEFORE any transport call. - REVERT.
run_rollback(adapter, action, pre_state)per affected action — the freeze deliberately does NOT block the rollback path. Every rollback is verified by read-back (verify_rollback); a rollback without read-back proof is not a revert, it is a hope. Irreversible classes (see table) have no revert leg: containment is the kill switch plus provider-side remediation, which is why they are permanently approval-gated and can never reach Stage 4. - ROOT-CAUSE INTO THE LEDGER. Record via
python3 scripts/claude_memory.py recordwith the alarm line, theidempotency_key, pre/post-state evidence, and the fix. TheFreezeRecord(reason, timestamp, idempotency_key) is the starting evidence; the audit chain holds the action lifecycle. A freeze cleared without a recorded cause is a defect, not a recovery. - RECERTIFY BEFORE RE-PROMOTION. Clear the freeze only with an attributable operator
identity (
FREEZES.clear(provider, class, operator="<human>")— blank identities are refused). Stage-4 re-promotion (if the class ever held it) requires a fresh certification pack pinning currentcontract_fingerprint()values per consumed metric (D6-4) plus the five proofs incl. a recorded rollback drill — GOVERNANCE Art. 3.4; promotion is an owner decision, never part of this runbook.
1. Per-class register
| Actuator class | Reversible | Revert leg | Class-specific notes |
|---|---|---|---|
meta_capi.conversion_upload |
NO | none — a delivered conversion event cannot be un-sent | Permanently sub-Stage-4 by registration. Containment: EXECUTION_ADAPTER_META_CAPI_KILL, then provider-side: shared event_id dedup (48h window) means a re-send is absorbed, and mis-sent values are corrected forward via value_restatement, never deleted |
meta_capi.value_restatement |
yes | restate the prior value (the adapter's own compensating action, read-back verified) | The corrective tool for the row above — a restatement of a restatement is still forward-only truth |
meta_ads.campaign_budget |
yes | rollback to pre_state budget, read-back verified |
Bounds refuse unbounded increases; a frozen class blocks further budget writes while the revert stays open |
meta_ads.campaign_status |
yes | restore pre_state status |
Status flips are the lowest-risk revert; verify serving state, not the API ack |
meta_ads.adset_bid |
yes | restore pre_state bid |
|
meta_ads.adset_audience |
yes | restore pre_state audience config |
Audience reverts can lag provider-side; read-back until converged or escalate |
google_ads.campaign_budget |
yes | rollback to pre_state.amount_micros, read-back verified |
The drilled reference path in tests (test_rollback_path_stays_open_while_frozen) |
google_ads.campaign_status |
yes | restore pre_state status |
REMOVED is refused at execute time (irreversible state) — never "revert" by removing |
google_ads.ad_group_bid |
yes | restore pre_state bid |
|
google_ads.keyword_bid |
yes | restore pre_state bid |
|
google_ads.keyword_status |
yes | restore pre_state status |
|
linkedin_ads.campaign_status |
yes | restore pre_state status |
Wave-3, available: false — no tenant can hold credentials yet (EXEC-LI-1); a freeze here indicates catalog drift, treat as a finding |
linkedin_ads.campaign_budget |
yes | restore pre_state budget |
Same wave-3 caveat |
2. What closes the D6-3 ACT half
One recorded recovery drill: seed a freeze on a reversible class against a real
provider account (under EXEC-1's own governance pass), execute steps 1–4 end to end,
and record the drill with its read-back evidence. Until then this runbook is
implemented, the drill is blocked on EXEC-1's live engagement, and neither claims
the other's status.