SITE A/B/C TEST — DESIGN FIX: implementation report

Date: 2026-08-22 · Lane: site-abc-test-fix (5o8sng) · Branch: claude/site-abc-test-design-5o8sng Spec of record: SITE A/B/C TEST — DESIGN FIX r1.0 (owner-issued design spec; was "not yet implemented") State on merge: implemented — NOT deployed, NOT live. The marketing site ships only via an owner-dispatched deploy-homepage.yml run with the typed APPROVED token (CONSTITUTION III.1); nothing in this change alters production until that dispatch happens.

1. What the spec found, and what the tree actually held

The spec's three defects were checked against the tree before building (rule 01):

Spec claim	Tree reality
§1.1 all three variants indexed (`index,follow`, distinct canonicals)	Partly stale: `/marketing` was `index,follow` + own canonical; `/media` had own canonical and no robots meta (indexable by default); `/signal` was `noindex, nofollow` ("IN BUILD — flip at owner-approved launch"). The confounded-assignment defect stands regardless: traffic reaches the three URLs by search/nav intent, not randomization.
§1.2 every CTA a bare `mailto:` with no event and no variant attribution	Confirmed — 7 bare mailto CTAs across the three landings (3 distinct destination addresses), zero outcome instrumentation of any kind.
§1.3 three arms likely underpowered	No analytics exist to size it — which is itself defect §1.2. Resolved structurally (see §4: dark-mode baseline collection before any arming decision).

2. What was built

Canonical choice (checklist item 1): /signal — the spec's own worked example, the product name, and the URL already carrying five sub-pages in the sitemap.

mizoki_runtime/abtest.py (new): variant registry (a=signal.html, b=marketing/index.html, c=media/index.html), deterministic assignment (hash(visitor) % 3; cookie mv, 90d, SameSite=Lax, HttpOnly; cookieless clients keyed by salted hash of IP+UA so crawlers see stable content with no user-agent-conditional serving), head-posture rewriter, closed CTA registry, bot classifier, coarse referrer classing, structured mizoki_abtest JSON event log to stdout (→ Cloud Logging), per-instance SRM counters.
app.py: /signal now serves the assigned variant's file with the serving-truth head injected (canonical=/signal, index,follow — a variant file's baked noindex can never leak onto the canonical URL; Cache-Control: private, no-store + Vary: Cookie so no cache can pin one arm for everyone). /marketing and /media stay byte-identical passthroughs of their files and log off-protocol exposures. New routes: GET /go/pilot?cta=… (tracked 302 → the variant's own mailto with an [ref A|B|C] — or off-protocol [ref X·] — subject token, registry-only destinations, no open-redirect surface), POST /api/abtest/goal (tier-3 beacon, telemetry-rate-limited), GET /api/abtest/state (per-instance smoke counters + df=2 SRM, with the per-instance caveat printed in the payload). Sitemap: /marketing landing removed (sub-pages stay), /signal lastmod 2026-08-22.
Dark by default (§4 below): MIZOKI_ABTEST_MODE must be exactly on for randomization; unset/anything else serves variant A to everyone with no cookie and no assignment events, while CTA-click and pilot-view instrumentation runs — that is the baseline-collection phase §2 of the spec needs.
Index hygiene (spec §3.2, "retain + canonical + noindex,follow" option): baked into marketing/index.html and media/index.html (canonical + og:url → /signal, noindex,follow). The 301 alternative was rejected: the owner's standing 2026-08-03 directive keeps the parallel sites reachable for side-by-side comparison, and the drift guard requires them served. signal.html flips to index,follow + self-canonical — the file's old comment reserved that flip for "owner-approved launch", which this owner-issued spec is; it still only reaches production via the owner's deploy dispatch.
CTAs (spec §3.3): all 7 bare mailtos replaced with /go/pilot?cta=… (signal-briefing, marketing-{header,hero,pilot,final}, media-{final,footer}). Each keeps its page's original destination address and subject; the signal CTA (previously subject-less) gains "MIZ OKI Executive Briefing". The [ref] token closes §1.2's attribution gap for inquiries that arrive by email. Three-tier outcomes: inquiry ([ref] count in the destination inboxes) · cta_click events · goal events from a byte-identical inline beacon on all three landings (IntersectionObserver + sendBeacon; deterministic — no Math.random, no Date.now, honoring the /media determinism rule).
Pre-registration (spec §3.4): # MIZ OKI 3.5/docs/marketing/abtest-preregistration-signal-landing.md — outcomes with operational definitions, ITT + bot + off-protocol exclusions, power table (labeled illustrative scenario pending real baselines), fixed end condition, no interim analysis, omnibus-then-Holm multiple-comparison handling, SRM plan, privacy posture, and a hard activation checklist with owner-filled fields. Status: REGISTERED — NOT COLLECTING.
SRM + power tooling (spec §2/§5): scripts/abtest_stats.py (stdlib; power, srm — inline counts, JSONL, or gcloud logging read --format=json — and a both-directions selftest). SRM verdict on alarm: stop, discard, fix assignment — never reweight. Verified against the spec's own arithmetic: 2%→3% ⇒ 3,826/arm; 8%→11% ⇒ 1,499/arm; 25%→32% ⇒ 652/arm.
Tests: new tests/test_abtest.py (34) covering dark/live modes, the noindex-leak rule, stickiness, bot no-cookie, event-log privacy shape, tracked-CTA both directions (page→tracker AND tracker→approved mailto), open-redirect refusal, beacon parity across the three files, and registry⇔page CTA equivalence. Six existing pins updated with their intent kept and the reason in-file (marketing canonical/allowed-destinations, self-containment's single declared beacon, two sitemap tests, media metadata + CTA, two /signal noindex pins).

3. Verification (all measured in this session, on this branch)

Site suite in a fresh venv (Flask==3.1.3 + markdown): 560 tests, 0 failures, 1 pre-existing skip (was 544/9F/2E before the deliberate pin updates; the 2 errors were the venv missing the markdown dev dependency, present in CI's generator step).
Deploy gates exactly as deploy-homepage.yml runs them: check_design_canon.py 20/20 (no canon-pinned file touched — no re-pin needed); content_qa.py --self-test fires + 59 scoped files clean (edited signal.html and marketing/index.html are in scope); check_marketing_surfaces.py OK (all markers preserved).
scripts/abtest_stats.py selftest 11/11 (chi-square reference values, power references, SRM fires on a seeded broken split and stays quiet on a fair one, log-ingestion filters).
Rule-03 pre-merge greps V1/V2/V3 clean on the diff; skills_sync --check + ontology_skills_sync --check green; claude_memory.py check --strict valid.

4. Arming path (deliberately not done here)

Owner dispatches a deploy of this merge (APPROVED) → index hygiene + instrumentation go live, test still dark; baseline rates start accruing in Cloud Logging.
After ≥14 days: fill the pre-registration's §6 fields from measured baselines (abtest_stats.py power); if the arithmetic says the test can't resolve in the pages' stable lifetime, don't run it (spec §7) — the instrumentation still ends the unmeasurable-CTA defect.
To arm: add MIZOKI_ABTEST_MODE=on to the workflow's --set-env-vars list (protected path ⇒ human-review PR; the list REPLACES, so a hand-set flag would be wiped by the next dispatch) and dispatch again. Weekly SRM per the pre-registration.

P-5 CLOSED 2026-09-15 as superseded by docs/product/ABC_NOT_RUN_DECLARATION.md (owner ruling 2026-09-15, docs/reports/OWNER_RULINGS_2026-09-15_REGISTER_CLOSEOUT.md, P-5): the test was declared not run by absence of evidence, so there is no look-and-approve left to take. The dark instrumentation ships with whichever homepage dispatch comes next; MIZOKI_ABTEST_MODE stays unset.

5. Owner follow-ups surfaced (not in this change's scope)

privacy.html is canon-pinned — when arming, add a functional-cookie (mv) disclosure line there (owner instruction + same-commit canon re-pin required by AGENTS 7.1).
The mirror repo (mizoki-3-5-website) parity port carries these same surfaces when the owner next runs the parity flow — parity is a union merge, never a byte-copy (site CLAUDE.md rule).
If a CDN/LB cache layer is ever enabled in front of mizoki-website, confirm it honors Cache-Control: private, no-store on /signal before arming — a cached arm is an SRM incident.
GET /api/abtest/state is a deliberate public read-only surface (mode, per-instance counts, SRM p) — in the spirit of the spec's §6 transparency suggestion; contains no secrets or PII. Publishing the pre-registration itself (spec §6) is an owner option: copy it under repo docs/ and it appears on the /docs portal at the next dispatch.

No figure in this report upgrades any claim label; the power numbers are illustrative scenarios until the pre-registration's §6 holds measured values (TRUTH.md).