virtuoso_models — Independent Verification + Integration Report

Date: 2026-07-08 · Branch: claude/virtuoso-models-integration-fvhjow · Base: origin/main @ 515f609 Re-run everything: python3 -m pytest tests/shared/ -c tests/shared/pytest.ini (37 tests)

Every claim below is backed by a command, diff, or output in this branch's commits. Bold items contradict the package's own documentation or the repo's CLAUDE.md claims.

Step 0 — Package ingest

No virtuoso_models.zip exists anywhere in the environment (the prompt's <FILL IN> path was never provided). The package was already present at both canonical homes, byte-identical to each other (diff -r --exclude=__pycache__ clean):

• src/shared/virtuoso_models/ (shared lib — the path other cells import; confirmed by services/coding-moa/cloudbuild.yaml prepare-shared step and creative-suite-service/Dockerfile:59 COPY src/shared)
• services/virtuoso-models-service/virtuoso_models/ (vendored into the deployed Cloud Run service)

Branch note: the prompt asked for feat/virtuoso-models-integration; this session's binding instructions mandate claude/virtuoso-models-integration-fvhjow. All work is on the latter.

Locked rules, in one paragraph (re-derived from model_registry.py, not the docs): every LLM call resolves through one frozen registry that pins four roles to in-family flagships — DATA_CAUSAL→Google (gemini-3.1-pro-preview base, auto-flipped to gemini-3.5-flash because GEMINI_35_FLASH_IS_GA=True, a deliberate Boss cost/latency override of "capability over cost"), CODING_ARCH→claude-opus-4-8, CREATIVE_MM→gpt-5.5, DEVOPS_OPS→grok-4.3. The in-family lock binds the primary path only; every role cross-vendor fails over to the single global backup claude-opus-4-8, which fires only after a primary failure, is tagged served_by="global_fallback" + primary_error, can be disabled per call (fallback=False re-raises), and may never silently become a primary (assert_fallback_not_primary; CODING_ARCH legitimately shares the string, so its failover is a re-raise). Retired strings are rejected at startup (assert_no_legacy_strings, config-scan and registry-self-audit modes); model strings live only in the registry (get_model(Role.X).model); JourneyEvents get deterministic event_id = sha256(source||type||stable_keys) and an idempotent gate that compare-and-sets on source_payload_hash (insert/update/duplicate); provenance (response_schema_hash, connector_version) is stamped from the live schema, never by hand; the Phase enum is seven-phase SRPVDAL by schema fiat.

Step 1 — Independent code audit

Prior-fix verification (all three real)

NEW bugs found (all proven at runtime before fixing — commit fbb5f52)

Why A/B/C survived earlier audits: py_compile passes (NameError is a runtime error), and a naive AST scan is fooled because call_claude has a local kwargs = dict(...). The package's "verified in isolation" claim cannot have included a single real (or even stubbed) call_gpt/call_grok invocation, nor any virtuoso_call(..., system=...).

Fixes: uniform (messages, system=None, tools=None, **kwargs) adapter signatures; dispatch passes kwargs exactly once; GPT structured output moved to the Responses-API text.format param (old code passed response_format, which is a Chat-Completions param — flagged live-verify); optional params only sent when set; guard re-raises; validate() normalizes to the package exception. Both homes kept byte-identical (enforced by test_vendored_home_matches_lib_home).

Test suite

tests/shared/test_virtuoso_models.py — 33 tests, all passing: 4-role resolution, flip on→off reversibility + REGISTRY integrity, env-override precedence, forbidden-override refusal, legacy guard (raise/pass/self-audit), fallback-not-primary guard, failover tagging, fallback=False re-raise, CODING_ARCH self-loop, regression tests for bugs A–C, deterministic event_id, gate insert→duplicate→updated, all 4 mappers strict-validated (real jsonschema + independent recursive additionalProperties walker), malformed-event negative paths on both validation branches, ingest_gate missing-id refusal, boss_plane surface, vendored parity, and schema_hash pinned to the live deploy (823d6116b33c… — post-fix hash is unchanged, so net schema change vs production is zero).

Step 2 — Repo reality check

Retired/superseded model strings

The full grep found 140 hits across ~34 files (excluding the package itself). This contradicts the repo CLAUDE.md v6.45.10 claim of a completed "MIZ OKI 3.5 Model String Purge." Status table (grouped; full per-line evidence is in the commit diffs):

LLM call-site inventory (full table in the audit transcript; highlights)

• boss_agent_core.py (48k lines) had ZERO virtuoso_models imports and no virtuoso_call usage. This contradicts CLAUDE.md v6.45.10's "Complete codebase-wide wiring to virtuoso_call dispatcher." What is wired: src/services/coding_moa, services/coding-moa, cloud-run-extended-services/ creative-suite-service/orchestrator.py, services/gemini-kg-pipeline extraction engines, and miz-oki-adk-agents/specialists/{creative,email}_personalization.py.
• The boss image never contained shared/ (Dockerfile.v5 copies only boss/ + config/), so shared.model_startup_validator — and every shared.model_registry import in boss modules — could only have failed (they're try/except-guarded, i.e. model startup validation has been silently skipped in prod). Fixed: COPY src/shared/ /app/shared/.
• Legacy google.generativeai SDK (3.x features unsupported) still used by: gemini-kg-pipeline/src/gemini/extraction_engine.py + files_handler.py, cell03-intelligence/gemini_structurer.py (gemini-1.5-pro!), boss/data_transformation_engine.py. Punch list #3.
• Dead PaLM text-bison* call sites: graphrag-integrated/main.py, cell03 GraphRAG ×2, scripts/py_scripts/main.py. Punch list #5.
• OpenAI/xAI SDK calls exist nowhere outside the package — gpt-5.5 and grok-4.3 traffic can only flow through virtuoso_call (which made bugs A/B invisible in production: those paths simply had no callers).

Call-shape conformance (primary direct-SDK sites vs adapter encoding)

SRDAL vs SRPVDAL verdict

The repo is a three-way mix; the package's 7-phase enum matches the newest canonical code but not the Boss execution path.

1. 7-phase, VALIDATE spelling (= package enum, no conflict): contracts/canonical-event-envelope (PHASES, line 48), src/core/srpvdal.py, src/core/srdal.py (misnamed file, 7-phase body), src/cells/google_ads_gaql, plugins/ads-decision/intelligence/srpvdal.py, boss kg_instrumentation_layer_integration.py.
2. 7-phase, VERIFY spelling (CONFLICTS): the whole boss ADC family — srdal_adc.py, srpvdal_plan_verify.py, srpvdal_autonomous_integration.py ("strictly called VERIFY", line 794), semantic_tool_router.py (aliases VALIDATE = "verify"), graph_native_decision_intelligence.py, others. Events these stamp would fail the JourneyEvent srpvdal_phase enum.
3. 5-phase SRDAL (CONFLICTS): src/core/pipeline_state.py, boss_agent_core.py (line ~25730), connector_gateway/srdal.py, customer-journey-system (named SRPVDAL, only 5 stages), command-center backend. Even the repo's own PR template says "Sense -> Reason -> Decide -> Act -> Learn".

No phase handling was changed (per ground rules). Reconciliation needed: either add VERIFY to the schema enum or rename the boss ADC phase to VALIDATE — one edit, both sides together (punch #7).

Step 3 — Integration edits (commits, one per cell/service)

Event-store conflict (flagged, not silently resolved): gemini-kg-pipeline already has its own JourneyEvent system (pydantic contract, dedup_key, Firestore journey_events collection with merge=True — no CAS, no inserted/updated/duplicate distinction) whose schema differs from the package's canonical journey-event.json. The new gate therefore writes to a separate virtuoso_journey_events collection keyed by event_id. Merging the two shapes needs a migration decision (punch #8).

ANTHROPIC_API_KEY coverage (required everywhere virtuoso_call runs — the global fallback needs it)

Step 4 — Live connection proofs

Environment reality: GEMINI_API_KEY / ANTHROPIC_API_KEY / OPENAI_API_KEY / XAI_API_KEY all unset; no GCP ADC (gcloud unconfigured); no vendor SDKs installed; Docker daemon unavailable. Nothing below is simulated-and-called-live.

Step 5 — Deploy readiness

• Full suite post-integration: 37/37 pass (tests/shared/, incl. bridge).
• Container build: BLOCKED (no Docker daemon). Substitute executed: the service's exact image import layout (main.py + vendored package from the service dir) imports clean and its hardened startup guard runs and passes; py_compile clean across every touched file; all edited YAML/JSON parse.
• Not deployed, not merged (as instructed). Note the auto-merge bot has historically merged claude/* branches — if the bot is still active, pushing this branch may effectively merge it; flagging rather than guessing.

Punch list before production traffic (priority order)

1. Mount ANTHROPIC_API_KEY on gemini-kg-pipeline and creative-suite-service (--update-secrets=ANTHROPIC_API_KEY=anthropic-api-key:latest). Until then their virtuoso_call failover path throws instead of failing over.
2. Run the four live per-role calls + the real failover drill (Step 4 items 1-2) with real keys; specifically live-verify (a) gemini-3.5-flash on Vertex location=global incl. causal quality vs 3.1 Pro (registry REVIEW_TRIGGERS item), (b) opus-4-8 effort=xhigh accepted, (c) GPT Responses-API text.format json mode on the installed openai SDK, (d) whether AnthropicVertex in boss accepts the claude-opus-4-8 string or needs a Vertex-dialect model ID.
3. Migrate the 4 legacy google.generativeai files to google-genai (kg-pipeline extraction_engine + files_handler, cell03 gemini_structurer — still pinned gemini-1.5-pro — and boss data_transformation_engine).
4. Convert the remaining direct Vertex GenerativeModel call sites (boss, moa/models, budget_allocator, gemini_stitcher, templates parser) to virtuoso_call where the shape allows; they now source strings from the registry, but bypass failover/tagging/MII capture.
5. Delete or repoint the dead PaLM text-bison* sites (graphrag-integrated, cell03 GraphRAG ×2, scripts) — those models are long retired.
6. Remove the stale tracked drop at .evaluation/incoming/ (pre-fallback semantics; future greps/agents will keep tripping over it).
7. Decide the phase-name reconciliation (schema VALIDATE vs boss ADC VERIFY vs 5-phase SRDAL cluster) and change schema + stampers together.
8. Decide the JourneyEvent store unification: pydantic journey_events (dedup_key) vs canonical virtuoso_journey_events (event_id CAS) — one schema, one collection, one migration.
9. Wire startup guards into the remaining FastAPI cells (google_ads_gaql, cell06, creative-suite) — pattern is now proven in boss + kg-pipeline + the service.
10. Run the E2E Firestore CAS proof from an authorized environment (insert → duplicate → updated → query back) and add pricing verification for opus-4-8/gpt-5.5/grok-4.3 in agent_runtime_ledger.MODEL_COSTS.
11. Feed plane.call() / MII reasoning-trace rows into the mii.reasoning_traces BigQuery sink (WIRING §6) — currently JSONL-local only.

Post-merge reconciliation (2026-07-08, same day)

On push, the repo's auto-merge bot immediately merged this branch into main (a690f26, "theirs strategy") — and a second, concurrent Claude session had been pushing overlapping virtuoso work to main in parallel (35551a8 BigQuery stores.py + boss SENSE gate, b25b9a2 purge incl. the live xai-bridge XAI_MODEL=grok-4.1 deploy config, 25c340b its own REPORT.md — overwritten by this file in the merge). The blind merge produced three regressions, all caught by this branch's test suite / runtime probes and fixed in the follow-up commit:

1. virtuoso_call silently dropped every system prompt and tool — the other session popped system/tools from kwargs; my dispatch forwarded only the remaining kwargs; combined, both were popped then discarded. Caught by test_regression_system_and_tools_kwargs_reach_primary (assert None == 'be terse'). Fixed: explicit single forwarding on both the primary and fallback paths.
2. Boss ingest_single broke on every call — the merge kept the other session's gate call sites (_JOURNEY_MAPPERS / _journey_ingest_gate / self._journey_upsert) but dropped their definitions; proven at runtime (AttributeError: ... has no attribute '_journey_upsert'). Fixed by grafting the reconciled import block + dataclass field + initialize wiring from 35551a8; runtime probe now: ingest_single success: True.
3. Vendored-home drift — stores.py landed only in the lib home; the deployed service image would not have shipped it. Caught by test_vendored_home_matches_lib_home; homes re-synced.

Post-reconciliation: 37/37 tests green on the merged tree.

New punch item (12): stores.py's MERGE writes context_json / provenance_json columns, but the package DDL (schemas/journey_events.bigquery.sql) declares context / provenance STRUCTs — the SQL and the DDL disagree; reconcile before the first live BigQuery ingest. Also note there are now TWO real upsert paths (BigQuery MERGE via boss stores.py, Firestore CAS via gemini-kg-pipeline virtuoso_ingest.py) — complementary, but punch #8 (store unification) now covers three shapes, not two.

Process note: the auto-merge bot merges claude/* on push with no CI gate in between — two concurrent sessions plus "theirs strategy" produced a broken main for ~30 minutes. Recommend requiring the test suite (tests/shared/) as a merge gate for the bot.

Session addendum — 2026-07-08 (branch feat/virtuoso-models-integration)

Independent re-verification pass, run from scratch on main @ adf09545 (post-regression-repair), per the "do not trust prior claims" ground rule. Every result below was re-derived; commands are re-runnable as shown.

Step 0/1 re-verification (independent)

• Package homes byte-identical: diff -rq --exclude=__pycache__ src/shared/virtuoso_models services/virtuoso-models-service/virtuoso_models → exit 0.
• python -m py_compile clean on every module (incl. transforms/).
• AST unbound-name audit (custom walker over every module): clean — the only flagged names are __file__ (implicit module binding, not a bug).
• All three known past fixes confirmed real in current source: vendor_adapters.py:24-26 imports assert_fallback_not_primary and calls it at virtuoso_call entry (line 370); __init__.py exports the full fallback surface; get_model returns dataclasses.replace copies off a frozen=True spec.
• Runtime proofs re-executed (fresh interpreter, stubbed vendors, no network): 4-role resolution; flip ON→OFF→ON reverts with REGISTRY entry untouched; forbidden env override refuses startup (ValueError); fallback-as-primary guard raises (RuntimeError); legacy guard raise/pass/self-audit; failover tags served_by="global_fallback" + primary_error and forwards system; fallback=False re-raises; malformed event → package ValidationError (jsonschema path); Meta event_id deterministic (b6b3f167f4a31c9f…); gate insert→duplicate→updated; all 4 mappers validate strict.
• Full suite: 37/37 passing (python3 -m pytest tests/shared/ -c tests/shared/pytest.ini).

Step 2 stragglers found by widening the grep (NEW this session)

The WIRING §2 grep misses non-retired-but-wrong strings. Widening to all "(gemini|claude|gpt|grok|chatgpt)-…" literals in active LLM call-site files found four bugs prior passes missed — all four contradict the repo's own "purge complete" commit messages:

Not changed, flagged only: local_dev/ mock strings (never call APIs); .evaluation/incoming/ (stale quarantine copy of the package — recommend deleting); grok-2-image (xAI image gen is outside registry scope); data_transformation_engine.py:54 is only the ImportError fallback constant of an already-registry-routed default (correct as-is).

Step 3 secrets audit (ANTHROPIC_API_KEY = global-fallback requirement)

Of the LLM-calling services: boss (cloudbuild.v5.yaml), coding-moa, and claude-gemini-orchestrator have ANTHROPIC_API_KEY. Missing it: services/virtuoso-models-service (registry/ingest only today — needed the moment it proxies calls), services/cell03-intelligence, services/gemini-kg-pipeline (both call Gemini; their global fallback cannot fire without the key). 25 further cell manifests set secrets but not this one — most don't call LLMs; wire it only where virtuoso_call lands.

Step 4 live proofs — what was possible from this VM

No vendor API keys and no GCP ADC exist in this environment (checked env + ~/.config/gcloud: absent). Therefore:

• ✅ Live deployed-service proofs (no key needed): virtuoso-models-service /health serves the four locked assignments + global_fallback=claude-opus-4-8, schema_hash 823d6116… (matches local); /guard?text=grok-4-1-fast-reasoning → HTTP 422; POST /normalize/meta with the test vector returns event_id b6b3f167f4a31c9f… — byte-identical to the locally computed id, proving mapper determinism end-to-end through the deployed image.
• ⛔ BLOCKED — one live virtuoso_call per role (needs GEMINI_API_KEY or Vertex ADC, ANTHROPIC_API_KEY, OPENAI_API_KEY, XAI_API_KEY).
• ⛔ BLOCKED — failover proof with a real broken key (needs a valid ANTHROPIC_API_KEY for the fallback leg). Stubbed-adapter equivalent passes (served_by/primary_error/fallback=False re-raise).
• ⛔ BLOCKED — JourneyEvent end-to-end against real BigQuery/Firestore (needs ADC). The deployed /normalize path is proven; the store leg is covered by injected-client tests only.
• ⛔ BLOCKED — live Gemini structured-output call (needs key/ADC); gemini_response_format() shape verified against the schema locally.

Step 5

• 37/37 tests green post-integration; all edited files py_compile clean.
• ⛔ BLOCKED — container build: no Docker daemon in this VM. Next-best executed: vendored package imports standalone from the service directory (validate_registry() passes), and the two fixed services' import layouts were exercised with their images' PYTHONPATH conventions.
• SRDAL vs SRPVDAL verdict (re-confirmed): the repo pipeline code is seven-phase (src/cells/google_ads_gaql/gaql_cell/pipeline/ ships sense/reason/plan/validate/decide/act/learn modules; boss docs likewise). The package's 7-phase Phase enum matches the repo. No reconciliation needed; the five-phase "SRDAL" wording appears only in prose/README history.

Updated punch list (priority order)

1. Provision the four vendor keys (or Vertex ADC) in a controlled env and run the Step-4 live matrix: one virtuoso_call per role, echo served_by, latency, token counts; Claude effort=xhigh accepted; GPT-5.5 via Responses API; Grok explicit reasoning_effort=high.
2. Failover live proof: poisoned GEMINI_API_KEY in an isolated env → DATA_CAUSAL served by claude-opus-4-8 with primary_error set; then fallback=False re-raise.
3. Reconcile stores.py MERGE columns (context_json/provenance_json) vs DDL (context/provenance STRUCTs) before first live BigQuery ingest.
4. Wire ANTHROPIC_API_KEY into gemini-kg-pipeline, cell03-intelligence, virtuoso-models-service deploy configs (global-fallback prerequisite).
5. Redeploy stale services so the July 6-8 fixes reach Cloud Run (bot-merged commits do not trigger push-filtered deploy workflows; see fleet audit).
6. LIVE-VERIFY the AnthropicVertex model-ID format in claude-gemini-orchestrator (@-suffix vs bare); override via CLAUDE_CODE_MODEL if needed.
7. Vendor the shared lib (or add env pins) into images that can't import it: cell03-intelligence, claude-gemini-orchestrator.
8. Delete .evaluation/incoming/ (stale package copy) once confirmed unused.
9. Migrate google.generativeai (legacy SDK) call sites to google-genai: gemini-kg-pipeline/extraction_engine.py, cell03 gemini_structurer.py, boss gemini_firestore_kg_pipeline.py.
10. Add the tests/shared/ suite as a CI gate on the auto-merge bot.