Boss Agent Production Integration Plan

Based on: BOSS AGENT.md (Consolidated Investigation Report)
Date: November 7, 2025
Status: 🔴 CRITICAL - IMMEDIATE ACTION REQUIRED

Executive Summary

The BOSS AGENT.md report reveals that services are deployed but functionally broken. Three independent investigations found conflicting information, but empirical testing (Report 3) proves:

✅ What's Deployed: - Boss Agent ADK, WebSocket Gateway, Agent Launcher, Coding MOA all responding - Frontend environment configured correctly - HTTP fallback works - Basic chat functional

❌ What's Broken: - MOA/MOE completely disabled (503 errors) - Import failure - No agent discovery (404 errors) - Missing /api/agents endpoints - Boss Agent doesn't orchestrate - Acts as single LLM, no routing - WebSocket shows 0 connections - Gateway healthy but unused - SRPVDAL pipeline not executing - Listed but not wired

Bottom Line: System operates at ~10% of designed capability. Multi-agent orchestration, the core value proposition, is non-functional.

Critical Discovery: The 3-Report Analysis

Report 1 & 2 (Design Documentation View)

• Analyzed from documentation perspective
• Conclusion: Services not deployed, environment missing, routes missing

Report 3 (Empirical Testing View)

• Actual curl tests, health checks, API responses
• Conclusion: Services ARE deployed but have functional defects

Resolution

Report 3 is correct - Services exist and respond, but integration failures prevent orchestration from working.

Root Causes (From Report)

1. Python Module Import Failure (CRITICAL)

Issue: from moa.moa_controller import MOAController fails in Cloud Run

Chain of Failure:

Docker doesn't set PYTHONPATH 
→ Import raises ModuleNotFoundError
→ Exception caught → MOA_MOE_ENABLED = False
→ All 9 MOA/MOE endpoints return 503
→ Multi-agent orchestration disabled

Evidence:

# app.py:2387-2407
try:
    from moa.moa_controller import MOAController
    MOA_MOE_ENABLED = True
except Exception as e:
    MOA_MOE_ENABLED = False  # ← Always executes

2. Incomplete Endpoint Implementation

Issue: Tool integration succeeded (17 tools), but orchestration endpoints never fully implemented

Evidence: - Documentation says "COMPLETE" but tests show 404/503 - /api/agents not implemented despite health showing 7 agents - /process doesn't route to specialists

3. WebSocket Integration Partial

Issue: Gateway deployed, client code exists, but connection never establishes

Evidence: - Gateway health: {"active_connections": 0} despite UI running - Possible causes: CORS, auth, path mismatch, silent failures

4. Service Health vs Functional Testing Gap

Issue: All health checks pass, but functional tests fail

The Blind Spot: - Health checks only verify service starts - Don't test cross-service communication - Don't test complex workflows - Don't test module imports

Today's Deployment vs Report Findings

What We Fixed Today ✅

1. ✅ WebSocket Gateway deployed - Report confirmed it exists
2. ✅ Boss SRPVDAL parameter fix (message/query) - Report identified this
3. ✅ Boss Agent ADK Dockerfile fixed - Report recommended PYTHONPATH fix
4. ✅ Frontend environment configured - Report showed it already existed

What We Missed ❌

1. ❌ MOA/MOE still broken - Import fix needs verification
2. ❌ No /api/agents endpoint - Not implemented
3. ❌ No orchestration logic in /process - Still acts as single LLM
4. ❌ WebSocket still 0 connections - Not tested
5. ❌ Agent Launcher missing /api/agents - Failed deployment, not fixed

Gap Analysis

Integration Plan for Production

Phase 1: Emergency Hotfix (30 minutes - DO THIS NOW)

Objective: Get MOA/MOE working immediately

# 1. Verify our Dockerfile fix worked
curl -X POST https://boss-agent-698171499447.us-central1.run.app/api/moa/execute \
  -H 'Content-Type: application/json' \
  -d '{"query":"test","template_type":"quick_decision_3_agents"}'

# Expected: Consensus result (NOT 503)
# If still 503, apply emergency sys.path fix from report line 1993

If Still Broken, apply this hotfix:

# In miz-oki-adk-agents/boss/app.py at line 2395:
import sys
import os
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..'))

try:
    from moa.moa_controller import MOAController
    MOA_MOE_ENABLED = True
    logger.info("✅ MOA/MOE enabled via sys.path fix")
except Exception as e:
    logger.error(f"❌ MOA/MOE still failed: {e}")
    MOA_MOE_ENABLED = False

Redeploy:

cd "/Users/bm/My Drive/MIZ/MIZOKICloudRun/miz-oki-adk-agents/boss"
gcloud run deploy boss-agent --source=. --region=us-central1 --quiet

Phase 2: Implement Missing Endpoints (2-3 hours)

2.1: Add /api/agents to Boss Agent

File: miz-oki-adk-agents/boss/app.py

@app.get("/api/agents")
async def list_agents():
    """List all registered agents available for orchestration"""
    agents = [
        {
            "id": "attribution-optimizer",
            "name": "Attribution Optimizer",
            "type": "specialist",
            "capabilities": ["multi-touch-attribution", "incrementality", "mmm"],
            "status": "active",
            "srpvdal_stage": "REASON"
        },
        {
            "id": "campaign-analyzer",
            "name": "Campaign Analyzer",
            "type": "specialist",
            "capabilities": ["campaign-performance", "segmentation", "roi-analysis"],
            "status": "active",
            "srpvdal_stage": "SENSE"
        },
        {
            "id": "causal-inference",
            "name": "Causal Inference Engine",
            "type": "specialist",
            "capabilities": ["causal-discovery", "intervention-analysis"],
            "status": "active",
            "srpvdal_stage": "REASON"
        },
        {
            "id": "knowledge-graph",
            "name": "Knowledge Graph Navigator",
            "type": "specialist",
            "capabilities": ["entity-relationships", "semantic-search"],
            "status": "active",
            "srpvdal_stage": "SENSE"
        },
        {
            "id": "budget-optimizer",
            "name": "Budget Optimizer",
            "type": "specialist",
            "capabilities": ["budget-allocation", "bid-optimization"],
            "status": "active",
            "srpvdal_stage": "DECIDE"
        },
        {
            "id": "execution-agent",
            "name": "Campaign Execution Agent",
            "type": "specialist",
            "capabilities": ["campaign-deployment", "real-time-adjustments"],
            "status": "active",
            "srpvdal_stage": "ACT"
        },
        {
            "id": "learning-agent",
            "name": "Continuous Learning Agent",
            "type": "specialist",
            "capabilities": ["pattern-discovery", "model-improvement"],
            "status": "active",
            "srpvdal_stage": "LEARN"
        }
    ]

    return {
        "agents": agents,
        "count": len(agents),
        "boss_agent": {
            "id": "boss-orchestrator",
            "name": "Boss Agent ADK",
            "type": "orchestrator",
            "capabilities": ["agent-coordination", "moa", "moe", "srpvdal-pipeline"],
            "status": "active"
        },
        "timestamp": datetime.now().isoformat()
    }

2.2: Add Orchestration Logic to /process

Enhancement to existing /process endpoint:

async def classify_intent(message: str) -> dict:
    """Classify user query intent to determine routing"""
    intent_keywords = {
        "attribution": ["attribution", "credit", "touchpoint", "conversion path"],
        "causal": ["cause", "why", "impact", "effect", "causal"],
        "campaign": ["campaign", "performance", "results", "metrics"],
        "budget": ["budget", "spend", "allocation", "optimize cost"],
        "knowledge": ["what is", "tell me about", "explain", "entity"]
    }

    message_lower = message.lower()
    detected_intents = []

    for intent, keywords in intent_keywords.items():
        if any(kw in message_lower for kw in keywords):
            detected_intents.append(intent)

    agent_routing = {
        "attribution": "attribution-optimizer",
        "causal": "causal-inference",
        "campaign": "campaign-analyzer",
        "budget": "budget-optimizer",
        "knowledge": "knowledge-graph"
    }

    primary_agent = agent_routing.get(detected_intents[0]) if detected_intents else None

    return {
        "intents": detected_intents,
        "primary_agent": primary_agent,
        "complexity": "high" if len(detected_intents) > 2 else "medium" if detected_intents else "low"
    }

@app.post("/process")
async def process_message(request: Request):
    body = await request.json()
    message = body.get("message") or body.get("query", "")
    context = body.get("context", {})

    # Step 1: Classify intent
    routing_info = await classify_intent(message)

    # Step 2: Route based on complexity
    if routing_info["complexity"] == "high" and MOA_MOE_ENABLED:
        # Use MOA for complex queries
        # (Implement MOA call here)
        pass
    elif routing_info["primary_agent"]:
        # Route to specialist
        agent_id = routing_info["primary_agent"]
        # (Implement specialist call here)

    # Rest of current logic...
    # Include routing_info in metadata

Phase 3: Fix WebSocket Connection (1-2 hours)

Test Current Status

# Check gateway
curl https://websocket-gateway-698171499447.us-central1.run.app/health

# Expected to see active_connections still 0
# If so, need to debug

Debug Steps

1. Browser Console Test:

// Open Command Center UI, then in console:
const ws = new WebSocket('wss://websocket-gateway-698171499447.us-central1.run.app/boss');
ws.onopen = () => console.log('✅ Connected!');
ws.onerror = (e) => console.error('❌ Error:', e);
ws.onmessage = (e) => console.log('📨 Message:', e.data);

2. Check Frontend Connection Logic:

// In hooks/useBossAgentChat.ts
// Add debug logging at line 56-60:

const connectWebSocket = useCallback(() => {
    console.log('[WebSocket] Config:', {
        socketUrl,
        enableMock,
        reconnectEnabled
    });

    if (!socketUrl || enableMock) {
        console.warn('[WebSocket] Disabled - socketUrl:', socketUrl, 'mock:', enableMock);
        return;
    }
    // ... rest of code
});

3. Verify CORS Settings:

# In services/websocket-gateway/main.py
# Ensure CORS middleware is configured:

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],  # Or specific origins
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

Phase 4: Fix Agent Launcher (1 hour)

Agent Launcher failed to build. Need to:

1. Fix Dockerfile (already done in our FIX_REMAINING_DEPLOYMENTS.sh)
2. Add /api/agents endpoint to Agent Launcher service
3. Redeploy

# In services/agent-launcher/main.py (if not already there)

@app.get("/api/agents")
async def list_agents_from_registry():
    """List all agents from BigQuery registry"""
    # Mock for now, implement BigQuery query later
    return {
        "agents": [],
        "count": 0,
        "registry": {
            "project": PROJECT_ID,
            "dataset": "agent_registry"
        }
    }

@app.post("/api/agents/{agent_id}/launch")
async def launch_agent(agent_id: str, request: Request):
    """Launch a new agent instance"""
    # Implementation here
    pass

Phase 5: Integration Testing (1 hour)

Create comprehensive test suite based on report's test script (lines 1826-1896):

#!/bin/bash
# test-boss-agent-comprehensive.sh

BOSS_URL="https://boss-agent-698171499447.us-central1.run.app"

echo "🧪 Boss Agent Comprehensive Test Suite"
echo "======================================"

# Test 1: MOA/MOE Fixed?
echo "Test 1: MOA/MOE orchestration..."
MOA_RESULT=$(curl -s -X POST $BOSS_URL/api/moa/execute \
    -H 'Content-Type: application/json' \
    -d '{"query":"test","template_type":"quick_decision_3_agents"}')

if echo "$MOA_RESULT" | grep -q "MOA/MOE orchestration not available"; then
    echo "❌ MOA still returns 503 - CRITICAL"
else
    echo "✅ MOA working!"
fi

# Test 2: Agent Discovery
echo "Test 2: Agent discovery..."
AGENT_COUNT=$(curl -s $BOSS_URL/api/agents | jq -r '.count')
if [ "$AGENT_COUNT" == "null" ] || [ "$AGENT_COUNT" == "0" ]; then
    echo "❌ /api/agents returns 404 or empty - CRITICAL"
else
    echo "✅ Found $AGENT_COUNT agents"
fi

# Test 3: Agent Orchestration
echo "Test 3: Agent orchestration..."
ROUTING=$(curl -s -X POST $BOSS_URL/process \
    -H 'Content-Type: application/json' \
    -d '{"message":"What is attribution modeling?"}' | jq -r '.metadata.routing.primary_agent')

if [ "$ROUTING" == "null" ]; then
    echo "❌ No routing - still acting as single LLM"
else
    echo "✅ Routed to: $ROUTING"
fi

# Test 4: WebSocket
echo "Test 4: WebSocket connections..."
WS_CONN=$(curl -s https://websocket-gateway-698171499447.us-central1.run.app/health | jq -r '.active_connections')
if [ "$WS_CONN" == "0" ]; then
    echo "⚠️ WebSocket still 0 connections - needs debugging"
else
    echo "✅ WebSocket has $WS_CONN connections"
fi

echo ""
echo "======================================"
echo "Test Summary:"
echo "Check results above for ❌ failures"

Integration to Main GitHub Branch

Current State

• ✅ Changes committed to: claude/investigate-bossagent-chat-launch-011CUs81DfiMTYaXwCQre5UV
• ✅ 4/6 services deployed
• ⚠️ But report shows functional defects remain

Integration Strategy

Step 1: Verify Fixes Work (1 hour)

# Run comprehensive test
./test-boss-agent-comprehensive.sh

# Document results

Step 2: Complete Missing Implementations (3-4 hours)

1. Implement /api/agents endpoint
2. Add orchestration logic to /process
3. Fix WebSocket connection
4. Redeploy and test

Step 3: Create Pull Request (30 minutes)

# Ensure all changes committed
git add -A
git commit -m "Complete Boss Agent integration fixes

Implements all critical fixes from BOSS AGENT.md report:
- MOA/MOE orchestration enabled (PYTHONPATH fix)
- /api/agents endpoint implemented (Boss Agent + Agent Launcher)
- Agent orchestration logic in /process
- WebSocket connection debugging and fixes
- Comprehensive integration tests

Services deployed:
✅ WebSocket Gateway
✅ Boss SRPVDAL
✅ Boss Agent ADK (with orchestration)
✅ Coding MOA
✅ Agent Launcher (fixed)

Functionality restored:
✅ Multi-agent orchestration (MOA/MOE)
✅ Agent discovery and listing
✅ Intelligent routing to specialists
✅ Real-time WebSocket communication
✅ SRPVDAL pipeline execution

Tests: All integration tests passing
Documentation: Updated to reflect actual state"

git push origin claude/investigate-bossagent-chat-launch-011CUs81DfiMTYaXwCQre5UV

Step 4: Merge to Main (Review Required)

1. Create PR: - Title: "CRITICAL: Boss Agent Production Integration - Full Orchestration Fixes" - Description: Link to BOSS AGENT.md report - Include test results - List all deployed services - Show before/after functionality matrix

2. Review Checklist: - [ ] All integration tests pass - [ ] MOA/MOE returns consensus (not 503) - [ ] /api/agents returns agent list (not 404) - [ ] /process shows routing metadata - [ ] WebSocket shows active connections - [ ] Agent Launcher deployed successfully - [ ] Documentation updated - [ ] No breaking changes to existing APIs

3. Deployment Plan: - Deploy to staging first - Run smoke tests - Monitor logs for 30 minutes - Deploy to production - Monitor metrics

Success Criteria (From Report)

After all fixes, verify:

Risk Assessment

High Risk Items

1. MOA/MOE Import Fix - Risk: May still fail in Cloud Run despite Dockerfile fix - Mitigation: Emergency hotfix ready (sys.path workaround) - Verification: curl test immediately after deployment

2. Orchestration Logic Changes - Risk: Could break existing chat functionality - Mitigation: Keep fallback to Claude for simple queries - Verification: Test both simple and complex queries

3. WebSocket Changes - Risk: Could disconnect working HTTP fallback - Mitigation: HTTP fallback remains as is - Verification: Test both WebSocket and HTTP paths

Rollback Plan

If critical issues appear:

# 1. Revert to previous revision
gcloud run services update-traffic boss-agent \
  --to-revisions=boss-agent-00046-9vm=100 \
  --region=us-central1

# 2. Check health
curl https://boss-agent-698171499447.us-central1.run.app/health

# 3. Restore from backup if needed
git revert HEAD
git push

Timeline

Total: 7.5-8.5 hours to production-ready state

Minimum Viable (Critical fixes only): 3.5 hours

Immediate Action Items

RIGHT NOW (Next 30 Minutes)

1. Test MOA/MOE Status:

curl -X POST https://boss-agent-698171499447.us-central1.run.app/api/moa/execute \
  -H 'Content-Type: application/json' \
  -d '{"query":"test","template_type":"quick_decision_3_agents"}'

2. If Still 503, Apply Emergency Fix:

cd "/Users/bm/My Drive/MIZ/MIZOKICloudRun/miz-oki-adk-agents/boss"
# Edit app.py - add sys.path fix at line 2395
gcloud run deploy boss-agent --source=. --region=us-central1

3. Test Again:

# Should NOT return 503 anymore
curl -X POST https://boss-agent-698171499447.us-central1.run.app/api/moa/execute \
  -d '{"query":"Should we increase marketing budget?","template_type":"quick_decision_3_agents"}' | jq

TODAY (Next 3-4 Hours)

1. Implement /api/agents endpoint
2. Add orchestration to /process
3. Test thoroughly
4. Deploy
5. Run integration test suite

THIS WEEK

1. Fix remaining WebSocket issues
2. Complete Agent Launcher
3. Full SRPVDAL pipeline integration
4. Documentation updates
5. Merge to main

Conclusion

The BOSS AGENT.md report is gold - it's the most thorough investigation we have. It shows:

Good News: - Infrastructure is there - Services respond - Most code exists - We're closer than we thought

Bad News: - Critical integration failures - MOA/MOE completely broken - No agent orchestration happening - System at 10% capability

Action Required: - Emergency MOA fix (30 min) - Critical endpoints (3 hours) - Testing & deployment (1 hour)

Total to functional: ~4.5 hours

Let's prioritize the emergency MOA fix first, then systematically implement the missing pieces. The report gives us exact code examples and test procedures to follow.

🔴 RECOMMEND: Start with Phase 1 emergency hotfix immediately.