Secret Management Guide

Version: 1.0
Last Updated: 2026-01-10
Classification: Internal Use

Overview
Google Secret Manager Setup
Creating Secrets
Cloud Run Integration
Local Development Setup
Secret Rotation Procedures
Troubleshooting
Best Practices

Overview

MIZ OKI 3.0 uses Google Secret Manager for all production secrets. This ensures:

✅ Secrets are never committed to version control
✅ Centralized secret management with audit logging
✅ Automatic secret rotation support
✅ IAM-based access control
✅ Encrypted at rest and in transit

NEVER commit actual secrets to the repository. Use .env.example templates for documentation only.

Google Secret Manager Setup

Prerequisites

Google Cloud SDK installed and configured
Project with Secret Manager API enabled
Appropriate IAM permissions

Enable Secret Manager API

# Enable the Secret Manager API for your project
gcloud services enable secretmanager.googleapis.com \
  --project=YOUR_PROJECT_ID

Grant Permissions

Grant your Cloud Build service account access to Secret Manager:

# Get your project number
PROJECT_NUMBER=$(gcloud projects describe YOUR_PROJECT_ID --format="value(projectNumber)")

# Grant Secret Manager access to Cloud Build
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:${PROJECT_NUMBER}@cloudbuild.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"

# Grant Secret Manager access to Cloud Run (default compute service account)
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"

Creating Secrets

Required Secrets

The following secrets must be created for MIZ OKI 3.0:

Secret Name	Purpose	Example Value
`anthropic-api-key`	Anthropic Claude API	`sk-ant-api03-...`
`gemini-api-key`	Google Gemini API	`AIza...`
`jwt-secret`	JWT token signing	Strong random string (32+ chars)
~~`neo4j-password`~~	~~Neo4j database~~ — RETIRED 2026-08-09, do not create. Neo4j is not being re-provisioned; Firestore is the knowledge-graph backend	—
`redis-password`	Redis cache	Strong password
`miz-oki-unified-auth`	Service-to-service auth	Strong random token
`openai-api-key`	OpenAI API (optional)	`sk-...`
`xai-api-key`	xAI Grok API (optional)	`xai-...`

Creating Secrets - Method 1: From File

# Create secret from file
echo -n "your-secret-value-here" > /tmp/secret.txt
gcloud secrets create SECRET_NAME \
  --data-file=/tmp/secret.txt \
  --replication-policy="automatic" \
  --project=YOUR_PROJECT_ID

# Clean up
rm /tmp/secret.txt

Creating Secrets - Method 2: From stdin

# Create secret from stdin (preferred - no file artifacts)
echo -n "your-secret-value-here" | \
  gcloud secrets create SECRET_NAME \
  --data-file=- \
  --replication-policy="automatic" \
  --project=YOUR_PROJECT_ID

Example: Creating All Required Secrets

# Set your project ID
export PROJECT_ID="your-gcp-project-id"

# Anthropic API Key
echo -n "sk-ant-api03-YOUR-ACTUAL-KEY-HERE" | \
  gcloud secrets create anthropic-api-key \
  --data-file=- \
  --replication-policy="automatic" \
  --project=${PROJECT_ID}

# Gemini API Key
echo -n "YOUR-ACTUAL-GEMINI-KEY-HERE" | \
  gcloud secrets create gemini-api-key \
  --data-file=- \
  --replication-policy="automatic" \
  --project=${PROJECT_ID}

# JWT Secret (generate strong random value)
python3 -c "import secrets; print(secrets.token_urlsafe(32), end='')" | \
  gcloud secrets create jwt-secret \
  --data-file=- \
  --replication-policy="automatic" \
  --project=${PROJECT_ID}

# Neo4j Password — RETIRED 2026-08-09. DO NOT RUN.
# Owner decision: Neo4j is not being re-provisioned; Firestore is the
# knowledge-graph backend and the Aura host in `neo4j-uri` is NXDOMAIN by
# choice. Creating this secret wires nothing to anything. Retained only so the
# shape is on record. See docs/lii/RUNBOOK.md, section 1 "Cell 35 graph backend".
#
# echo -n "YOUR-STRONG-NEO4J-PASSWORD-HERE" | \
#   gcloud secrets create neo4j-password \
#   --data-file=- \
#   --replication-policy="automatic" \
#   --project=${PROJECT_ID}

# Redis Password
echo -n "YOUR-STRONG-REDIS-PASSWORD-HERE" | \
  gcloud secrets create redis-password \
  --data-file=- \
  --replication-policy="automatic" \
  --project=${PROJECT_ID}

# Unified Auth Token (generate strong random value)
python3 -c "import secrets; print(secrets.token_urlsafe(32), end='')" | \
  gcloud secrets create miz-oki-unified-auth \
  --data-file=- \
  --replication-policy="automatic" \
  --project=${PROJECT_ID}

Verify Secrets

# List all secrets
gcloud secrets list --project=${PROJECT_ID}

# View secret metadata (not the actual value)
gcloud secrets describe SECRET_NAME --project=${PROJECT_ID}

# Access secret value (requires secretAccessor role)
gcloud secrets versions access latest \
  --secret=SECRET_NAME \
  --project=${PROJECT_ID}

Cloud Run Integration

Method 1: Using --set-secrets Flag

Reference secrets directly in gcloud run deploy command:

gcloud run deploy SERVICE_NAME \
  --image=IMAGE_URL \
  --region=us-central1 \
  --set-secrets="ANTHROPIC_API_KEY=anthropic-api-key:latest,GEMINI_API_KEY=gemini-api-key:latest,JWT_SECRET=jwt-secret:latest"

Method 2: Using YAML Manifest

Reference secrets in Cloud Run service YAML:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: my-service
spec:
  template:
    spec:
      containers:
      - image: gcr.io/project/image
        env:
        # Regular environment variables
        - name: ENVIRONMENT
          value: production
        - name: LOG_LEVEL
          value: info
        # Secrets from Secret Manager
        - name: ANTHROPIC_API_KEY
          valueFrom:
            secretKeyRef:
              name: anthropic-api-key
              key: latest
        - name: GEMINI_API_KEY
          valueFrom:
            secretKeyRef:
              name: gemini-api-key
              key: latest
        - name: JWT_SECRET
          valueFrom:
            secretKeyRef:
              name: jwt-secret
              key: latest

Deploy the manifest:

gcloud run services replace service.yaml --region=us-central1

Example: Complete cloudbuild.yaml

steps:
  # Build Docker image
  - name: 'gcr.io/cloud-builders/docker'
    args: ['build', '-t', 'gcr.io/$PROJECT_ID/my-service', '.']

  # Push to Container Registry
  - name: 'gcr.io/cloud-builders/docker'
    args: ['push', 'gcr.io/$PROJECT_ID/my-service']

  # Deploy to Cloud Run with secrets
  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
    entrypoint: gcloud
    args:
      - 'run'
      - 'deploy'
      - 'my-service'
      - '--image=gcr.io/$PROJECT_ID/my-service'
      - '--region=us-central1'
      - '--platform=managed'
      - '--set-env-vars=ENVIRONMENT=production,LOG_LEVEL=info'
      - '--set-secrets=ANTHROPIC_API_KEY=anthropic-api-key:latest,GEMINI_API_KEY=gemini-api-key:latest,JWT_SECRET=jwt-secret:latest'

images:
  - 'gcr.io/$PROJECT_ID/my-service'

Local Development Setup

Step 1: Copy Template Files

# Root environment
cp .env.example .env

# Services environment
cp services/.env.example services/.env

# EKIS service
cp services/ekis/.env.example services/ekis/.env

Step 2: Fill in Local Values

Edit each .env file and replace placeholder values:

# .env
ANTHROPIC_API_KEY=sk-ant-api03-YOUR-DEV-KEY
GEMINI_API_KEY=YOUR-DEV-GEMINI-KEY
GCP_PROJECT_ID=your-dev-project-id

Step 3: Never Commit .env Files

The .gitignore is configured to exclude all .env files. Verify:

# This should show no .env files (only .env.example files)
git status

Using Google Secret Manager Locally

For local development that needs production secrets:

# Authenticate with gcloud
gcloud auth application-default login

# Set project
gcloud config set project YOUR_PROJECT_ID

# Access secrets programmatically
gcloud secrets versions access latest --secret=anthropic-api-key

Or use the Secret Manager client library in your code:

from google.cloud import secretmanager

def access_secret(project_id: str, secret_id: str, version: str = "latest") -> str:
    """Access secret from Secret Manager"""
    client = secretmanager.SecretManagerServiceClient()
    name = f"projects/{project_id}/secrets/{secret_id}/versions/{version}"
    response = client.access_secret_version(request={"name": name})
    return response.payload.data.decode("UTF-8")

# Usage
api_key = access_secret("your-project-id", "anthropic-api-key")

Secret Rotation Procedures

When to Rotate Secrets

Quarterly (every 3 months) - routine rotation
Immediately - if a secret is compromised or exposed
On team member changes - when someone with access leaves
After security audit - as recommended by security review

Rotation Process

Step 1: Create New Secret Version

# Add new version to existing secret
echo -n "new-secret-value-here" | \
  gcloud secrets versions add SECRET_NAME \
  --data-file=- \
  --project=${PROJECT_ID}

Step 2: Update Services

Cloud Run services using secret:latest will automatically pick up the new version on next deployment or restart.

To force immediate update:

# Redeploy service (picks up latest secret version)
gcloud run services update SERVICE_NAME \
  --region=us-central1 \
  --project=${PROJECT_ID}

# Or update the service to trigger a new revision
gcloud run services update SERVICE_NAME \
  --update-env-vars=FORCE_UPDATE=$(date +%s) \
  --region=us-central1 \
  --project=${PROJECT_ID}

Step 3: Verify New Secret

# Check service is using new secret
gcloud run services describe SERVICE_NAME \
  --region=us-central1 \
  --format="value(spec.template.spec.containers[0].env)" \
  --project=${PROJECT_ID}

# Test the service endpoint
curl https://SERVICE_URL/health

Step 4: Disable Old Secret Version

# List all versions
gcloud secrets versions list SECRET_NAME --project=${PROJECT_ID}

# Disable old version (keeps it for rollback if needed)
gcloud secrets versions disable VERSION_NUMBER \
  --secret=SECRET_NAME \
  --project=${PROJECT_ID}

Step 5: Destroy Old Version (After Confirmation)

After confirming new secret works (wait 24-48 hours):

# Permanently destroy old version
gcloud secrets versions destroy VERSION_NUMBER \
  --secret=SECRET_NAME \
  --project=${PROJECT_ID}

Automated Rotation Script

#!/bin/bash
# rotate-secret.sh - Automated secret rotation

set -e

SECRET_NAME=$1
NEW_VALUE=$2
PROJECT_ID=${3:-$(gcloud config get-value project)}

if [ -z "$SECRET_NAME" ] || [ -z "$NEW_VALUE" ]; then
  echo "Usage: ./rotate-secret.sh SECRET_NAME NEW_VALUE [PROJECT_ID]"
  exit 1
fi

echo "Rotating secret: $SECRET_NAME"

# Create new version
echo -n "$NEW_VALUE" | \
  gcloud secrets versions add $SECRET_NAME \
  --data-file=- \
  --project=$PROJECT_ID

# Get all services using this secret
SERVICES=$(gcloud run services list \
  --platform=managed \
  --format="value(metadata.name)" \
  --project=$PROJECT_ID)

# Update each service
for SERVICE in $SERVICES; do
  echo "Updating service: $SERVICE"
  REGION=$(gcloud run services describe $SERVICE \
    --platform=managed \
    --format="value(metadata.labels.'cloud.googleapis.com/location')" \
    --project=$PROJECT_ID)

  gcloud run services update $SERVICE \
    --update-env-vars=LAST_ROTATED=$(date +%s) \
    --region=$REGION \
    --project=$PROJECT_ID \
    --quiet
done

echo "Secret rotation complete!"

Usage:

chmod +x rotate-secret.sh
./rotate-secret.sh anthropic-api-key "new-key-value-here"

Troubleshooting

Secret Not Found

Error: Secret [SECRET_NAME] not found

Solution:

# Create the secret
echo -n "your-value" | gcloud secrets create SECRET_NAME --data-file=-

# Verify it exists
gcloud secrets list | grep SECRET_NAME

Permission Denied

Error: Permission denied on secret

Solution:

# Grant access to service account
gcloud secrets add-iam-policy-binding SECRET_NAME \
  --member="serviceAccount:SERVICE_ACCOUNT_EMAIL" \
  --role="roles/secretmanager.secretAccessor"

Service Can't Access Secret

Error: Service logs show secret access errors

Solution:

# Check service account has access
gcloud secrets get-iam-policy SECRET_NAME

# Grant access if missing
PROJECT_NUMBER=$(gcloud projects describe PROJECT_ID --format="value(projectNumber)")
gcloud secrets add-iam-policy-binding SECRET_NAME \
  --member="serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"

# Redeploy service
gcloud run services update SERVICE_NAME --region=REGION

Old Secret Version Still Being Used

Solution:

# Force service to update
gcloud run services update SERVICE_NAME \
  --update-env-vars=FORCE_UPDATE=$(date +%s) \
  --region=REGION

# Or use specific version instead of 'latest'
gcloud run services update SERVICE_NAME \
  --update-secrets=SECRET_VAR=secret-name:VERSION_NUMBER \
  --region=REGION

Best Practices

1. Never Commit Secrets

✅ Use .env.example templates
✅ Add .env files to .gitignore
✅ Use Secret Manager for production
❌ Never commit .env files
❌ Never hardcode secrets in code

2. Use Strong Secrets

# Generate strong random secrets
import secrets

# For tokens (32 bytes = 256 bits)
token = secrets.token_urlsafe(32)

# For passwords (16+ chars with special characters)
password = secrets.token_urlsafe(24)

3. Rotate Regularly

Set calendar reminders for quarterly rotation
Document rotation in change log
Test after rotation before disabling old version

4. Audit Access

# Check who has access to secrets
gcloud secrets get-iam-policy SECRET_NAME

# Review audit logs
gcloud logging read "resource.type=secretmanager.googleapis.com/Secret" \
  --limit=50 \
  --format=json

5. Use Least Privilege

Grant secretmanager.secretAccessor only to services that need it
Use separate secrets for different environments
Don't reuse production secrets in development

6. Monitor Secret Usage

Set up alerts for: - Secret access failures - Unusual access patterns - Secret modifications

# Example alert policy (Cloud Monitoring)
gcloud alpha monitoring policies create \
  --notification-channels=CHANNEL_ID \
  --display-name="Secret Access Failures" \
  --condition-display-name="High error rate" \
  --condition-threshold-value=5 \
  --condition-threshold-duration=300s

7. Document Secret Purpose

Maintain a secret inventory:

| Secret Name | Purpose | Owner | Rotation Schedule | Last Rotated |
|-------------|---------|-------|-------------------|--------------|
| anthropic-api-key | Claude API access | Platform Team | Quarterly | 2026-01-10 |
| gemini-api-key | Gemini API access | Platform Team | Quarterly | 2026-01-10 |
| jwt-secret | Token signing | Auth Team | Quarterly | 2026-01-10 |

8. Handle Secret Exposure

If a secret is accidentally exposed:

Immediately rotate the secret in Secret Manager
Immediately revoke/disable the secret in the upstream provider (Anthropic, Google, etc.)
Redeploy all affected services
Review audit logs to assess impact
Update incident response documentation

Additional Resources

Questions or Issues?

Contact the Platform Security Team or file an issue in the repository.

Secret Management Guide

Table of Contents

Overview

Google Secret Manager Setup

Prerequisites

Enable Secret Manager API

Grant Permissions

Creating Secrets

Required Secrets

Creating Secrets - Method 1: From File

Creating Secrets - Method 2: From stdin

Example: Creating All Required Secrets

Verify Secrets

Cloud Run Integration

Method 1: Using --set-secrets Flag

Method 2: Using YAML Manifest

Example: Complete cloudbuild.yaml

Local Development Setup

Step 1: Copy Template Files

Step 2: Fill in Local Values

Step 3: Never Commit .env Files

Using Google Secret Manager Locally

Secret Rotation Procedures

When to Rotate Secrets

Rotation Process

Step 1: Create New Secret Version

Step 2: Update Services

Step 3: Verify New Secret

Step 4: Disable Old Secret Version

Step 5: Destroy Old Version (After Confirmation)

Automated Rotation Script

Troubleshooting

Secret Not Found

Permission Denied

Service Can't Access Secret

Old Secret Version Still Being Used

Best Practices

1. Never Commit Secrets

2. Use Strong Secrets

3. Rotate Regularly

4. Audit Access

5. Use Least Privilege

6. Monitor Secret Usage

7. Document Secret Purpose

8. Handle Secret Exposure

Additional Resources