Issue jgwill/jgtd#131 - COAIA Arena Production Setup Guide

🎯 Overview

This guide provides complete instructions for deploying the COAIA (Creative Oriented AI Agent) arena on a production machine with GPU resources. The arena enables systematic evaluation of COAIA models through anonymous pairwise battles and human preference collection.

📋 Prerequisites

Hardware Requirements

  • GPU: NVIDIA GPU with 8GB+ VRAM (for model inference)
  • RAM: 16GB+ system memory
  • Storage: 50GB+ free disk space
  • Network: Stable internet connection for API access

Software Requirements

  • Operating System: Linux (Ubuntu 20.04+ recommended)
  • Conda/Miniconda: For environment management
  • Git: For repository management
  • Python: 3.8+ (managed through Conda)

API Access

  • OpenAI API Key: For fine-tuned COAIA models
  • Google Cloud Project: With Vertex AI API enabled
  • GCP Service Account: With appropriate permissions

🚀 Quick Deployment

1. Clone Repository

git clone <repository-url>
cd arenas_250706

2. Set Environment Variables

export OPENAI_API_KEY="your-openai-api-key"
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/gcp-credentials.json"

3. Run Automated Deployment

./deploy_coaia_arena.sh start

4. Access Arena

  • Web Interface: http://localhost:7861
  • Battle Mode: Click "Chatbot Arena" tab
  • Models: OpenAI COAIA vs Gemini COAIA

🔧 Manual Setup (Alternative)

Step 1: Create Conda Environment

conda create -n coaia_arena python=3.10 -y
conda activate coaia_arena

Step 2: Install FastChat

cd FastChat
pip install --upgrade pip
pip install -e ".[model_worker,webui]" --timeout=3600

Step 3: Install Cloud Dependencies

pip install google-cloud-aiplatform vertexai

Step 4: Verify Installation

./verify_coaia_arena_setup.sh

Step 5: Start Services

Terminal 1 - Controller:

./run_controller.sh

Terminal 2 - Web Server:

./run_web_server.sh

📊 COAIA Models Configuration

The arena includes the following COAIA models for evaluation:

OpenAI COAIA

  • Model ID: ft:gpt-4.1-nano-2025-04-14:jgwill:coaia250709n1b3:BrWkpMDi
  • Type: Fine-tuned GPT-4.1-nano
  • Training: 54 model iterations with creative orientation focus

Gemini COAIA Models

  • Latest: projects/242650888445/locations/us-central1/models/7008782995188350976
  • Version B: projects/242650888445/locations/us-central1/models/6886059905342504960
  • Manual: projects/242650888445/locations/us-central1/models/6815304133071863808

🎮 Arena Usage

Battle Flow

  1. Navigate: Open http://localhost:7861
  2. Select Mode: Click "Chatbot Arena" tab
  3. Enter Prompt: Type creative orientation challenge
  4. View Responses: Anonymous "Model A" vs "Model B"
  5. Vote: Select better response
  6. Reveal: See OpenAI vs Gemini identity
  7. Repeat: Continue battles for data collection

Evaluation Focus

  • Structural Tension vs gap-thinking patterns
  • Desired Outcomes vs problem-solving approaches
  • Natural Advancement vs oscillating behaviors
  • Creative Process Phases vs inspiration/ideation cycles

📈 Data Collection

Battle Results

  • Stored in FastChat logs: FastChat/battle_logs/
  • ELO ratings automatically calculated
  • Win/loss statistics tracked per model

Conversion to Training Data

python3 arena_to_training_adapter.py --input battle_logs/ --output training_data/

🔍 Troubleshooting

Common Issues

1. Web Interface Not Loading - Check controller is running on port 21001 - Verify web server is running on port 7861 - Check firewall settings

2. Models Not Appearing - Verify api_endpoints.json configuration - Check API keys and credentials - Test model connectivity with verification script

3. GPU Memory Issues - Reduce batch size in model configurations - Monitor GPU usage with nvidia-smi - Consider model quantization if needed

4. Authentication Failures - Verify OpenAI API key validity - Check GCP service account permissions - Ensure Vertex AI API is enabled

Diagnostic Commands

# Check service status
./deploy_coaia_arena.sh status

# View logs
./deploy_coaia_arena.sh logs

# Restart services
./deploy_coaia_arena.sh restart

# Full verification
./verify_coaia_arena_setup.sh

📊 Service Management

Start Arena

./deploy_coaia_arena.sh start

Stop Arena

./deploy_coaia_arena.sh stop

Restart Arena

./deploy_coaia_arena.sh restart

Check Status

./deploy_coaia_arena.sh status

View Live Logs

./deploy_coaia_arena.sh logs

🔧 Configuration Files

Key Files

  • api_endpoints.json: COAIA model definitions
  • deploy_coaia_arena.sh: Automated deployment script
  • run_controller.sh: Controller startup script
  • run_web_server.sh: Web server startup script
  • verify_coaia_arena_setup.sh: Setup verification

Port Configuration

  • Controller: 21001
  • Web Server: 7861
  • Gradio Share: Optional external access

📋 Environment Variables

Required environment variables for production deployment:

# OpenAI Configuration
export OPENAI_API_KEY="your-openai-api-key"

# Google Cloud Configuration
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"
export GOOGLE_CLOUD_PROJECT="top-moment-465521-v9"

# Optional: Custom paths
export COAIA_DIR="/path/to/arena/directory"
export FASTCHAT_PORT="7861"
export CONTROLLER_PORT="21001"

🎯 Success Metrics

Technical Metrics

  • Model Load Success: 100% of COAIA models accessible
  • Battle Completion Rate: >90% of initiated battles completed
  • System Uptime: >95% availability
  • Response Time: <10 seconds for model generation

Quality Metrics

  • Vote Consistency: Human preferences align with creative orientation
  • ELO Differentiation: Clear ranking emerges between approaches
  • Data Quality: Preference votes provide actionable DPO training signal

🚀 Next Steps

Phase 1: Deployment Validation

  1. Complete Setup: Deploy arena on production machine
  2. Verify Models: Test all COAIA model accessibility
  3. Initial Battles: Conduct test battles to verify functionality
  4. Performance Monitoring: Track response times and system resources

Phase 2: Data Collection

  1. Battle Campaign: Systematic human evaluation battles
  2. Preference Analysis: Analyze voting patterns and ELO evolution
  3. Quality Assessment: Validate response quality differences
  4. Data Pipeline: Convert battle results to training datasets

Phase 3: Model Improvement

  1. DPO Training: Use preference data for Direct Preference Optimization
  2. Iterative Refinement: Deploy improved models for further evaluation
  3. Cross-Platform Analysis: Compare OpenAI vs Vertex AI approaches
  4. Production Integration: Deploy superior models in production systems

📞 Support

Documentation

Verification

Run the comprehensive verification script before deployment:

./verify_coaia_arena_setup.sh

This will validate all components and provide detailed setup status.

🎉 Ready for Production: All components tested and validated for production deployment with systematic COAIA model evaluation and human preference collection.

issue #129 - Index

Report abuse