mtgonline/DEVELOPMENT_PLAN.md

MTG Online Backend - Development Plan

Last Updated: 2025-08-27
Status: Active Development
Version: 2.0

---

Executive Summary

This document outlines the development roadmap for the MTG Online backend API. The system is designed to support Magic: The Gathering Online gameplay with a focus on Commander format, while supporting Standard, Modern, and Pauper formats. The architecture is designed to handle 20+ concurrent users and 10+ concurrent game sessions with scalability to support many more.

---

Current Status

Overall Progress: 10-15% complete
Last Verified: 2025-08-27

✅ Completed (Phases 1-4)

• Architecture simplification (unified database config, security, settings)
• Model consolidation (10+ files → 2: models.py, mtg_models.py)
• Router consolidation (54+ files → 37 files)
• Docker packaging (Dockerfile, docker-compose.yml)
• Authentication system (JWT-based login/register)
• Room management (CRUD operations)
• MTGJSON data management (download, refresh, verify)
• Basic WebSocket server (game_server.py - not integrated with DB)

⚠️ Partially Implemented

• Card search (basic structure exists, needs optimization)
• Deck management (stubs only, no real implementation)
• User data (profile endpoints exist, statistics not implemented)

❌ Not Started

• Game engine (all endpoints are mocks/stubs)
• Game models (no MTG OnlineGames, GamePlayer, or GameHistory models)
• Commander deck validation
• User statistics and history
• Real-time game state persistence

---

Project Scope

Core Requirements

• Primary Format: Commander (EDH)
• Secondary Formats: Standard, Modern, Pauper
• Concurrent Users: 20+ (scalable to 100+)
• Concurrent Games: 10+ (scalable to 50+)
• Card Data Source: MTGJSON exclusively
• Technology Stack: Python 3.12, FastAPI, SQLAlchemy, PostgreSQL, Redis

Non-Goals (Out of Scope)

• Frontend development (handled separately)
• Cockatrice client integration
• Third-party tournament management
• Card image hosting (use MTGJSON links)

---

Architecture Overview

Target Architecture

┌─────────────────────────────────────────────────────────────┐
│                      API Gateway (FastAPI)                   │
├─────────────────────────────────────────────────────────────┤
│  Auth │ Users │ Decks │ Cards │ Games │ Rooms │ MTGJSON    │
├─────────────────────────────────────────────────────────────┤
│                     Service Layer                            │
│  GameServer │ CardSearch │ DeckParser │ ImportProcessor     │
├─────────────────────────────────────────────────────────────┤
│                   Data Layer                                 │
│  PostgreSQL (mtgonline + mtg_data) │ Redis (cache/sessions) │
└─────────────────────────────────────────────────────────────┘

Current Implementation:
├── ✅ Authentication (JWT)
├── ✅ Room Management (CRUD)
├── ✅ MTGJSON Data (download/refresh/verify)
├── ⚠️ Card Search (basic)
├── ❌ Game Engine (stubs only)
├── ❌ Deck Management (stubs only)
└── ❌ User Statistics (not implemented)

---

Development Phases

Phase 1: Core Game Engine (Weeks 1-3) ⭐ PRIORITY

Status: Not Started
Current State: All game endpoints are mocks/stubs returning hardcoded data

Objective: Implement functional game management with real-time capabilities

1.1 Game Models & Database

• Create MTG OnlineGames model in app/models/mtg_models.py
  • Game ID, room ID, host ID
  • Game type (Commander, Standard, Modern, Pauper)
  • Game state (waiting, in_progress, finished, aborted)
  • Player slots (max 4 for Commander, configurable)
  • Password protection
  • Timestamps (created, started, finished)
• Create GamePlayer association table
  • Player ID, game ID
  • Seat number, color identity
  • Life total, status (ready, left)
• Create GameHistory table
  • Game ID, participants, winner
  • Game length, result type
  • Replay data reference

1.2 Game Management Endpoints

• Replace mock POST /games/ with real implementation
  • Validate game type and format rules
  • Check room availability
  • Set host as player 1
  • Save to database
  • Return game ID and join link
• Replace mock GET /games/ with real implementation
  • Query database for active games
  • Filter by room, game type
  • Include player count
  • Pagination support
• Replace mock GET /games/{game_id} with real implementation
  • Query database for game details
  • Include player list (without passwords)
  • Game state and settings
• Implement POST /games/{game_id}/join
  • Validate player availability
  • Check if game is full
  • Add player to game in database
• Implement POST /games/{game_id}/leave
  • Remove player from game in database
  • Handle host transfer if needed
• Implement POST /games/{game_id}/start (host only)
  • Validate all players ready
  • Transition to in_progress state
  • Broadcast to all players
• Implement POST /games/{game_id}/end
  • Record results to GameHistory
  • Update player statistics
  • Clean up game state

1.3 WebSocket Game Server Integration

• Integrate existing game_server.py with database
  • Load game state from database on connection
  • Persist game state changes to database
  • Handle reconnection scenarios
• Enhance WebSocket endpoints
  • WS /ws/game/{game_id} - Game connection
  • Real-time player join/leave notifications
  • Game state updates (life totals, turns)
  • Chat/messaging within games
• Game state management
  • Turn tracking
  • Life total updates
  • Game log/events
  • Victory/defeat conditions

1.4 Commander-Specific Logic

• Commander deck validation
  • 100-card minimum (excluding basic lands)
  • Color identity rules
  • Commander uniqueness
  • Ban list enforcement
• Commander game rules
  • Commander damage tracking
  • Command zone management
  • Popularity tracker (optional)

---

Phase 2: Card System (Weeks 4-5)

Status: Not Started

Objective: Complete card search, filtering, and collection tracking

2.1 Card Search Service

• Implement app/services/card_search_service.py
  • Full-text search on card names
  • Filter by: set, type, color, mana cost, rarity
  • Card name autocomplete (debounced)
  • Card detail views with images
• Search optimization
  • PostgreSQL full-text search configuration
  • Redis caching for popular searches
  • Pagination and sorting

2.2 Card Database Integration

• Optimize app/services/card_database.py
  • Production-ready query patterns
  • Connection pooling
  • Batch operations for card imports
• Card collection tracking
  • User card collection per format
  • Card count by set/type
  • Collection statistics

2.3 Card Import System

• Complete app/services/import_batch_processor.py
  • Excel/CSV import support
  • Fuzzy matching with fuzzy_card_matcher.py
  • Validation and error reporting
  • Bulk import with progress tracking
• MTGJSON integration
  • Automatic card data sync
  • Set and card version tracking
  • Image URL management

---

Phase 3: Deck Management (Weeks 6-7)

Status: Not Started

Objective: Full deck CRUD with validation and sharing

3.1 Deck Models

• Create UserDeck model
  • Deck ID, user ID, name
  • Format (Commander, Standard, Modern, Pauper)
  • Deck list (JSON or separate table)
  • Commander/legendary creature reference
  • Privacy settings (public, private, friends-only)
• Create DeckCard model
  • Deck ID, card ID
  • Quantity, sideboard flag
  • Card list ordering

3.2 Deck Endpoints

• POST /decks/ - Create deck
  • Validate format rules
  • Check card legality
  • Save deck with card list
• GET /decks/ - List user decks
  • Filter by format
  • Pagination
• GET /decks/{deck_id} - Get deck details
  • Include card list with quantities
  • Format validation status
• PUT /decks/{deck_id} - Update deck
  • Modify card list
  • Update metadata
• DELETE /decks/{deck_id} - Delete deck
• POST /decks/{deck_id}/share - Generate share link
• GET /decks/shared/{token} - View shared deck

3.3 Deck Parser

• Implement app/services/deck_parser.py
  • Parse MTGO deck list format
  • Extract card names and quantities
  • Validate against format rules
  • Error reporting for invalid decks

3.4 Deck Suggestions

• Implement app/services/deck_suggestion_service.py
  • Meta analysis (most popular decks)
  • Card recommendations based on commander
  • Deck archetypes and templates
  • Counter-pick suggestions

---

Phase 4: User Data & Profiles (Weeks 8-9)

Status: Not Started

Objective: Enhanced user profiles with statistics and history

4.1 User Profile Enhancement

• User statistics
  • Games played (by format)
  • Win/loss record
  • Win rate by format
  • Average game length
  • Commander damage dealt/taken
• Card collection
  • Total cards owned
  • Cards by set/type/color
  • Collection completion percentage
• Deck collection
  • Total decks created
  • Decks by format
  • Most played decks

4.2 User Data Endpoints

• Complete app/routers/user_data_router.py
  • GET /api/v1/user-data/profile - Get profile
  • PUT /api/v1/user-data/profile - Update profile
  • GET /api/v1/user-data/statistics - Get statistics
  • GET /api/v1/user-data/games - Game history
  • GET /api/v1/user-data/collection - Card collection
  • GET /api/v1/user-data/decks - Deck collection
• Data export
  • JSON export of user data
  • CSV export for statistics
  • Privacy controls

4.3 Game History & Replay

• Game history tracking
  • Store game results
  • Track participant performance
  • Format-specific statistics
• Replay data storage
  • Game event log
  • Turn-by-turn actions
  • Save/load replay functionality

---

Phase 5: Infrastructure & Quality (Weeks 10-11)

Status: Not Started

Objective: Testing, documentation, and performance optimization

5.1 Testing

• Unit tests
  • Service layer tests (card search, deck parser, game logic)
  • Model validation tests
  • Utility function tests
• Integration tests
  • API endpoint tests (all routers)
  • Database transaction tests
  • WebSocket connection tests
• Load testing
  • Concurrent user simulation
  • Concurrent game session testing
  • Performance benchmarking
• Test coverage target: 80%+

5.2 Documentation

• API documentation
  • OpenAPI/Swagger auto-generation (already enabled)
  • Endpoint examples
  • Authentication guide
• Architecture documentation
  • System design diagrams
  • Data flow diagrams
  • Service interaction docs
• Deployment guide
  • Docker setup instructions
  • Environment configuration
  • Database migration steps
• Contributing guide
  • Code style guidelines
  • PR process
  • Issue reporting

5.3 Performance Optimization

• Database optimization
  • Query plan analysis
  • Index optimization
  • Connection pooling configuration
• Redis caching strategy
  • Cache invalidation policies
  • Session management
  • Popular data caching
• Rate limiting
  • API request limits
  • WebSocket connection limits
  • Brute force protection

---

Technical Specifications

Game Format Rules

Commander (EDH)

• Players: 2-4 (configurable)
• Deck Size: Minimum 100 cards (excluding basic lands)
• Commander: 1 legendary creature or planeswalker with "Commander" ability
• Color Identity: All cards in deck must match commander's color identity
• Ban List: Enforce official Commander ban list
• Special Rules: Commander damage (21 points = loss)

Standard

• Players: 2-4
• Deck Size: Minimum 60 cards
• Set Rotation: Current Standard-legal sets
• Ban List: Standard ban list

Modern

• Players: 2-4
• Deck Size: Minimum 60 cards
• Card Pool: All cards printed prior to 8th Edition forward
• Ban List: Modern ban list

Pauper

• Players: 2-4
• Deck Size: Minimum 60 cards
• Card Pool: Only common rarity cards
• Ban List: Pauper ban list

Database Schema

Primary Database (mtgonline)

• users - User accounts
• rooms - Game rooms
• mtg_online_games - Game sessions
• game_players - Game participant mapping
• game_history - Game results
• user_decks - User deck collections
• deck_cards - Deck card lists
• user_collections - Card collection tracking
• game_events - In-game event logs

Card Data Database (mtg_data)

• cards - Card definitions
• sets - Set information
• card_sets - Card-set relationships
• rulings - Card rulings
• prices - Card price history

API Endpoints

Authentication (/auth)

• POST /login - User login
• POST /refresh - Token refresh
• POST /register - User registration
• GET /me - Get current user

Users (/users)

• GET / - List users (admin)
• GET /{user_id} - Get user profile
• PUT /{user_id} - Update user (admin)
• DELETE /{user_id} - Delete user (admin)

Decks (/decks)

• POST / - Create deck
• GET / - List user decks
• GET /{deck_id} - Get deck details
• PUT /{deck_id} - Update deck
• DELETE /{deck_id} - Delete deck
• POST /{deck_id}/share - Generate share link

Cards (/api/cards)

• GET /search - Search cards
• GET /{card_id} - Get card details
• GET /autocomplete - Card name autocomplete

Games (/games)

• POST / - Create game
• GET / - List active games
• GET /{game_id} - Get game details
• POST /{game_id}/join - Join game
• POST /{game_id}/leave - Leave game
• POST /{game_id}/start - Start game (host)
• POST /{game_id}/end - End game

Rooms (/rooms)

• GET / - List rooms
• GET /{room_id} - Get room details
• POST / - Create room (admin)
• PUT /{room_id} - Update room (admin)
• DELETE /{room_id} - Delete room (admin)

User Data (/api/v1/user-data)

• GET /profile - Get profile
• PUT /profile - Update profile
• GET /statistics - Get statistics
• GET /games - Game history
• GET /collection - Card collection
• GET /decks - Deck collection

MTGJSON (/mtgjson)

• POST /refresh - Trigger data refresh (admin)
• GET /status - Get refresh status (admin)
• POST /verify - Verify files (admin)

WebSocket (/ws)

• WS /game/{game_id} - Game connection

---

Scalability Considerations

Current Capacity

• Concurrent Users: 20
• Concurrent Games: 10
• Expected Growth: 5x within 6 months

Scaling Strategy

Horizontal Scaling

• Stateless API servers (multiple instances behind load balancer)
• Redis cluster for session management and caching
• PostgreSQL read replicas for card data queries

Database Optimization

• Partition game_history by date
• Index frequently queried columns
• Archive old game data (6+ months)

WebSocket Scaling

• Redis pub/sub for cross-instance messaging
• Sticky sessions for WebSocket connections
• Connection pooling and limits

Caching Strategy

• L1: Application memory (hot data, 5min TTL)
• L2: Redis (session data, popular cards, 1hr TTL)
• L3: PostgreSQL (primary data)

---

Security Requirements

Authentication & Authorization

• JWT tokens with 15min access, 7day refresh
• Password hashing with bcrypt (cost factor 12)
• Rate limiting on auth endpoints
• Account lockout after 5 failed attempts

API Security

• HTTPS enforcement in production
• CORS configuration (whitelist allowed origins)
• Request validation on all endpoints
• SQL injection prevention (parameterized queries)
• XSS prevention (output encoding)

Data Protection

• Passwords never stored in plaintext
• Personal data encrypted at rest
• API keys rotated quarterly
• Audit logging for admin actions

---

Monitoring & Observability

Metrics

• API request rate and latency
• WebSocket connection count
• Database query performance
• Redis hit/miss ratio
• Error rates by endpoint

Logging

• Structured JSON logs
• Request ID tracking
• User action logging
• Error context inclusion

Alerts

• High error rate (>5% over 5min)
• Database connection pool exhaustion
• Redis memory usage >80%
• WebSocket connection failures

---

Development Guidelines

Code Style

• Python 3.12+ with type hints
• PEP 8 compliance
• Docstrings for all public functions
• Maximum line length: 88 characters (Black formatter)

Git Workflow

• Feature branches for all work
• Descriptive commit messages
• Squash merges for feature branches
• Semantic versioning for releases

Testing Requirements

• Unit tests for all service logic
• Integration tests for API endpoints
• Minimum 80% code coverage
• Tests run on every PR

Documentation

• Update README for new features
• API docs auto-generated from code
• Architecture decisions documented in ADRs
• Changelog maintained

---

Risk Mitigation

Technical Risks

Risk	Impact	Probability	Mitigation
WebSocket performance issues	High	Medium	Load test early, optimize connections
Card data sync failures	Medium	Low	Retry logic, manual override endpoint
Database lock contention	High	Medium	Connection pooling, query optimization
Third-party API downtime	Medium	Low	Caching, fallback data

Business Risks

Risk	Impact	Probability	Mitigation
Low user adoption	High	Medium	Marketing, UX improvements
Format rule changes	Medium	Low	Configurable ban lists, easy updates
Security breach	Critical	Low	Regular audits, penetration testing

---

Success Metrics

Technical Metrics

• API response time: <200ms (p95)
• WebSocket latency: <100ms
• Database query time: <50ms (p95)
• Uptime: 99.9%
• Test coverage: 80%+

Business Metrics

• Concurrent users: 20+ (Month 1), 50+ (Month 3)
• Active games: 10+ (Month 1), 30+ (Month 3)
• User retention: 60% (30-day)
• Deck creation: 100+ decks/week
• Card searches: 1000+ searches/day

---

Next Steps

Immediate (This Week)

1. ✅ Review and approve this development plan
2. ⏳ Create MTG OnlineGames model in app/models/mtg_models.py
3. ⏳ Create GamePlayer association model
4. ⏳ Create GameHistory model
5. ⏳ Replace mock game endpoints with real database operations

Short-term (Next 2 Weeks)

1. Integrate WebSocket server with database
2. Implement Commander deck validation
3. Write unit tests for game services
4. Test concurrent game sessions

Medium-term (Month 2)

1. Complete card search system
2. Implement deck management
3. Add user statistics
4. Performance optimization

---

Appendix

A. References

• MTG Official Rules
• Commander Rules
• MTGJSON API
• FastAPI Documentation
• SQLAlchemy Documentation

B. Tools & Dependencies

• Python: 3.12+
• Framework: FastAPI 0.115.0
• Database: PostgreSQL 16, SQLAlchemy 2.0
• Cache: Redis 7
• Auth: python-jose, bcrypt
• Testing: pytest, pytest-asyncio
• Deployment: Docker, docker-compose

C. Environment Variables

# Database
DATABASE_URL=postgresql://user:pass@localhost:5432/mtgonline
MTG_DATA_URL=postgresql://user:pass@localhost:5432/mtg_data

# Redis
REDIS_URL=redis://localhost:6379/0

# Authentication
SECRET_KEY=your-secret-key-here
ACCESS_TOKEN_EXPIRE_MINUTES=15
REFRESH_TOKEN_EXPIRE_DAYS=7

# Application
DEBUG=false
APP_NAME=MTG Online Backend
CORS_ORIGINS=http://localhost:3000,http://localhost:8000

---

Document Version: 2.0
Last Updated: 2025-08-27
Author: Development Team
Status: Active