mtgonline/DEVELOPMENT_PLAN.md

# MTG Online Backend - Development Plan

**Last Updated**: 2025-08-27  
**Status**: Active Development  
**Version**: 2.0

---

## Executive Summary

This document outlines the development roadmap for the MTG Online backend API. The system is designed to support Magic: The Gathering Online gameplay with a focus on Commander format, while supporting Standard, Modern, and Pauper formats. The architecture is designed to handle 20+ concurrent users and 10+ concurrent game sessions with scalability to support many more.

---

## Current Status

**Overall Progress**: 10-15% complete  
**Last Verified**: 2025-08-27

### ✅ Completed (Phases 1-4)
- Architecture simplification (unified database config, security, settings)
- Model consolidation (10+ files → 2: models.py, mtg_models.py)
- Router consolidation (54+ files → 37 files)
- Docker packaging (Dockerfile, docker-compose.yml)
- Authentication system (JWT-based login/register)
- Room management (CRUD operations)
- MTGJSON data management (download, refresh, verify)
- Basic WebSocket server (game_server.py - not integrated with DB)

### ⚠️ Partially Implemented
- Card search (basic structure exists, needs optimization)
- Deck management (stubs only, no real implementation)
- User data (profile endpoints exist, statistics not implemented)

### ❌ Not Started
- Game engine (all endpoints are mocks/stubs)
- Game models (no MTG OnlineGames, GamePlayer, or GameHistory models)
- Commander deck validation
- User statistics and history
- Real-time game state persistence

---

## Project Scope

### Core Requirements
- **Primary Format**: Commander (EDH)
- **Secondary Formats**: Standard, Modern, Pauper
- **Concurrent Users**: 20+ (scalable to 100+)
- **Concurrent Games**: 10+ (scalable to 50+)
- **Card Data Source**: MTGJSON exclusively
- **Technology Stack**: Python 3.12, FastAPI, SQLAlchemy, PostgreSQL, Redis

### Non-Goals (Out of Scope)
- Frontend development (handled separately)
- Cockatrice client integration
- Third-party tournament management
- Card image hosting (use MTGJSON links)

---

## Architecture Overview

### Target Architecture
```
┌─────────────────────────────────────────────────────────────┐
│                      API Gateway (FastAPI)                   │
├─────────────────────────────────────────────────────────────┤
│  Auth │ Users │ Decks │ Cards │ Games │ Rooms │ MTGJSON    │
├─────────────────────────────────────────────────────────────┤
│                     Service Layer                            │
│  GameServer │ CardSearch │ DeckParser │ ImportProcessor     │
├─────────────────────────────────────────────────────────────┤
│                   Data Layer                                 │
│  PostgreSQL (mtgonline + mtg_data) │ Redis (cache/sessions) │
└─────────────────────────────────────────────────────────────┘

Current Implementation:
├── ✅ Authentication (JWT)
├── ✅ Room Management (CRUD)
├── ✅ MTGJSON Data (download/refresh/verify)
├── ⚠️ Card Search (basic)
├── ❌ Game Engine (stubs only)
├── ❌ Deck Management (stubs only)
└── ❌ User Statistics (not implemented)
```

---

## Development Phases

### Phase 1: Core Game Engine (Weeks 1-3) ⭐ PRIORITY

**Status**: Not Started  
**Current State**: All game endpoints are mocks/stubs returning hardcoded data

**Objective**: Implement functional game management with real-time capabilities

#### 1.1 Game Models & Database
- [ ] Create `MTG OnlineGames` model in `app/models/mtg_models.py`
  - Game ID, room ID, host ID
  - Game type (Commander, Standard, Modern, Pauper)
  - Game state (waiting, in_progress, finished, aborted)
  - Player slots (max 4 for Commander, configurable)
  - Password protection
  - Timestamps (created, started, finished)
- [ ] Create `GamePlayer` association table
  - Player ID, game ID
  - Seat number, color identity
  - Life total, status (ready, left)
- [ ] Create `GameHistory` table
  - Game ID, participants, winner
  - Game length, result type
  - Replay data reference

#### 1.2 Game Management Endpoints
- [ ] Replace mock `POST /games/` with real implementation
  - Validate game type and format rules
  - Check room availability
  - Set host as player 1
  - Save to database
  - Return game ID and join link
- [ ] Replace mock `GET /games/` with real implementation
  - Query database for active games
  - Filter by room, game type
  - Include player count
  - Pagination support
- [ ] Replace mock `GET /games/{game_id}` with real implementation
  - Query database for game details
  - Include player list (without passwords)
  - Game state and settings
- [ ] Implement `POST /games/{game_id}/join`
  - Validate player availability
  - Check if game is full
  - Add player to game in database
- [ ] Implement `POST /games/{game_id}/leave`
  - Remove player from game in database
  - Handle host transfer if needed
- [ ] Implement `POST /games/{game_id}/start` (host only)
  - Validate all players ready
  - Transition to in_progress state
  - Broadcast to all players
- [ ] Implement `POST /games/{game_id}/end`
  - Record results to GameHistory
  - Update player statistics
  - Clean up game state

#### 1.3 WebSocket Game Server Integration
- [ ] Integrate existing `game_server.py` with database
  - Load game state from database on connection
  - Persist game state changes to database
  - Handle reconnection scenarios
- [ ] Enhance WebSocket endpoints
  - `WS /ws/game/{game_id}` - Game connection
  - Real-time player join/leave notifications
  - Game state updates (life totals, turns)
  - Chat/messaging within games
- [ ] Game state management
  - Turn tracking
  - Life total updates
  - Game log/events
  - Victory/defeat conditions

#### 1.4 Commander-Specific Logic
- [ ] Commander deck validation
  - 100-card minimum (excluding basic lands)
  - Color identity rules
  - Commander uniqueness
  - Ban list enforcement
- [ ] Commander game rules
  - Commander damage tracking
  - Command zone management
  - Popularity tracker (optional)

---

### Phase 2: Card System (Weeks 4-5)

**Status**: Not Started

**Objective**: Complete card search, filtering, and collection tracking

#### 2.1 Card Search Service
- [ ] Implement `app/services/card_search_service.py`
  - Full-text search on card names
  - Filter by: set, type, color, mana cost, rarity
  - Card name autocomplete (debounced)
  - Card detail views with images
- [ ] Search optimization
  - PostgreSQL full-text search configuration
  - Redis caching for popular searches
  - Pagination and sorting

#### 2.2 Card Database Integration
- [ ] Optimize `app/services/card_database.py`
  - Production-ready query patterns
  - Connection pooling
  - Batch operations for card imports
- [ ] Card collection tracking
  - User card collection per format
  - Card count by set/type
  - Collection statistics

#### 2.3 Card Import System
- [ ] Complete `app/services/import_batch_processor.py`
  - Excel/CSV import support
  - Fuzzy matching with `fuzzy_card_matcher.py`
  - Validation and error reporting
  - Bulk import with progress tracking
- [ ] MTGJSON integration
  - Automatic card data sync
  - Set and card version tracking
  - Image URL management

---

### Phase 3: Deck Management (Weeks 6-7)

**Status**: Not Started

**Objective**: Full deck CRUD with validation and sharing

#### 3.1 Deck Models
- [ ] Create `UserDeck` model
  - Deck ID, user ID, name
  - Format (Commander, Standard, Modern, Pauper)
  - Deck list (JSON or separate table)
  - Commander/legendary creature reference
  - Privacy settings (public, private, friends-only)
- [ ] Create `DeckCard` model
  - Deck ID, card ID
  - Quantity, sideboard flag
  - Card list ordering

#### 3.2 Deck Endpoints
- [ ] `POST /decks/` - Create deck
  - Validate format rules
  - Check card legality
  - Save deck with card list
- [ ] `GET /decks/` - List user decks
  - Filter by format
  - Pagination
- [ ] `GET /decks/{deck_id}` - Get deck details
  - Include card list with quantities
  - Format validation status
- [ ] `PUT /decks/{deck_id}` - Update deck
  - Modify card list
  - Update metadata
- [ ] `DELETE /decks/{deck_id}` - Delete deck
- [ ] `POST /decks/{deck_id}/share` - Generate share link
- [ ] `GET /decks/shared/{token}` - View shared deck

#### 3.3 Deck Parser
- [ ] Implement `app/services/deck_parser.py`
  - Parse MTGO deck list format
  - Extract card names and quantities
  - Validate against format rules
  - Error reporting for invalid decks

#### 3.4 Deck Suggestions
- [ ] Implement `app/services/deck_suggestion_service.py`
  - Meta analysis (most popular decks)
  - Card recommendations based on commander
  - Deck archetypes and templates
  - Counter-pick suggestions

---

### Phase 4: User Data & Profiles (Weeks 8-9)

**Status**: Not Started

**Objective**: Enhanced user profiles with statistics and history

#### 4.1 User Profile Enhancement
- [ ] User statistics
  - Games played (by format)
  - Win/loss record
  - Win rate by format
  - Average game length
  - Commander damage dealt/taken
- [ ] Card collection
  - Total cards owned
  - Cards by set/type/color
  - Collection completion percentage
- [ ] Deck collection
  - Total decks created
  - Decks by format
  - Most played decks

#### 4.2 User Data Endpoints
- [ ] Complete `app/routers/user_data_router.py`
  - `GET /api/v1/user-data/profile` - Get profile
  - `PUT /api/v1/user-data/profile` - Update profile
  - `GET /api/v1/user-data/statistics` - Get statistics
  - `GET /api/v1/user-data/games` - Game history
  - `GET /api/v1/user-data/collection` - Card collection
  - `GET /api/v1/user-data/decks` - Deck collection
- [ ] Data export
  - JSON export of user data
  - CSV export for statistics
  - Privacy controls

#### 4.3 Game History & Replay
- [ ] Game history tracking
  - Store game results
  - Track participant performance
  - Format-specific statistics
- [ ] Replay data storage
  - Game event log
  - Turn-by-turn actions
  - Save/load replay functionality

---

### Phase 5: Infrastructure & Quality (Weeks 10-11)

**Status**: Not Started

**Objective**: Testing, documentation, and performance optimization

#### 5.1 Testing
- [ ] Unit tests
  - Service layer tests (card search, deck parser, game logic)
  - Model validation tests
  - Utility function tests
- [ ] Integration tests
  - API endpoint tests (all routers)
  - Database transaction tests
  - WebSocket connection tests
- [ ] Load testing
  - Concurrent user simulation
  - Concurrent game session testing
  - Performance benchmarking
- [ ] Test coverage target: 80%+

#### 5.2 Documentation
- [ ] API documentation
  - OpenAPI/Swagger auto-generation (already enabled)
  - Endpoint examples
  - Authentication guide
- [ ] Architecture documentation
  - System design diagrams
  - Data flow diagrams
  - Service interaction docs
- [ ] Deployment guide
  - Docker setup instructions
  - Environment configuration
  - Database migration steps
- [ ] Contributing guide
  - Code style guidelines
  - PR process
  - Issue reporting

#### 5.3 Performance Optimization
- [ ] Database optimization
  - Query plan analysis
  - Index optimization
  - Connection pooling configuration
- [ ] Redis caching strategy
  - Cache invalidation policies
  - Session management
  - Popular data caching
- [ ] Rate limiting
  - API request limits
  - WebSocket connection limits
  - Brute force protection

---

## Technical Specifications

### Game Format Rules

#### Commander (EDH)
- **Players**: 2-4 (configurable)
- **Deck Size**: Minimum 100 cards (excluding basic lands)
- **Commander**: 1 legendary creature or planeswalker with "Commander" ability
- **Color Identity**: All cards in deck must match commander's color identity
- **Ban List**: Enforce official Commander ban list
- **Special Rules**: Commander damage (21 points = loss)

#### Standard
- **Players**: 2-4
- **Deck Size**: Minimum 60 cards
- **Set Rotation**: Current Standard-legal sets
- **Ban List**: Standard ban list

#### Modern
- **Players**: 2-4
- **Deck Size**: Minimum 60 cards
- **Card Pool**: All cards printed prior to 8th Edition forward
- **Ban List**: Modern ban list

#### Pauper
- **Players**: 2-4
- **Deck Size**: Minimum 60 cards
- **Card Pool**: Only common rarity cards
- **Ban List**: Pauper ban list

### Database Schema

#### Primary Database (mtgonline)
- `users` - User accounts
- `rooms` - Game rooms
- `mtg_online_games` - Game sessions
- `game_players` - Game participant mapping
- `game_history` - Game results
- `user_decks` - User deck collections
- `deck_cards` - Deck card lists
- `user_collections` - Card collection tracking
- `game_events` - In-game event logs

#### Card Data Database (mtg_data)
- `cards` - Card definitions
- `sets` - Set information
- `card_sets` - Card-set relationships
- `rulings` - Card rulings
- `prices` - Card price history

### API Endpoints

#### Authentication (`/auth`)
- `POST /login` - User login
- `POST /refresh` - Token refresh
- `POST /register` - User registration
- `GET /me` - Get current user

#### Users (`/users`)
- `GET /` - List users (admin)
- `GET /{user_id}` - Get user profile
- `PUT /{user_id}` - Update user (admin)
- `DELETE /{user_id}` - Delete user (admin)

#### Decks (`/decks`)
- `POST /` - Create deck
- `GET /` - List user decks
- `GET /{deck_id}` - Get deck details
- `PUT /{deck_id}` - Update deck
- `DELETE /{deck_id}` - Delete deck
- `POST /{deck_id}/share` - Generate share link

#### Cards (`/api/cards`)
- `GET /search` - Search cards
- `GET /{card_id}` - Get card details
- `GET /autocomplete` - Card name autocomplete

#### Games (`/games`)
- `POST /` - Create game
- `GET /` - List active games
- `GET /{game_id}` - Get game details
- `POST /{game_id}/join` - Join game
- `POST /{game_id}/leave` - Leave game
- `POST /{game_id}/start` - Start game (host)
- `POST /{game_id}/end` - End game

#### Rooms (`/rooms`)
- `GET /` - List rooms
- `GET /{room_id}` - Get room details
- `POST /` - Create room (admin)
- `PUT /{room_id}` - Update room (admin)
- `DELETE /{room_id}` - Delete room (admin)

#### User Data (`/api/v1/user-data`)
- `GET /profile` - Get profile
- `PUT /profile` - Update profile
- `GET /statistics` - Get statistics
- `GET /games` - Game history
- `GET /collection` - Card collection
- `GET /decks` - Deck collection

#### MTGJSON (`/mtgjson`)
- `POST /refresh` - Trigger data refresh (admin)
- `GET /status` - Get refresh status (admin)
- `POST /verify` - Verify files (admin)

#### WebSocket (`/ws`)
- `WS /game/{game_id}` - Game connection

---

## Scalability Considerations

### Current Capacity
- **Concurrent Users**: 20
- **Concurrent Games**: 10
- **Expected Growth**: 5x within 6 months

### Scaling Strategy

#### Horizontal Scaling
- Stateless API servers (multiple instances behind load balancer)
- Redis cluster for session management and caching
- PostgreSQL read replicas for card data queries

#### Database Optimization
- Partition game_history by date
- Index frequently queried columns
- Archive old game data (6+ months)

#### WebSocket Scaling
- Redis pub/sub for cross-instance messaging
- Sticky sessions for WebSocket connections
- Connection pooling and limits

#### Caching Strategy
- L1: Application memory (hot data, 5min TTL)
- L2: Redis (session data, popular cards, 1hr TTL)
- L3: PostgreSQL (primary data)

---

## Security Requirements

### Authentication & Authorization
- JWT tokens with 15min access, 7day refresh
- Password hashing with bcrypt (cost factor 12)
- Rate limiting on auth endpoints
- Account lockout after 5 failed attempts

### API Security
- HTTPS enforcement in production
- CORS configuration (whitelist allowed origins)
- Request validation on all endpoints
- SQL injection prevention (parameterized queries)
- XSS prevention (output encoding)

### Data Protection
- Passwords never stored in plaintext
- Personal data encrypted at rest
- API keys rotated quarterly
- Audit logging for admin actions

---

## Monitoring & Observability

### Metrics
- API request rate and latency
- WebSocket connection count
- Database query performance
- Redis hit/miss ratio
- Error rates by endpoint

### Logging
- Structured JSON logs
- Request ID tracking
- User action logging
- Error context inclusion

### Alerts
- High error rate (>5% over 5min)
- Database connection pool exhaustion
- Redis memory usage >80%
- WebSocket connection failures

---

## Development Guidelines

### Code Style
- Python 3.12+ with type hints
- PEP 8 compliance
- Docstrings for all public functions
- Maximum line length: 88 characters (Black formatter)

### Git Workflow
- Feature branches for all work
- Descriptive commit messages
- Squash merges for feature branches
- Semantic versioning for releases

### Testing Requirements
- Unit tests for all service logic
- Integration tests for API endpoints
- Minimum 80% code coverage
- Tests run on every PR

### Documentation
- Update README for new features
- API docs auto-generated from code
- Architecture decisions documented in ADRs
- Changelog maintained

---

## Risk Mitigation

### Technical Risks
| Risk | Impact | Probability | Mitigation |
|------|--------|-------------|------------|
| WebSocket performance issues | High | Medium | Load test early, optimize connections |
| Card data sync failures | Medium | Low | Retry logic, manual override endpoint |
| Database lock contention | High | Medium | Connection pooling, query optimization |
| Third-party API downtime | Medium | Low | Caching, fallback data |

### Business Risks
| Risk | Impact | Probability | Mitigation |
|------|--------|-------------|------------|
| Low user adoption | High | Medium | Marketing, UX improvements |
| Format rule changes | Medium | Low | Configurable ban lists, easy updates |
| Security breach | Critical | Low | Regular audits, penetration testing |

---

## Success Metrics

### Technical Metrics
- API response time: <200ms (p95)
- WebSocket latency: <100ms
- Database query time: <50ms (p95)
- Uptime: 99.9%
- Test coverage: 80%+

### Business Metrics
- Concurrent users: 20+ (Month 1), 50+ (Month 3)
- Active games: 10+ (Month 1), 30+ (Month 3)
- User retention: 60% (30-day)
- Deck creation: 100+ decks/week
- Card searches: 1000+ searches/day

---

## Next Steps

### Immediate (This Week)
1. ✅ Review and approve this development plan
2. ⏳ Create `MTG OnlineGames` model in `app/models/mtg_models.py`
3. ⏳ Create `GamePlayer` association model
4. ⏳ Create `GameHistory` model
5. ⏳ Replace mock game endpoints with real database operations

### Short-term (Next 2 Weeks)
1. Integrate WebSocket server with database
2. Implement Commander deck validation
3. Write unit tests for game services
4. Test concurrent game sessions

### Medium-term (Month 2)
1. Complete card search system
2. Implement deck management
3. Add user statistics
4. Performance optimization

---

## Appendix

### A. References
- [MTG Official Rules](https://magic.wizards.com/en/gameplay/rules)
- [Commander Rules](https://mtgcommander.com/rules)
- [MTGJSON API](https://mtgjson.com/api/)
- [FastAPI Documentation](https://fastapi.tiangolo.com/)
- [SQLAlchemy Documentation](https://docs.sqlalchemy.org/)

### B. Tools & Dependencies
- **Python**: 3.12+
- **Framework**: FastAPI 0.115.0
- **Database**: PostgreSQL 16, SQLAlchemy 2.0
- **Cache**: Redis 7
- **Auth**: python-jose, bcrypt
- **Testing**: pytest, pytest-asyncio
- **Deployment**: Docker, docker-compose

### C. Environment Variables
```bash
# Database
DATABASE_URL=postgresql://user:pass@localhost:5432/mtgonline
MTG_DATA_URL=postgresql://user:pass@localhost:5432/mtg_data

# Redis
REDIS_URL=redis://localhost:6379/0

# Authentication
SECRET_KEY=your-secret-key-here
ACCESS_TOKEN_EXPIRE_MINUTES=15
REFRESH_TOKEN_EXPIRE_DAYS=7

# Application
DEBUG=false
APP_NAME=MTG Online Backend
CORS_ORIGINS=http://localhost:3000,http://localhost:8000
```

---

**Document Version**: 2.0  
**Last Updated**: 2025-08-27  
**Author**: Development Team  
**Status**: Active