Update state.json and development plan to reflect actual progress (~15%)

- state.json: Phase 5 status changed to pending, next steps updated - DEVELOPMENT_PLAN.md v1.1: Added actual progress, verified current state, renumbered phases 5-9, marked all remaining work as PENDING - Deleted outdated plans (BACKEND_TEST_PLAN, DATA_LOADING_VERIFICATION, DOCKER_MIGRATION_PLAN, ENDPOINT_AUDIT, ROADMAP, SIMPLIFICATION_PLAN, STATEMENT_OF_INTENT, backend/TEST_PLAN) - Added new files: DEVELOPMENT_PLAN.md, backend/ARCHITECTURAL_REPORT.md, backend/SIMPLIFICATION_SUMMARY.md, backend/run.py, backend/.dockerignore
2026-08-27 04:41:30 +00:00
parent 900f51af3a
commit 0e6506d97f
17 changed files with 2938 additions and 2218 deletions
@@ -0,0 +1,690 @@
+# MTG Online Backend - Development Plan
+
+**Last Updated**: 2025-08-27  
+**Status**: Active Development  
+**Version**: 2.0
+
+---
+
+## Executive Summary
+
+This document outlines the development roadmap for the MTG Online backend API. The system is designed to support Magic: The Gathering Online gameplay with a focus on Commander format, while supporting Standard, Modern, and Pauper formats. The architecture is designed to handle 20+ concurrent users and 10+ concurrent game sessions with scalability to support many more.
+
+---
+
+## Current Status
+
+**Overall Progress**: 10-15% complete  
+**Last Verified**: 2025-08-27
+
+### ✅ Completed (Phases 1-4)
+- Architecture simplification (unified database config, security, settings)
+- Model consolidation (10+ files → 2: models.py, mtg_models.py)
+- Router consolidation (54+ files → 37 files)
+- Docker packaging (Dockerfile, docker-compose.yml)
+- Authentication system (JWT-based login/register)
+- Room management (CRUD operations)
+- MTGJSON data management (download, refresh, verify)
+- Basic WebSocket server (game_server.py - not integrated with DB)
+
+### ⚠️ Partially Implemented
+- Card search (basic structure exists, needs optimization)
+- Deck management (stubs only, no real implementation)
+- User data (profile endpoints exist, statistics not implemented)
+
+### ❌ Not Started
+- Game engine (all endpoints are mocks/stubs)
+- Game models (no MTG OnlineGames, GamePlayer, or GameHistory models)
+- Commander deck validation
+- User statistics and history
+- Real-time game state persistence
+
+---
+
+## Project Scope
+
+### Core Requirements
+- **Primary Format**: Commander (EDH)
+- **Secondary Formats**: Standard, Modern, Pauper
+- **Concurrent Users**: 20+ (scalable to 100+)
+- **Concurrent Games**: 10+ (scalable to 50+)
+- **Card Data Source**: MTGJSON exclusively
+- **Technology Stack**: Python 3.12, FastAPI, SQLAlchemy, PostgreSQL, Redis
+
+### Non-Goals (Out of Scope)
+- Frontend development (handled separately)
+- Cockatrice client integration
+- Third-party tournament management
+- Card image hosting (use MTGJSON links)
+
+---
+
+## Architecture Overview
+
+### Target Architecture
+```
+┌─────────────────────────────────────────────────────────────┐
+│                      API Gateway (FastAPI)                   │
+├─────────────────────────────────────────────────────────────┤
+│  Auth │ Users │ Decks │ Cards │ Games │ Rooms │ MTGJSON    │
+├─────────────────────────────────────────────────────────────┤
+│                     Service Layer                            │
+│  GameServer │ CardSearch │ DeckParser │ ImportProcessor     │
+├─────────────────────────────────────────────────────────────┤
+│                   Data Layer                                 │
+│  PostgreSQL (mtgonline + mtg_data) │ Redis (cache/sessions) │
+└─────────────────────────────────────────────────────────────┘
+
+Current Implementation:
+├── ✅ Authentication (JWT)
+├── ✅ Room Management (CRUD)
+├── ✅ MTGJSON Data (download/refresh/verify)
+├── ⚠️ Card Search (basic)
+├── ❌ Game Engine (stubs only)
+├── ❌ Deck Management (stubs only)
+└── ❌ User Statistics (not implemented)
+```
+
+---
+
+## Development Phases
+
+### Phase 1: Core Game Engine (Weeks 1-3) ⭐ PRIORITY
+
+**Status**: Not Started  
+**Current State**: All game endpoints are mocks/stubs returning hardcoded data
+
+**Objective**: Implement functional game management with real-time capabilities
+
+#### 1.1 Game Models & Database
+- [ ] Create `MTG OnlineGames` model in `app/models/mtg_models.py`
+  - Game ID, room ID, host ID
+  - Game type (Commander, Standard, Modern, Pauper)
+  - Game state (waiting, in_progress, finished, aborted)
+  - Player slots (max 4 for Commander, configurable)
+  - Password protection
+  - Timestamps (created, started, finished)
+- [ ] Create `GamePlayer` association table
+  - Player ID, game ID
+  - Seat number, color identity
+  - Life total, status (ready, left)
+- [ ] Create `GameHistory` table
+  - Game ID, participants, winner
+  - Game length, result type
+  - Replay data reference
+
+#### 1.2 Game Management Endpoints
+- [ ] Replace mock `POST /games/` with real implementation
+  - Validate game type and format rules
+  - Check room availability
+  - Set host as player 1
+  - Save to database
+  - Return game ID and join link
+- [ ] Replace mock `GET /games/` with real implementation
+  - Query database for active games
+  - Filter by room, game type
+  - Include player count
+  - Pagination support
+- [ ] Replace mock `GET /games/{game_id}` with real implementation
+  - Query database for game details
+  - Include player list (without passwords)
+  - Game state and settings
+- [ ] Implement `POST /games/{game_id}/join`
+  - Validate player availability
+  - Check if game is full
+  - Add player to game in database
+- [ ] Implement `POST /games/{game_id}/leave`
+  - Remove player from game in database
+  - Handle host transfer if needed
+- [ ] Implement `POST /games/{game_id}/start` (host only)
+  - Validate all players ready
+  - Transition to in_progress state
+  - Broadcast to all players
+- [ ] Implement `POST /games/{game_id}/end`
+  - Record results to GameHistory
+  - Update player statistics
+  - Clean up game state
+
+#### 1.3 WebSocket Game Server Integration
+- [ ] Integrate existing `game_server.py` with database
+  - Load game state from database on connection
+  - Persist game state changes to database
+  - Handle reconnection scenarios
+- [ ] Enhance WebSocket endpoints
+  - `WS /ws/game/{game_id}` - Game connection
+  - Real-time player join/leave notifications
+  - Game state updates (life totals, turns)
+  - Chat/messaging within games
+- [ ] Game state management
+  - Turn tracking
+  - Life total updates
+  - Game log/events
+  - Victory/defeat conditions
+
+#### 1.4 Commander-Specific Logic
+- [ ] Commander deck validation
+  - 100-card minimum (excluding basic lands)
+  - Color identity rules
+  - Commander uniqueness
+  - Ban list enforcement
+- [ ] Commander game rules
+  - Commander damage tracking
+  - Command zone management
+  - Popularity tracker (optional)
+
+---
+
+### Phase 2: Card System (Weeks 4-5)
+
+**Status**: Not Started
+
+**Objective**: Complete card search, filtering, and collection tracking
+
+#### 2.1 Card Search Service
+- [ ] Implement `app/services/card_search_service.py`
+  - Full-text search on card names
+  - Filter by: set, type, color, mana cost, rarity
+  - Card name autocomplete (debounced)
+  - Card detail views with images
+- [ ] Search optimization
+  - PostgreSQL full-text search configuration
+  - Redis caching for popular searches
+  - Pagination and sorting
+
+#### 2.2 Card Database Integration
+- [ ] Optimize `app/services/card_database.py`
+  - Production-ready query patterns
+  - Connection pooling
+  - Batch operations for card imports
+- [ ] Card collection tracking
+  - User card collection per format
+  - Card count by set/type
+  - Collection statistics
+
+#### 2.3 Card Import System
+- [ ] Complete `app/services/import_batch_processor.py`
+  - Excel/CSV import support
+  - Fuzzy matching with `fuzzy_card_matcher.py`
+  - Validation and error reporting
+  - Bulk import with progress tracking
+- [ ] MTGJSON integration
+  - Automatic card data sync
+  - Set and card version tracking
+  - Image URL management
+
+---
+
+### Phase 3: Deck Management (Weeks 6-7)
+
+**Status**: Not Started
+
+**Objective**: Full deck CRUD with validation and sharing
+
+#### 3.1 Deck Models
+- [ ] Create `UserDeck` model
+  - Deck ID, user ID, name
+  - Format (Commander, Standard, Modern, Pauper)
+  - Deck list (JSON or separate table)
+  - Commander/legendary creature reference
+  - Privacy settings (public, private, friends-only)
+- [ ] Create `DeckCard` model
+  - Deck ID, card ID
+  - Quantity, sideboard flag
+  - Card list ordering
+
+#### 3.2 Deck Endpoints
+- [ ] `POST /decks/` - Create deck
+  - Validate format rules
+  - Check card legality
+  - Save deck with card list
+- [ ] `GET /decks/` - List user decks
+  - Filter by format
+  - Pagination
+- [ ] `GET /decks/{deck_id}` - Get deck details
+  - Include card list with quantities
+  - Format validation status
+- [ ] `PUT /decks/{deck_id}` - Update deck
+  - Modify card list
+  - Update metadata
+- [ ] `DELETE /decks/{deck_id}` - Delete deck
+- [ ] `POST /decks/{deck_id}/share` - Generate share link
+- [ ] `GET /decks/shared/{token}` - View shared deck
+
+#### 3.3 Deck Parser
+- [ ] Implement `app/services/deck_parser.py`
+  - Parse MTGO deck list format
+  - Extract card names and quantities
+  - Validate against format rules
+  - Error reporting for invalid decks
+
+#### 3.4 Deck Suggestions
+- [ ] Implement `app/services/deck_suggestion_service.py`
+  - Meta analysis (most popular decks)
+  - Card recommendations based on commander
+  - Deck archetypes and templates
+  - Counter-pick suggestions
+
+---
+
+### Phase 4: User Data & Profiles (Weeks 8-9)
+
+**Status**: Not Started
+
+**Objective**: Enhanced user profiles with statistics and history
+
+#### 4.1 User Profile Enhancement
+- [ ] User statistics
+  - Games played (by format)
+  - Win/loss record
+  - Win rate by format
+  - Average game length
+  - Commander damage dealt/taken
+- [ ] Card collection
+  - Total cards owned
+  - Cards by set/type/color
+  - Collection completion percentage
+- [ ] Deck collection
+  - Total decks created
+  - Decks by format
+  - Most played decks
+
+#### 4.2 User Data Endpoints
+- [ ] Complete `app/routers/user_data_router.py`
+  - `GET /api/v1/user-data/profile` - Get profile
+  - `PUT /api/v1/user-data/profile` - Update profile
+  - `GET /api/v1/user-data/statistics` - Get statistics
+  - `GET /api/v1/user-data/games` - Game history
+  - `GET /api/v1/user-data/collection` - Card collection
+  - `GET /api/v1/user-data/decks` - Deck collection
+- [ ] Data export
+  - JSON export of user data
+  - CSV export for statistics
+  - Privacy controls
+
+#### 4.3 Game History & Replay
+- [ ] Game history tracking
+  - Store game results
+  - Track participant performance
+  - Format-specific statistics
+- [ ] Replay data storage
+  - Game event log
+  - Turn-by-turn actions
+  - Save/load replay functionality
+
+---
+
+### Phase 5: Infrastructure & Quality (Weeks 10-11)
+
+**Status**: Not Started
+
+**Objective**: Testing, documentation, and performance optimization
+
+#### 5.1 Testing
+- [ ] Unit tests
+  - Service layer tests (card search, deck parser, game logic)
+  - Model validation tests
+  - Utility function tests
+- [ ] Integration tests
+  - API endpoint tests (all routers)
+  - Database transaction tests
+  - WebSocket connection tests
+- [ ] Load testing
+  - Concurrent user simulation
+  - Concurrent game session testing
+  - Performance benchmarking
+- [ ] Test coverage target: 80%+
+
+#### 5.2 Documentation
+- [ ] API documentation
+  - OpenAPI/Swagger auto-generation (already enabled)
+  - Endpoint examples
+  - Authentication guide
+- [ ] Architecture documentation
+  - System design diagrams
+  - Data flow diagrams
+  - Service interaction docs
+- [ ] Deployment guide
+  - Docker setup instructions
+  - Environment configuration
+  - Database migration steps
+- [ ] Contributing guide
+  - Code style guidelines
+  - PR process
+  - Issue reporting
+
+#### 5.3 Performance Optimization
+- [ ] Database optimization
+  - Query plan analysis
+  - Index optimization
+  - Connection pooling configuration
+- [ ] Redis caching strategy
+  - Cache invalidation policies
+  - Session management
+  - Popular data caching
+- [ ] Rate limiting
+  - API request limits
+  - WebSocket connection limits
+  - Brute force protection
+
+---
+
+## Technical Specifications
+
+### Game Format Rules
+
+#### Commander (EDH)
+- **Players**: 2-4 (configurable)
+- **Deck Size**: Minimum 100 cards (excluding basic lands)
+- **Commander**: 1 legendary creature or planeswalker with "Commander" ability
+- **Color Identity**: All cards in deck must match commander's color identity
+- **Ban List**: Enforce official Commander ban list
+- **Special Rules**: Commander damage (21 points = loss)
+
+#### Standard
+- **Players**: 2-4
+- **Deck Size**: Minimum 60 cards
+- **Set Rotation**: Current Standard-legal sets
+- **Ban List**: Standard ban list
+
+#### Modern
+- **Players**: 2-4
+- **Deck Size**: Minimum 60 cards
+- **Card Pool**: All cards printed prior to 8th Edition forward
+- **Ban List**: Modern ban list
+
+#### Pauper
+- **Players**: 2-4
+- **Deck Size**: Minimum 60 cards
+- **Card Pool**: Only common rarity cards
+- **Ban List**: Pauper ban list
+
+### Database Schema
+
+#### Primary Database (mtgonline)
+- `users` - User accounts
+- `rooms` - Game rooms
+- `mtg_online_games` - Game sessions
+- `game_players` - Game participant mapping
+- `game_history` - Game results
+- `user_decks` - User deck collections
+- `deck_cards` - Deck card lists
+- `user_collections` - Card collection tracking
+- `game_events` - In-game event logs
+
+#### Card Data Database (mtg_data)
+- `cards` - Card definitions
+- `sets` - Set information
+- `card_sets` - Card-set relationships
+- `rulings` - Card rulings
+- `prices` - Card price history
+
+### API Endpoints
+
+#### Authentication (`/auth`)
+- `POST /login` - User login
+- `POST /refresh` - Token refresh
+- `POST /register` - User registration
+- `GET /me` - Get current user
+
+#### Users (`/users`)
+- `GET /` - List users (admin)
+- `GET /{user_id}` - Get user profile
+- `PUT /{user_id}` - Update user (admin)
+- `DELETE /{user_id}` - Delete user (admin)
+
+#### Decks (`/decks`)
+- `POST /` - Create deck
+- `GET /` - List user decks
+- `GET /{deck_id}` - Get deck details
+- `PUT /{deck_id}` - Update deck
+- `DELETE /{deck_id}` - Delete deck
+- `POST /{deck_id}/share` - Generate share link
+
+#### Cards (`/api/cards`)
+- `GET /search` - Search cards
+- `GET /{card_id}` - Get card details
+- `GET /autocomplete` - Card name autocomplete
+
+#### Games (`/games`)
+- `POST /` - Create game
+- `GET /` - List active games
+- `GET /{game_id}` - Get game details
+- `POST /{game_id}/join` - Join game
+- `POST /{game_id}/leave` - Leave game
+- `POST /{game_id}/start` - Start game (host)
+- `POST /{game_id}/end` - End game
+
+#### Rooms (`/rooms`)
+- `GET /` - List rooms
+- `GET /{room_id}` - Get room details
+- `POST /` - Create room (admin)
+- `PUT /{room_id}` - Update room (admin)
+- `DELETE /{room_id}` - Delete room (admin)
+
+#### User Data (`/api/v1/user-data`)
+- `GET /profile` - Get profile
+- `PUT /profile` - Update profile
+- `GET /statistics` - Get statistics
+- `GET /games` - Game history
+- `GET /collection` - Card collection
+- `GET /decks` - Deck collection
+
+#### MTGJSON (`/mtgjson`)
+- `POST /refresh` - Trigger data refresh (admin)
+- `GET /status` - Get refresh status (admin)
+- `POST /verify` - Verify files (admin)
+
+#### WebSocket (`/ws`)
+- `WS /game/{game_id}` - Game connection
+
+---
+
+## Scalability Considerations
+
+### Current Capacity
+- **Concurrent Users**: 20
+- **Concurrent Games**: 10
+- **Expected Growth**: 5x within 6 months
+
+### Scaling Strategy
+
+#### Horizontal Scaling
+- Stateless API servers (multiple instances behind load balancer)
+- Redis cluster for session management and caching
+- PostgreSQL read replicas for card data queries
+
+#### Database Optimization
+- Partition game_history by date
+- Index frequently queried columns
+- Archive old game data (6+ months)
+
+#### WebSocket Scaling
+- Redis pub/sub for cross-instance messaging
+- Sticky sessions for WebSocket connections
+- Connection pooling and limits
+
+#### Caching Strategy
+- L1: Application memory (hot data, 5min TTL)
+- L2: Redis (session data, popular cards, 1hr TTL)
+- L3: PostgreSQL (primary data)
+
+---
+
+## Security Requirements
+
+### Authentication & Authorization
+- JWT tokens with 15min access, 7day refresh
+- Password hashing with bcrypt (cost factor 12)
+- Rate limiting on auth endpoints
+- Account lockout after 5 failed attempts
+
+### API Security
+- HTTPS enforcement in production
+- CORS configuration (whitelist allowed origins)
+- Request validation on all endpoints
+- SQL injection prevention (parameterized queries)
+- XSS prevention (output encoding)
+
+### Data Protection
+- Passwords never stored in plaintext
+- Personal data encrypted at rest
+- API keys rotated quarterly
+- Audit logging for admin actions
+
+---
+
+## Monitoring & Observability
+
+### Metrics
+- API request rate and latency
+- WebSocket connection count
+- Database query performance
+- Redis hit/miss ratio
+- Error rates by endpoint
+
+### Logging
+- Structured JSON logs
+- Request ID tracking
+- User action logging
+- Error context inclusion
+
+### Alerts
+- High error rate (>5% over 5min)
+- Database connection pool exhaustion
+- Redis memory usage >80%
+- WebSocket connection failures
+
+---
+
+## Development Guidelines
+
+### Code Style
+- Python 3.12+ with type hints
+- PEP 8 compliance
+- Docstrings for all public functions
+- Maximum line length: 88 characters (Black formatter)
+
+### Git Workflow
+- Feature branches for all work
+- Descriptive commit messages
+- Squash merges for feature branches
+- Semantic versioning for releases
+
+### Testing Requirements
+- Unit tests for all service logic
+- Integration tests for API endpoints
+- Minimum 80% code coverage
+- Tests run on every PR
+
+### Documentation
+- Update README for new features
+- API docs auto-generated from code
+- Architecture decisions documented in ADRs
+- Changelog maintained
+
+---
+
+## Risk Mitigation
+
+### Technical Risks
+| Risk | Impact | Probability | Mitigation |
+|------|--------|-------------|------------|
+| WebSocket performance issues | High | Medium | Load test early, optimize connections |
+| Card data sync failures | Medium | Low | Retry logic, manual override endpoint |
+| Database lock contention | High | Medium | Connection pooling, query optimization |
+| Third-party API downtime | Medium | Low | Caching, fallback data |
+
+### Business Risks
+| Risk | Impact | Probability | Mitigation |
+|------|--------|-------------|------------|
+| Low user adoption | High | Medium | Marketing, UX improvements |
+| Format rule changes | Medium | Low | Configurable ban lists, easy updates |
+| Security breach | Critical | Low | Regular audits, penetration testing |
+
+---
+
+## Success Metrics
+
+### Technical Metrics
+- API response time: <200ms (p95)
+- WebSocket latency: <100ms
+- Database query time: <50ms (p95)
+- Uptime: 99.9%
+- Test coverage: 80%+
+
+### Business Metrics
+- Concurrent users: 20+ (Month 1), 50+ (Month 3)
+- Active games: 10+ (Month 1), 30+ (Month 3)
+- User retention: 60% (30-day)
+- Deck creation: 100+ decks/week
+- Card searches: 1000+ searches/day
+
+---
+
+## Next Steps
+
+### Immediate (This Week)
+1. ✅ Review and approve this development plan
+2. ⏳ Create `MTG OnlineGames` model in `app/models/mtg_models.py`
+3. ⏳ Create `GamePlayer` association model
+4. ⏳ Create `GameHistory` model
+5. ⏳ Replace mock game endpoints with real database operations
+
+### Short-term (Next 2 Weeks)
+1. Integrate WebSocket server with database
+2. Implement Commander deck validation
+3. Write unit tests for game services
+4. Test concurrent game sessions
+
+### Medium-term (Month 2)
+1. Complete card search system
+2. Implement deck management
+3. Add user statistics
+4. Performance optimization
+
+---
+
+## Appendix
+
+### A. References
+- [MTG Official Rules](https://magic.wizards.com/en/gameplay/rules)
+- [Commander Rules](https://mtgcommander.com/rules)
+- [MTGJSON API](https://mtgjson.com/api/)
+- [FastAPI Documentation](https://fastapi.tiangolo.com/)
+- [SQLAlchemy Documentation](https://docs.sqlalchemy.org/)
+
+### B. Tools & Dependencies
+- **Python**: 3.12+
+- **Framework**: FastAPI 0.115.0
+- **Database**: PostgreSQL 16, SQLAlchemy 2.0
+- **Cache**: Redis 7
+- **Auth**: python-jose, bcrypt
+- **Testing**: pytest, pytest-asyncio
+- **Deployment**: Docker, docker-compose
+
+### C. Environment Variables
+```bash
+# Database
+DATABASE_URL=postgresql://user:pass@localhost:5432/mtgonline
+MTG_DATA_URL=postgresql://user:pass@localhost:5432/mtg_data
+
+# Redis
+REDIS_URL=redis://localhost:6379/0
+
+# Authentication
+SECRET_KEY=your-secret-key-here
+ACCESS_TOKEN_EXPIRE_MINUTES=15
+REFRESH_TOKEN_EXPIRE_DAYS=7
+
+# Application
+DEBUG=false
+APP_NAME=MTG Online Backend
+CORS_ORIGINS=http://localhost:3000,http://localhost:8000
+```
+
+---
+
+**Document Version**: 2.0  
+**Last Updated**: 2025-08-27  
+**Author**: Development Team  
+**Status**: Active