Clean up backend folder - remove obsolete scripts and test files

Removed obsolete scripts that were not imported or used: - card_interaction_rule_engine.py - card_profile_extractor.py - create_card_interaction_graph.py - interaction_determinator.py - interaction_pipeline.py - interaction_recommender.py - interaction_schema.py - recommendation_engine.py - migrate_complete.py - migrate_schema.py - test_interaction_determinator.py - check_mtgjson_full.py - check_mtgjson_status.py - verify_integration.py - verify_mtgjson_data.py - sanity_check_mtgjson.py - investigate_sets.py - inspect_db.py - code_review.md - monitor/mtg_monitor.py Removed test artifacts: - test.db - test_download.py - test_system.py - setup_db.py - BACKEND_TESTING_SUMMARY.md - CHAT_PROMPT_TEST.md - CONTINUATION_PROMPT.md - PORTED_STATE.md - SPEC_synergy-mapping-engine.md - STATE.md - SUPPORTED_FILE_TYPES.md Removed sensitive/environment files: - .env.local - state.json (backend) Cleaned up: - __pycache__ directories - venv directory Backend scripts/ directory now contains only essential data loading and maintenance scripts.
2026-07-22 03:00:04 +00:00
parent b4a0b1f8be
commit a01e33eb5e
35 changed files with 1093 additions and 9215 deletions
@@ -51,100 +51,406 @@ A modern web-based implementation of the MTG Online multiplayer Magic: The Gathe
 - [x] Project Roadmap
 - [x] State tracking

-## Phase 2: Frontend Development (TODO)
+## Phase 2: V1 Backend — Deck Building & Card Management (IN PROGRESS)

-### 2.1 Project Setup
+### 2.0 Project Setup
+- [x] Initialize Python project structure
+- [x] Create requirements.txt with pinned dependencies
+- [x] Set up pydantic-settings configuration
+- [x] Configure async SQLAlchemy with PostgreSQL
+- [x] Create JWT authentication system with bcrypt
+- [x] Set up FastAPI application with CORS
+- [x] Fuzzy matching library setup (python-Levenshtein / thefuzz)
+
+### 2.1 Database Models
+- [x] User model (accounts, profiles, VIP status)
+- [x] Ban model (moderation, history)
+- [ ] **NEW: User Deck model** (`user_decks` table)
+  - `deck_id` (PK, auto-increment)
+  - `user_id` (FK → users)
+  - `name` (text)
+  - `status` (ENUM: DRAFT, FINAL)
+  - `cards` (JSONB or junction table with card_id, quantity)
+  - `created_at`, `updated_at` (timestamps)
+  - `folder_id` (FK → user folders, optional)
+- [ ] **NEW: User Card model** (`user_cards` table)
+  - `user_card_id` (PK, auto-increment)
+  - `user_id` (FK → users)
+  - `card_id` (FK → mtg_cards from mtgdata)
+  - `raw_name` (original name from import file)
+  - `confidence` (match score from fuzzy search)
+  - `imported_at` (timestamp)
+  - `import_id` (FK → import batch)
+- [ ] **NEW: Card Import Batch model**
+  - `import_id` (PK)
+  - `user_id` (FK → users)
+  - `file_name` (text)
+  - `status` (ENUM: PENDING, PROCESSING, COMPLETED, FAILED)
+  - `total_cards` (int)
+  - `matched_cards` (int)
+  - `created_at` (timestamp)
+
+### 2.2 API Endpoints
+
+#### User Deck CRUD
+- [ ] `POST /decks/` — Create new draft deck (auto status: DRAFT)
+- [ ] `GET /decks/` — List user's decks, filterable by status
+- [ ] `GET /decks/{deck_id}` — Get full deck details
+- [ ] `PATCH /decks/{deck_id}` — Update deck (name, status, card list)
+- [ ] `POST /decks/{deck_id}/finalize` — Transition DRAFT → FINAL
+- [ ] `DELETE /decks/{deck_id}` — Delete deck (only if FINAL, or admin override)
+- [ ] `GET /decks/{deck_id}/cards` — Get cards in deck with quantity counts
+
+#### Card Search & Suggestions
+- [ ] `GET /api/cards/search?q={query}&type={type}&set={set}&color={color}` — Search MTG cards
+- [ ] `GET /api/cards/{card_id}` — Get card details
+- [ ] `GET /api/sets/` — List available sets
+- [ ] `GET /api/cards/suggest?deck_id={deck_id}&limit={n}` — Suggest similar cards
+
+#### Card Import
+- [ ] `POST /cards/import` — Upload file (XLSX, CSV, JSON, ODS)
+- [ ] `GET /cards/import/{import_id}/status` — Check import progress
+- [ ] `GET /cards/import/{import_id}/results` — Get match results with confidence
+- [ ] `POST /cards/import/{import_id}/confirm` — Confirm import
+- [ ] `GET /user/cards` — List user's imported cards
+- [ ] `DELETE /user/cards/{card_import_id}` — Remove from user cards
+
+### 2.3 Services
+
+#### Deck Building Services
+- [ ] `services/deck_manager.py` — Deck CRUD operations, status transitions
+- [ ] `services/card_search.py` — Card search with filters (name, type, set, color)
+- [ ] `services/deck_suggestion.py` — Similar card suggestions based on existing deck
+  - Match by: same type, same color, same set, same mana cost
+  - Match by: cards often paired in existing user decks
+
+#### Card Import Services
+- [ ] `services/file_parser.py` — Parse XLSX, CSV, JSON, ODS files
+  - Each row = one card entry (duplicates allowed)
+- [ ] `services/fuzzy_card_matcher.py` — Fuzzy string matching service
+  - Handle spelling errors (e.g., "Wondrrland" → "Wonderland")
+  - Handle American vs British English (e.g., "color" vs "colour")
+  - Use python-Levenshtein or thefuzz for matching
+  - Return confidence scores for each match
+- [ ] `services/import_batch_processor.py` — Process import batches
+  - Batch fuzzy matching for all cards in file
+  - Update import batch status
+  - Save matched cards to user_cards table
+
+### 2.4 Fuzzy Matching Implementation
+
+#### Requirements
+- Must handle **spelling errors** in card names
+- Must handle **American vs British English** differences
+- Must return **confidence scores** for match quality
+- Must process **bulk imports** efficiently
+
+#### Approach
+1. Use `thefuzz` (Python) or `python-Levenshtein` for string matching
+2. Implement two-stage matching:
+   - Stage 1: Exact match (if card name exists exactly)
+   - Stage 2: Fuzzy match (if no exact match, find best match)
+3. Configure similarity threshold (e.g., 85% for auto-accept, 70-84% for manual review)
+4. Pre-process card names to normalize:
+   - Remove extra whitespace
+   - Normalize punctuation
+   - Handle known spelling variants
+
+#### Example Match Results
+```
+Raw: "Wondrrland Explorer"
+Match: "Wonderland Explorer" (confidence: 92%)
+
+Raw: "Armour Plated"
+Match: "Armor Plated" (confidence: 88%)
+
+Raw: "Colorful Burst"
+Match: "Color Burst" (confidence: 85%)
+```
+
+### 2.5 Multiplayer Play Backend — Architecture Blueprint (Derived from Cockatrice Analysis)
+
+A detailed architecture analysis of **Cockatrice** (v3.1.0 "Graduation Day") — the mature open-source MTG online client/server — has been completed at `/home/wall-o/projects/mtgonline/C++/ARCHITECTURE_ANALYSIS.md`.
+
+The play backend will follow the same authoritative server model, adapted for a modern web stack:
+
+#### Authority Model
+
+```
+┌─────────────────┐         WebSocket (JSON)             ┌─────────────────┐
+│   CLIENT        │◄────────────────────────────────────►│   SERVER        │
+│  (React/TS)     │                                      │  (FastAPI/Py)   │
+│                 │                                      │                 │
+│ • Game Scene    │   GameCommands (play, attack, etc.)  │ • PostgreSQL    │
+│ • Hand View     │◄────────────────────────────────────►│ • Game State    │
+│ • Chat Panel    │   GameEvents (state changes)         │ • Room/Player   │
+│ • Deck Panel    │                                      │   Management    │
+│ • Phase Toolbar │   Chat, Admin, Spectator             │ • Replay Log    │
+└─────────────────┘                                      └─────────────────┘
+```
+
+#### Game Engine Architecture
+
+| Cockatrice Component | Modern Web Equivalent | Role |
+|---|---|---|
+| `AbstractGame` | `Game` (server-side) | Core game instance holding state, players, event handler |
+| `GameMetaInfo` | `GameMetadata` | gameId, maxPlayers, description, started, spectators |
+| `GameState` | `GameBoardState` | currentPhase, activePlayer, hostId, gameTimer |
+| `GameEventHandler` | `GameEventDispatcher` | Central dispatch — processes events, prepares commands (~21KB in Cockatrice) |
+| `PlayerManager` | `PlayerRegistry` | Coordinates all players in a game |
+| `PlayerLogic` | `Player` | Per-player game logic (~10.7KB in Cockatrice) |
+| `PlayerActions` | `PlayerCommands` | Concrete commands: play, attack, tap, draw (~64KB in Cockatrice) |
+| `CardZone` | `Zone` (base class) | Abstract zone — Hand/Stack/Table/Graveyard/Exile/Library |
+| `HandZone` | `Hand` | Player's hand (secret/hidden zone) |
+| `StackZone` | `Stack` | Spells/abilities on the stack |
+| `TableZone` | `Battlefield` | Permanents on the battlefield |
+| `PileZone` | `Pile` (Graveyard, Exile, Library) | Discard/exile/draw piles |
+| `Replay` | `GameReplay` | Serialized event stream for replay |
+| `Phase` | `TurnPhase` | 11-phase MTG turn structure |
+
+#### Turn Phase System (11 Phases with Sub-Phases)
+
+```
+Untap → Upkeep → Draw → Main 1 → Combat → Main 2 → End → Cleanup
+                                        │
+                                        └── Sub-phases:
+                                            Beginning of Combat
+                                            Declare Attackers
+                                            Declare Blockers
+                                            Combat Damage
+                                            End of Combat
+```
+
+#### Command/Event Flow
+
+```
+User Action (React component)
+    │
+    ▼
+GameCommand (JSON message)
+    │
+    ▼
+GameEventDispatcher.process()
+    │
+    ▼
+Player.handleCommand()
+    │
+    ▼
+ZoneLogic (state mutation)
+    │
+    ▼
+GameEvent (broadcast to all clients via WebSocket)
+    │
+    ▼
+Client receives & updates UI via Zustand/XState
+```
+
+#### Network Protocol Design (Adapted from Cockatrice's protobuf)
+
+Cockatrice uses Protocol Buffers over TCP. The modern web equivalent replaces protobuf with JSON over WebSocket:
+
+| Cockatrice Proto Message | JSON WebSocket Message |
+|---|---|
+| `ServerInfo_Game` | `{ "type": "game_info", "game_id": 1, "max_players": 2, ... }` |
+| `ServerInfo_Player` | `{ "type": "player_info", "player_id": 1, "name": "...", ... }` |
+| `Command_PlayCard` | `{ "type": "cmd_play_card", "card_id": 42, "zone": "hand", ... }` |
+| `Command_Attack` | `{ "type": "cmd_attack", "attacker_id": 7, "targets": [3, 5], ... }` |
+| `Event_Join` | `{ "type": "event_join", "player_id": 2, "properties": {...} }` |
+| `Event_Leave` | `{ "type": "event_leave", "player_id": 1, "reason": "..." }` |
+| `Event_SetActivePlayer` | `{ "type": "event_active_player", "player_id": 1 }` |
+| `Event_SetActivePhase` | `{ "type": "event_active_phase", "phase": 5 }` |
+| `Event_GameSay` | `{ "type": "event_chat", "player_id": 1, "message": "..." }` |
+| `GameReplay` | `{ "type": "replay", "events": [...] }` |
+
+#### Architecture Patterns to Reuse
+
+| Pattern | Cockatrice Usage | Modern Equivalent |
+|---|---|---|
+| **Event Bus** | Qt signals/slots | WebSocket broadcast + Zustand stores |
+| **Command Pattern** | `Command_PlayCard`, `Command_Attack` | JSON command messages, validated server-side |
+| **Observer** | `GameEventHandler` emits signals | WebSocket events trigger UI updates |
+| **Strategy** | `CardZoneLogic` subclasses | Zone classes with strategy pattern (Hand, Stack, Table, Pile, View) |
+| **Facade** | `AbstractGame` | Single `Game` object wrapping all subsystems |
+| **Memento** | `DeckListMemento` for undo | Immutable state snapshots for undo/redo |
+| **Repository** | `ServatriceDatabaseInterface` | SQLAlchemy repositories for game state, decks, logs |
+
+#### Key Decisions (Informed by Cockatrice Analysis)
+
+1. **Server is authoritative** — clients send commands, server validates and broadcasts events. No client-side state manipulation.
+2. **Deterministic replay** — serialize all game events; same command sequence produces identical state.
+3. **Zone abstraction** — each zone type (Hand, Stack, Battlefield, Pile) is independently implementable.
+4. **Phase system** — 11-phase MTG turn with sub-phases, tracked server-side.
+5. **Command/Event separation** — what a player *wants* to do vs. what *happens*.
+
+#### What Modernizes Cockatrice
+
+| Cockatrice Limitation | Modern Web Solution |
+|---|---|
+| TCP only, no web protocol | WebSocket (JSON) |
+| Qt Widgets desktop UI | React/Next.js with PixiJS for game board |
+| C++/MySQL backend | Python/FastAPI + PostgreSQL |
+| Single-server clustering | Stateless game servers + Redis state cache |
+| No mobile support | Responsive design from start |
+| No REST API | FastAPI REST + WebSocket hybrid |
+| Protobuf serialization | JSON over WebSocket |
+| 106KB monolithic server handler | Modular service architecture |
+
+### 2.6 Play Backend Responsibilities
+
+The play backend will implement:
+- [ ] Real-time game state management (server-authoritative)
+- [ ] Multiplayer WebSocket communication
+- [ ] Game rule enforcement (combat, stack resolution, priority)
+- [ ] Deck validation during gameplay
+- [ ] Game history and replay (serialized event log)
+- [ ] Room and game lobbies
+- [ ] Spectator mode
+- [ ] Admin commands (kick, ban, game control)
+
+**Note**: The play backend lives in a separate codebase. Integration points with the card backend:
+- **Card Backend API** (`http://backend:8000`):
+  - Authentication via JWT
+  - Card lookups: `GET /api/cards/{card_id}`
+  - Card search: `GET /api/cards/search?q=...`
+  - Set lists: `GET /api/sets/`
+  - User decks: `GET /api/users/{user_id}/decks`
+- **Direct PSQL Access**:
+  - `mtgdata` database: Read card data, sets, etc.
+  - `mtgonline` database: Read user decks, validate deck legality
+   - Normalize punctuation
+   - Handle known spelling variants
+
+#### Example Match Results
+```
+Raw: "Wondrrland Explorer"
+Match: "Wonderland Explorer" (confidence: 92%)
+
+Raw: "Armour Plated"
+Match: "Armor Plated" (confidence: 88%)
+
+Raw: "Colorful Burst"
+Match: "Color Burst" (confidence: 85%)
+```
+
+### 2.5 Multiplayer Play Backend (SEPARATE CODEBASE)
+
+#### Architecture
+The multiplayer gameplay feature will live in a **separate backend codebase** to ensure smooth, independent development.
+
+#### Integration Points
+- **Card Backend API** (`http://backend:8000`):
+  - Authentication via JWT
+  - Card lookups: `GET /api/cards/{card_id}`
+  - Card search: `GET /api/cards/search?q=...`
+  - Set lists: `GET /api/sets/`
+  - User decks: `GET /api/users/{user_id}/decks`
+
+- **Direct PSQL Access**:
+  - `mtgdata` database: Read card data, sets, etc.
+  - `mtgonline` database: Read user decks, validate deck legality
+
+#### Play Backend Responsibilities
+- Real-time game state management
+- Multiplayer WebSocket communication
+- Game rule enforcement
+- Deck validation during gameplay
+- Game history and replay
+
+**Note**: This backend is out of scope for the current codebase. See separate repository when ready.
+
+### 2.6 Testing
+- [ ] Unit tests for deck CRUD operations
+- [ ] Unit tests for card search functionality
+- [ ] Unit tests for card suggestion algorithm
+- [ ] Unit tests for file parsers (XLSX, CSV, JSON, ODS)
+- [ ] Unit tests for fuzzy matching service
+- [ ] Integration tests for import workflow
+- [ ] Load tests for bulk import processing
+
+### 2.7 Documentation
+- [ ] API documentation (FastAPI auto-generated)
+- [ ] Database schema documentation
+- [ ] Fuzzy matching algorithm documentation
+- [ ] Import workflow documentation
+- [ ] Play backend integration guide
+
+## Phase 3: Frontend Development (TODO)
+
+### 3.1 Project Setup
 - [ ] Initialize React + TypeScript project with Vite
 - [ ] Configure ESLint, Prettier, TypeScript strict mode
 - [ ] Set up Zustand for state management
 - [ ] Configure Tailwind CSS for styling
 - [ ] Set up Vitest + React Testing Library

-### 2.2 Authentication
+### 3.2 Authentication
 - [ ] Login form with JWT token storage
 - [ ] Registration form with validation
 - [ ] Protected routes and auth context
 - [ ] Session management and token refresh

-### 2.3 Deck Builder
+### 3.3 Deck Builder
 - [ ] Card search with filters (name, color, type, set)
 - [ ] Deck list editor with drag-and-drop
 - [ ] Import/export deck formats (plain text, native XML)
 - [ ] Folder management UI
 - [ ] Real-time deck statistics (card count, mana curve)
+- [ ] **NEW: Deck status indicator** (DRAFT vs FINAL)
+- [ ] **NEW: Card suggestion panel** (shows similar cards)

-### 2.4 Game Interface
+### 3.4 Card Import Interface
+- [ ] File upload component (XLSX, CSV, JSON, ODS)
+- [ ] Import progress indicator
+- [ ] Match results display with confidence scores
+- [ ] Manual override for low-confidence matches
+- [ ] Import history and re-import capability
+
+### 3.5 Game Interface (TODO - Dependent on Play Backend)
 - [ ] Game board visualization (zones, cards)
 - [ ] Player hand (private zone)
 - [ ] Library, graveyard, exile, command zones
 - [ ] Card interaction (click, drag, hover)
 - [ ] Real-time WebSocket updates

-### 2.5 Chat System
+### 3.6 Chat System
 - [ ] Room chat interface
 - [ ] Game chat (in-game messaging)
 - [ ] Player list display
 - [ ] Moderator tools (kick, ban)

-### 2.6 Admin Dashboard
+### 3.7 Admin Dashboard
 - [ ] User management interface
 - [ ] Ban/unban controls
 - [ ] Game logs viewer
 - [ ] System statistics

-## Phase 3: Integration & Polish (TODO)
+## Phase 4: Integration & Polish (TODO)

-### 3.1 WebSocket Client
+### 4.1 WebSocket Client
 - [ ] WebSocket connection management
 - [ ] Reconnection logic with exponential backoff
 - [ ] Message serialization/deserialization
 - [ ] Protocol buffer message handling

-### 3.2 Card Database
+### 4.2 Card Database
 - [ ] Import MTJSON card data
 - [ ] Cache card images locally
 - [ ] Search and filter functionality
 - [ ] Card tooltips with oracle text

-### 3.3 Game Logic
+### 4.3 Game Logic (TODO - Dependent on Play Backend)
 - [ ] Turn-based state management
 - [ ] Priority system implementation
 - [ ] Stack resolution
 - [ ] Mana payment tracking
 - [ ] Life totals and counters

-### 3.4 Performance
+### 4.4 Performance
 - [ ] Virtual scrolling for card lists
 - [ ] Memoization and React.memo
 - [ ] Code splitting and lazy loading
 - [ ] WebSocket message batching

-## Phase 4: Deployment & Production (TODO)
-
-### 4.1 DevOps
- [ ] Docker Compose for development
- [ ] Production Docker images
- [ ] CI/CD pipeline (GitHub Actions)
- [ ] Environment configuration management
-
-### 4.2 Security
- [ ] Rate limiting
- [ ] Input validation and sanitization
- [ ] CORS configuration
- [ ] HTTPS/SSL configuration
-
-### 4.3 Monitoring
- [ ] Logging with structured formats
- [ ] Error tracking (Sentry)
- [ ] Performance monitoring
- [ ] Uptime monitoring
-
-### 4.4 Documentation
- [ ] User documentation
- [ ] Developer documentation
- [ ] API documentation
- [ ] Architecture documentation
-
 ## Phase 5: Advanced Features (FUTURE)

 ### 5.1 Multiplayer Enhancements