admin/mtgonline
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
6b38ef07eeca2551652808ece3a29d59facd36d0
mtgonline/HANDOFF.md
T
akadmin c9bb68c6bc docs: update HANDOFF.md and ROADMAP.md with user data schema progress 
- Handoff.md: document 16-table user data schema, Alembic migrations,
  and all API endpoints (replays, cards, groups, networks, preferences, activity)
- Handoff.md: consolidate state.json reference to project root
- Roadmap.md: restructure phases to reflect completed work
  Phase 1: Backend Foundation (complete)
  Phase 2: User Data Schema & API (complete) - 16 tables, 7 routers
  Phase 3: Frontend Development (complete)
  Phase 4: Testing & Deployment (pending)
2026-07-23 03:30:00 +00:00

524 lines
21 KiB
Markdown
Raw Blame History

# MTG Online Backend - Handoff Document
## Project Overview
**Project Name**: MTG Online Backend
**Location**: `/home/wall-o/projects/mtgonline`
**Purpose**: Python FastAPI application that processes Magic: The Gathering card data from MTGJSON and stores it in PostgreSQL databases.
## Vision Statement (from STATEMENT_OF_INTENT.md)
**MTG Online Web** — A modern, web-based implementation of the MTG Online multiplayer Magic: The Gathering platform.
To build a fully-featured, open-source multiplayer Magic: The Gathering platform that runs entirely in modern web browsers, eliminating the need for desktop software installations while maintaining compatibility with the existing MTG Online ecosystem.
## Core Objectives
### User Experience
- Intuitive, modern interface that rivals native desktop applications
- Real-time multiplayer gameplay with minimal latency
- Seamless deck building with import/export from MTG Online
- Responsive design that works across all screen sizes
### Technical Excellence
- **Backend**: Python 3.12 + FastAPI with async architecture
- **Database**: PostgreSQL with async SQLAlchemy ORM
- **Real-time**: WebSocket-based game server for live multiplayer
- **Protocol**: Full compatibility with MTG Online protocol buffer messages
### Feature Parity with Desktop
- User authentication and account management
- Deck creation, editing, and storage (per-user)
- Multiplayer game rooms with real-time state sync
- Card game mechanics (mana, phases, priority, stack)
- Admin/moderation tools (ban, warn, log viewing)
- Card database integration with comprehensive card data
### Performance Requirements
- API response times < 100ms for 95% of requests
- WebSocket latency < 50ms for game state updates
- Support 1000+ concurrent users
- Sub-second page loads with proper caching
### Success Criteria
- [ ] Users can create accounts and authenticate securely
- [ ] Users can create, edit, and manage decks (per-user)
- [ ] Users can join and play multiplayer games in real-time
- [ ] Game state syncs correctly across all connected players
- [ ] Admin users can manage accounts and moderate games
- [ ] Deck formats are compatible with MTG Online desktop client
## Current Status
All planned tasks have been completed:
- ✅ Documentation created (root README.md + backend/README.md)
- ✅ State.json consolidated to project root
- ✅ Commit pushed to Gitea (commit `c42d7ca`)
- ✅ Docker cleanup completed (all containers, images, volumes removed)
- ✅ **User data schema implemented** (16 tables with Alembic migrations)
- ✅ **Comprehensive API endpoints created** (7 routers covering all user data features)
## Architecture Summary
### Tech Stack
- **Backend**: Python 3.12, FastAPI, SQLAlchemy (async), asyncpg
- **Databases**: Dual PostgreSQL (14-alpine)
- Primary: `mtgonline` database (users, decks, auth)
- MTG Data: `mtgdata` database (card data, sets)
- **Cache**: Redis 7-alpine
- **Protocol**: Protocol buffer message compatibility
### Service Architecture
```
mtgonline/
├── backend/ # FastAPI application
│ ├── app/
│ │ ├── core/ # Settings, database engines, Redis client
│ │ ├── models/ # SQLAlchemy ORM models (user_data.py - 16 models)
│ │ ├── routers/ # API route modules (auth, users, decks, rooms, games, admin, cards, interactions, refresh, ws)
│ │ │ ├── user_data.py # User data endpoints (replays, collections, groups, etc.)
│ │ ├── schemas/ # Pydantic request/response schemas (user_data_schemas.py)
│ │ ├── services/ # Business logic (MTGJSON manager, card DB, game server, deck parser)
│ │ └── main.py # FastAPI app entry point (user-data mounted at /api/v1/user-data)
│ ├── alembic/ # Database migrations
│ │ ├── env.py # Async Alembic configuration
│ │ └── versions/ # Migration scripts
│ │ └── 001_initial_user_schema.py
│ ├── scripts/ # Utility scripts (downloads, migrations, checks)
│ │ └── run_migrations.sh
│ ├── Dockerfile # Updated to run migrations on startup
│ ├── requirements.txt
│ └── .env.example
├── docker-compose.dev.yml # Development stack
├── docker-compose.yml # Production stack
└── README.md # Project documentation
```
### Data Flow
1. Backend connects to PostgreSQL (both instances) and Redis on startup
2. Checks `mtg_refresh_log` in `mtgdata` database for existing data
3. If no data exists, downloads MTGJSON files from `https://mtgjson.com/api/v5/`
4. Upserts data into `mtgdata` tables (`mtg_sets`, `mtg_cards`, etc.)
5. Data available via REST endpoints
### Key Database Connections
- **Primary DB**: `postgresql+asyncpg://mtgonline_user:mtgonline_password@postgres:5432/mtgonline`
- **MTG DB**: `postgresql+asyncpg://mtgonline_user:mtgonline_password@mtgdata:5432/mtgdata`
- **Redis**: `redis://redis:6379`
## Environment Configuration
### Docker Compose Dev Environment Variables
```yaml
environment:
DATABASE_URL: "postgresql+asyncpg://mtgonline_user:mtgonline_password@postgres:5432/mtgonline"
MTG_DATABASE_URL: "postgresql+asyncpg://mtgonline_user:mtgonline_password@mtgdata:5432/mtgdata"
REDIS_URL: "redis://redis:6379"
```
### Service Ports (Host to Container)
- PostgreSQL: `5432:5432`
- MTG Data PostgreSQL: `5433:5432`
- Redis: `6379:6379`
- Backend: `5555:8000`
## API Endpoints
### Health & Status
- `GET /health` - Health check with MTGJSON status
- `GET /` - API info
### Authentication
- `POST /auth/login` - User login
- `POST /auth/register` - User registration
- `POST /auth/refresh` - Refresh JWT
- `GET /auth/me` - Current user
### Users
- `GET /users/{user_id}` - Get user
- `PATCH /users/{user_id}` - Update user
- `POST /users/{user_id}/ban` - Ban user (admin)
### Decks
- `GET /decks/` - List decks
- `POST /decks/` - Create deck
- `GET /decks/{deck_id}` - Get deck
- `PATCH /decks/{deck_id}` - Update deck
- `DELETE /decks/{deck_id}` - Delete deck
### MTG Cards
- `GET /api/cards/` - Search cards
- `GET /api/cards/{card_id}` - Get card
- `GET /api/sets/` - List sets
### Admin
- `GET /admin/users` - List all users
- `GET /admin/bans` - List bans
- `POST /admin/bans` - Create ban
### Data Management
- `POST /refresh` - Trigger MTGJSON refresh
### WebSocket
- `WS /ws/{room_id}` - Real-time game communication
## MTGJSON Data Pipeline
### Downloaded Files
- `AllPrintings.psql` - Main card data (PostgreSQL format)
- `AllIdentifiers.json` - Card identifiers
- `Keywords.json` - Card keywords
- `CardTypes.json` - Card type definitions
- `AllDeckFiles.zip` - Deck files (unzipped on load)
### Refresh Logic
1. **On Startup**: Checks `mtg_refresh_log` for existing data
2. **If No Data**: Downloads and loads all MTGJSON files (may take minutes)
3. **Manual Refresh**: `POST /refresh` triggers immediate reload
4. **Logging**: All refreshes logged to `mtg_refresh_log` with status, timing, and counts
## Git Repository
**Gitea Repository**: `https://git.optimex.systems/admin/mtgonline.git`
**Credentials**: Located at `/home/wall-o/projects/gitea_credentials.txt`
**Current Branch**: `main`
**Last Commit**: `c42d7ca` - "feat: implement user data schema and API endpoints"
### Commit History
```
c42d7ca - feat: implement user data schema and API endpoints
- Add Alembic migration setup with async configuration
- Create 16 user data models (users, decks, cards, replays, etc.)
- Implement comprehensive API endpoints with JWT auth
- Add replay, card collection, group, network, preferences, and activity log routers
- Include API documentation and migration test plan
46abfe5 - Add comprehensive documentation for MTG Online Backend
6f01e2d - Initial project setup
```
## Running the Project
### Start Services
```bash
cd /home/wall-o/projects/mtgonline
docker compose -f docker-compose.dev.yml up -d
```
### Check Services
```bash
docker compose -f docker-compose.dev.yml ps
docker compose -f docker-compose.dev.yml logs -f backend
```
### Stop Services
```bash
docker compose -f docker-compose.dev.yml down
```
### Full Cleanup
```bash
# Stop and remove all containers
docker compose -f docker-compose.dev.yml down
# Remove images
docker images rm mtgonline-backend:latest postgres:14-alpine redis:7-alpine
# Clear build cache and prune
docker builder prune -af
docker system prune -af --volumes
```
## Testing
### Run Tests
```bash
cd /home/wall-o/projects/mtgonline/backend
docker exec -it <backend_container_id> pytest
# OR
cd /home/wall-o/projects/mtgonline/backend
pytest
```
### Database Verification
```bash
# Check MTG data tables
docker exec <mtgdata_container_id> psql -U mtgonline_user mtgdata -c "\dt"
# Check refresh log
docker exec <mtgdata_container_id> psql -U mtgonline_user mtgdata -c "SELECT * FROM mtg_refresh_log ORDER BY refresh_time DESC LIMIT 5;"
```
## Troubleshooting
### Backend can't connect to databases
- Verify all services are running: `docker compose -f docker-compose.dev.yml ps`
- Check logs: `docker compose -f docker-compose.dev.yml logs backend`
- Ensure environment variables match docker-compose.dev.yml
### MTGJSON download fails
- Check network connectivity to mtgjson.com
- Verify DATA_DIR has write permissions
- Check disk space: `df -h`
- Manual download: `python scripts/download_mtgjson.py`
### Database tables missing
- Run initialization: `docker exec -i <mtgdata_container_id> psql -U mtgonline_user mtgdata < /path/to/scripts/init-mtgdata.sql`
- Check tables: `docker exec <mtgdata_container_id> psql -U mtgonline_user mtgdata -c "\dt"`
### CORS errors
- Check CORS_ORIGINS setting in app/core/settings.py
- Ensure frontend URL matches allowed origins
## Next Phase: V1 Backend — Deck Building & Card Management (NEW SCOPE)
The v1 app function focuses on three core capabilities:
1. **Per-user deck building** with card search, deck precedents, and card suggestions
2. **Card list import** from spreadsheet/text files with fuzzy matching
3. **Multiplayer gameplay** (handled by a separate backend)
### 1. Per-User Deck Building
#### Database Schema
- **`user_decks` table** (per-user storage in primary `mtgonline` database):
- `deck_id` (PK, auto-increment)
- `user_id` (FK → users)
- `name` (text)
- `status` (ENUM: `DRAFT`, `FINAL`)
- `DRAFT` = works in progress, can be edited freely
- `FINAL` = user considers it complete, no further changes expected
- `cards` (JSONB or separate junction table with `card_id`, `quantity`)
- `created_at`, `updated_at` (timestamps)
- `folder_id` (FK → user folders, optional)
#### API Endpoints to Implement
- `POST /decks/` — Create new draft deck (auto status: DRAFT)
- `GET /decks/` — List user's decks, filtered by status
- `GET /decks/{deck_id}` — Get full deck details
- `PATCH /decks/{deck_id}` — Update deck (name, status, card list)
- `POST /decks/{deck_id}/finalize` — Transition DRAFT → FINAL
- `DELETE /decks/{deck_id}` — Delete deck (only if FINAL, or admin override)
- `GET /decks/{deck_id}/cards` — Get cards in deck with quantity counts
#### Deck Building Features
- **Card Search**: Search the MTG card database by name, type, set, color, etc. Returns matching cards with full details.
- **Deck Precedents**: Preset/starting deck templates that users can use as a basis. Could be built-in (e.g., "Starter Deck") or user-saved as FINAL decks to be reused.
- **Card Suggestion**: Given a card already in the deck, suggest similar cards (same type, same color, same set, same mana cost, or cards often paired with the input card in existing decks).
### 2. Card Import from Files
#### Supported Formats
- XLSX (Excel)
- CSV
- JSON
- ODS (OpenDocument Spreadsheet)
#### Import Flow
1. User uploads a file (XLSX, CSV, JSON, or ODS)
2. Backend parses the file — each row is treated as one card entry
3. Duplicates within the file are allowed (each row → one card instance)
4. For each card name in the file, backend performs **fuzzy matching** against the `Cards` PSQL table
5. Matched cards are stored in a **user-owned card table** with metadata:
- `user_id` (FK → users)
- `card_id` (FK → mtg_cards from mtgdata, matched via fuzzy search)
- `raw_name` (original name from file, for traceability)
- `confidence` (match score from fuzzy search)
- `imported_at` (timestamp)
#### Fuzzy Search Requirements
- Must handle **spelling errors** (e.g., "Wondrrland" → "Wonderland")
- Must handle **American vs British English** differences (e.g., "color" vs "colour", "armor" vs "armour")
- Use a fuzzy string matching library (e.g., `python-Levenshtein`, `thefuzz`/`fuzzymatch`)
- Confidence threshold to auto-accept vs flag for manual review
- Bulk matching: process all cards in the file in a single batch operation
#### API Endpoints to Implement
- `POST /cards/import` — Upload file for import
- `GET /cards/import/{import_id}/status` — Check import progress/status
- `GET /cards/import/{import_id}/results` — Get match results with confidence scores
- `POST /cards/import/{import_id}/confirm` — Confirm import (save to user card table)
- `GET /user/cards` — List user's imported/owned cards
- `DELETE /user/cards/{card_import_id}` — Remove from user card table
### 3. Multiplayer Play Feature (Separate Backend)
#### Architecture Decision
The multiplayer gameplay feature will live in a **separate backend codebase** to ensure smooth, independent development. This backend will communicate with the card backend via:
- **API Calls**: For authentication, user data, deck retrieval, card lookups
- **Direct PSQL Queries**: For card data and user deck data
#### Integration Points
- **Card Backend API** (`http://backend:8000`):
- `GET /api/cards/{card_id}` — Get card details for in-game display
- `GET /api/cards/search?q=...` — Search cards during gameplay
- `GET /api/sets/` — List available sets for game formatting
- **PSQL Direct Access** (via shared connection string):
- `mtgdata` database — Read card data (cards, sets, etc.)
- `mtgonline` database — Read user decks (for deck validation, game setup)
#### API Endpoints for Play Backend
- `POST /play/decks/{deck_id}/validate` — Validate deck against card database
- `GET /play/users/{user_id}/decks` — Get user's FINAL decks for selection
- `GET /play/cards/{card_id}` — Get card details for game board display
**Note**: The play backend is out of scope for this document. See separate codebase/repository when ready.
### Summary of Work Required in This Backend
| Feature | Database | API | Notes |
|---------|----------|-----|-------|
| User deck CRUD | `mtgonline` (new tables) | Full CRUD + finalize | DRAFT/FINAL status |
| Card search | `mtgdata` (existing) | Search endpoint | Leverages existing card DB |
| Card suggestions | `mtgonline` + `mtgdata` | Suggestion endpoint | Based on similar cards |
| File import (XLSX/CSV/JSON/ODS) | `mtgonline` (new user cards table) | Upload + confirm | Fuzzy match required |
| Fuzzy matching service | N/A | Internal service | Handles spelling + EN variants |
| Play backend integration | Read-only access | API consumer | Separate codebase |
### Files to Create/Modify
- New models: `models/user_deck.py`, `models/user_card.py`
- New migrations: Alembic migrations for new tables
- New router: `routers/decks.py`, `routers/card_import.py`
- New service: `services/fuzzy_card_matcher.py`
- New service: `services/deck_suggestion.py`
- New schema: `schemas/deck.py`, `schemas/card_import.py`
- Update `core/database.py` if new engine needed
- Update `requirements.txt` with fuzzy matching libraries
See **ROADMAP.md Phase 2** for detailed task breakdown.
## User Data Schema & API
### Database Models (16 Tables)
- **`users`** - User accounts with authentication
- **`decks`** - User decks (DRAFT/FINAL status)
- **`cards`** - User card collections
- **`card_ownership`** - Card ownership tracking
- **`win_streaks`** - Win/loss statistics
- **`game_replays`** - Saved game replays (JSONB)
- **`groups`** - User groups
- **`group_members`** - Group membership
- **`networks`** - Network accounts (Twitch, X, YouTube)
- **`network_credentials`** - Network login info
- **`preferences`** - User preferences (JSONB)
- **`activity_log`** - User activity tracking (JSONB)
- **`suggested_cards`** - Card suggestions
- **`folders`** - Deck organization
- **`game_logs`** - Game audit trail
- **`user_decks`** - User deck storage (DRAFT/FINAL)
### API Endpoints
#### Replays (`/api/v1/user-data/replays`)
- `POST /api/v1/user-data/replays/` - Save replay
- `GET /api/v1/user-data/replays/{replay_id}` - Get replay
- `DELETE /api/v1/user-data/replays/{replay_id}` - Delete replay
#### Card Collection (`/api/v1/user-data/cards`)
- `GET /api/v1/user-data/cards/` - List user's cards
- `POST /api/v1/user-data/cards/` - Add card to collection
- `DELETE /api/v1/user-data/cards/{card_id}` - Remove card
#### Groups (`/api/v1/user-data/groups`)
- `GET /api/v1/user-data/groups/` - List user's groups
- `POST /api/v1/user-data/groups/` - Create group
- `PATCH /api/v1/user-data/groups/{group_id}` - Update group
- `DELETE /api/v1/user-data/groups/{group_id}` - Delete group
#### Networks (`/api/v1/user-data/networks`)
- `GET /api/v1/user-data/networks/` - List network accounts
- `POST /api/v1/user-data/networks/` - Add network
- `PATCH /api/v1/user-data/networks/{network_id}` - Update network
- `DELETE /api/v1/user-data/networks/{network_id}` - Remove network
#### Preferences (`/api/v1/user-data/preferences`)
- `GET /api/v1/user-data/preferences/` - Get preferences
- `PATCH /api/v1/user-data/preferences/` - Update preferences
#### Activity Log (`/api/v1/user-data/activity`)
- `GET /api/v1/user-data/activity/` - List activity
- `POST /api/v1/user-data/activity/` - Add activity entry
### Alembic Migrations
- **Async configuration** with `run_sync` for database operations
- **Initial migration**: `001_initial_user_schema.py` creates all 16 tables
- **Migration script**: `scripts/run_migrations.sh` runs on container startup
- **JSONB columns** used for flexible data storage (replay_data, activity_data, preferences)
### Architecture Decisions
- **CASCADE deletes** for data integrity in related tables
- **Composite unique constraints** for card collection uniqueness
- **RESTful API design** with pagination support
- **JWT authentication** for all endpoints
- **Permission checks** for group/network management
## State File
Current state saved at: `/home/wall-o/projects/mtgonline/state.json`
```json
{
"task_description": "Create Alembic migration setup for user data schema and complete API endpoints for all user data features",
"current_step": "Phase 3 completed - All API endpoints created with comprehensive documentation",
"commit_hash": "c42d7ca",
"timestamp": "2026-07-22T23:23:00-04:00"
}
```
## Access Information
- **Backend API**: `http://localhost:5555`
- **User Data API**: `http://localhost:5555/api/v1/user-data`
- **Swagger Docs**: `http://localhost:5555/docs`
- **Health Check**: `http://localhost:5555/health`
- **Gitea**: `https://git.optimex.systems/admin/mtgonline`
### Key Files to Review
#### Current State
1. `/home/wall-o/projects/mtgonline/state.json` - Current project state
2. `/home/wall-o/projects/mtgonline/README.md` - Project documentation
3. `/home/wall-o/projects/mtgonline/backend/README.md` - Backend documentation
#### Architecture & Configuration
4. `/home/wall-o/projects/mtgonline/docker-compose.dev.yml` - Docker configuration
5. `/home/wall-o/projects/mtgonline/backend/app/core/settings.py` - Application settings
6. `/home/wall-o/projects/mtgonline/backend/app/main.py` - FastAPI entry point (user-data mounted at /api/v1/user-data)
#### User Data Models
7. `/home/wall-o/projects/mtgonline/backend/app/models/user_data.py` - All user data models (16 tables)
8. `/home/wall-o/projects/mtgonline/backend/alembic/versions/001_initial_user_schema.py` - Migration script
#### API Endpoints
9. `/home/wall-o/projects/mtgonline/backend/app/routers/user_data.py` - User data API endpoints
10. `/home/wall-o/projects/mtgonline/backend/app/schemas/user_data_schemas.py` - Pydantic schemas for user data
#### Documentation
11. `/home/wall-o/projects/mtgonline/backend/API_DOCUMENTATION.md` - Comprehensive API documentation
12. `/home/wall-o/projects/mtgonline/backend/TEST_PLAN.md` - Migration test plan
13. `/home/wall-o/projects/mtgonline/ROADMAP.md` - Complete feature roadmap and timeline
14. `/home/wall-o/projects/mtgonline/STATEMENT_OF_INTENT.md` - Project vision and objectives
## Environment
- **OS**: Linux 6.8.0-136-generic (x86_64)
- **Docker**: Available
- **Python**: 3.12.3
- **Working Directory**: `/home/wall-o/projects/mtgonline`
---
**Last Updated**: 2026-07-22T23:23:00-04:00
**Status**: Phase 1-3 Complete. Ready for Phase 4: Testing and Deployment.
**Next Action**: Test migration execution in container, run API tests against all endpoints, add rate limiting, create integration tests, deploy to staging environment.
**Timeline**:
| Phase | Duration | Status |
|-------|----------|--------|
| Phase 1: Backend Foundation | 2 weeks | ✅ Complete |
| Phase 2: Frontend Development | 4 weeks | Not Started |
| Phase 3: Integration & Polish | 2 weeks | Not Started |
| Phase 4: Deployment & Production | 1 week | Not Started |
| Phase 5: Advanced Features | Ongoing | Future |
Reference in New Issue View Git Blame Copy Permalink