mtgonline/HANDOFF.md

MTG Online Backend - Handoff Document

Project Overview

Project Name: MTG Online Backend Location: /home/wall-o/projects/mtgonline Purpose: Python FastAPI application that processes Magic: The Gathering card data from MTGJSON and stores it in PostgreSQL databases.

Vision Statement (from STATEMENT_OF_INTENT.md)

MTG Online Web — A modern, web-based implementation of the MTG Online multiplayer Magic: The Gathering platform.

To build a fully-featured, open-source multiplayer Magic: The Gathering platform that runs entirely in modern web browsers, eliminating the need for desktop software installations while maintaining compatibility with the existing MTG Online ecosystem.

Core Objectives

User Experience

• Intuitive, modern interface that rivals native desktop applications
• Real-time multiplayer gameplay with minimal latency
• Seamless deck building with import/export from MTG Online
• Responsive design that works across all screen sizes

Technical Excellence

• Backend: Python 3.12 + FastAPI with async architecture
• Database: PostgreSQL with async SQLAlchemy ORM
• Real-time: WebSocket-based game server for live multiplayer
• Protocol: Full compatibility with MTG Online protocol buffer messages

Feature Parity with Desktop

• User authentication and account management
• Deck creation, editing, and storage (per-user)
• Multiplayer game rooms with real-time state sync
• Card game mechanics (mana, phases, priority, stack)
• Admin/moderation tools (ban, warn, log viewing)
• Card database integration with comprehensive card data

Performance Requirements

• API response times < 100ms for 95% of requests
• WebSocket latency < 50ms for game state updates
• Support 1000+ concurrent users
• Sub-second page loads with proper caching

Success Criteria

• Users can create accounts and authenticate securely
• Users can create, edit, and manage decks (per-user)
• Users can join and play multiplayer games in real-time
• Game state syncs correctly across all connected players
• Admin users can manage accounts and moderate games
• Deck formats are compatible with MTG Online desktop client

Current Status

All planned tasks have been completed:

• ✅ Documentation created (root README.md + backend/README.md)
• ✅ State.json consolidated to project root
• ✅ Commit pushed to Gitea (commit c42d7ca)
• ✅ Docker cleanup completed (all containers, images, volumes removed)
• ✅ User data schema implemented (16 tables with Alembic migrations)
• ✅ Comprehensive API endpoints created (7 routers covering all user data features)

Architecture Summary

Tech Stack

• Backend: Python 3.12, FastAPI, SQLAlchemy (async), asyncpg
• Databases: Dual PostgreSQL (14-alpine)
  • Primary: mtgonline database (users, decks, auth)
  • MTG Data: mtgdata database (card data, sets)
• Cache: Redis 7-alpine
• Protocol: Protocol buffer message compatibility

Service Architecture

mtgonline/
├── backend/                    # FastAPI application
│   ├── app/
│   │   ├── core/              # Settings, database engines, Redis client
│   │   ├── models/            # SQLAlchemy ORM models (user_data.py - 16 models)
│   │   ├── routers/           # API route modules (auth, users, decks, rooms, games, admin, cards, interactions, refresh, ws)
│   │   │   ├── user_data.py   # User data endpoints (replays, collections, groups, etc.)
│   │   ├── schemas/           # Pydantic request/response schemas (user_data_schemas.py)
│   │   ├── services/          # Business logic (MTGJSON manager, card DB, game server, deck parser)
│   │   └── main.py            # FastAPI app entry point (user-data mounted at /api/v1/user-data)
│   ├── alembic/               # Database migrations
│   │   ├── env.py             # Async Alembic configuration
│   │   └── versions/          # Migration scripts
│   │       └── 001_initial_user_schema.py
│   ├── scripts/               # Utility scripts (downloads, migrations, checks)
│   │   └── run_migrations.sh
│   ├── Dockerfile             # Updated to run migrations on startup
│   ├── requirements.txt
│   └── .env.example
├── docker-compose.dev.yml     # Development stack
├── docker-compose.yml         # Production stack
└── README.md                  # Project documentation

Data Flow

1. Backend connects to PostgreSQL (both instances) and Redis on startup
2. Checks mtg_refresh_log in mtgdata database for existing data
3. If no data exists, downloads MTGJSON files from https://mtgjson.com/api/v5/
4. Upserts data into mtgdata tables (mtg_sets, mtg_cards, etc.)
5. Data available via REST endpoints

Key Database Connections

• Primary DB: postgresql+asyncpg://mtgonline_user:mtgonline_password@postgres:5432/mtgonline
• MTG DB: postgresql+asyncpg://mtgonline_user:mtgonline_password@mtgdata:5432/mtgdata
• Redis: redis://redis:6379

Environment Configuration

Docker Compose Dev Environment Variables

environment:
  DATABASE_URL: "postgresql+asyncpg://mtgonline_user:mtgonline_password@postgres:5432/mtgonline"
  MTG_DATABASE_URL: "postgresql+asyncpg://mtgonline_user:mtgonline_password@mtgdata:5432/mtgdata"
  REDIS_URL: "redis://redis:6379"

Service Ports (Host to Container)

• PostgreSQL: 5432:5432
• MTG Data PostgreSQL: 5433:5432
• Redis: 6379:6379
• Backend: 5555:8000

API Endpoints

Health & Status

• GET /health - Health check with MTGJSON status
• GET / - API info

Authentication

• POST /auth/login - User login
• POST /auth/register - User registration
• POST /auth/refresh - Refresh JWT
• GET /auth/me - Current user

Users

• GET /users/{user_id} - Get user
• PATCH /users/{user_id} - Update user
• POST /users/{user_id}/ban - Ban user (admin)

Decks

• GET /decks/ - List decks
• POST /decks/ - Create deck
• GET /decks/{deck_id} - Get deck
• PATCH /decks/{deck_id} - Update deck
• DELETE /decks/{deck_id} - Delete deck

MTG Cards

• GET /api/cards/ - Search cards
• GET /api/cards/{card_id} - Get card
• GET /api/sets/ - List sets

Admin

• GET /admin/users - List all users
• GET /admin/bans - List bans
• POST /admin/bans - Create ban

Data Management

• POST /refresh - Trigger MTGJSON refresh

WebSocket

• WS /ws/{room_id} - Real-time game communication

MTGJSON Data Pipeline

Downloaded Files

• AllPrintings.psql - Main card data (PostgreSQL format)
• AllIdentifiers.json - Card identifiers
• Keywords.json - Card keywords
• CardTypes.json - Card type definitions
• AllDeckFiles.zip - Deck files (unzipped on load)

Refresh Logic

1. On Startup: Checks mtg_refresh_log for existing data
2. If No Data: Downloads and loads all MTGJSON files (may take minutes)
3. Manual Refresh: POST /refresh triggers immediate reload
4. Logging: All refreshes logged to mtg_refresh_log with status, timing, and counts

Git Repository

Gitea Repository: https://git.optimex.systems/admin/mtgonline.git Credentials: Located at /home/wall-o/projects/gitea_credentials.txt Current Branch: main Last Commit: c42d7ca - "feat: implement user data schema and API endpoints"

Commit History

c42d7ca - feat: implement user data schema and API endpoints
  - Add Alembic migration setup with async configuration
  - Create 16 user data models (users, decks, cards, replays, etc.)
  - Implement comprehensive API endpoints with JWT auth
  - Add replay, card collection, group, network, preferences, and activity log routers
  - Include API documentation and migration test plan
46abfe5 - Add comprehensive documentation for MTG Online Backend
6f01e2d - Initial project setup

Running the Project

Start Services

cd /home/wall-o/projects/mtgonline
docker compose -f docker-compose.dev.yml up -d

Check Services

docker compose -f docker-compose.dev.yml ps
docker compose -f docker-compose.dev.yml logs -f backend

Stop Services

docker compose -f docker-compose.dev.yml down

Full Cleanup

# Stop and remove all containers
docker compose -f docker-compose.dev.yml down

# Remove images
docker images rm mtgonline-backend:latest postgres:14-alpine redis:7-alpine

# Clear build cache and prune
docker builder prune -af
docker system prune -af --volumes

Testing

Run Tests

cd /home/wall-o/projects/mtgonline/backend
docker exec -it <backend_container_id> pytest
# OR
cd /home/wall-o/projects/mtgonline/backend
pytest

Database Verification

# Check MTG data tables
docker exec <mtgdata_container_id> psql -U mtgonline_user mtgdata -c "\dt"

# Check refresh log
docker exec <mtgdata_container_id> psql -U mtgonline_user mtgdata -c "SELECT * FROM mtg_refresh_log ORDER BY refresh_time DESC LIMIT 5;"

Troubleshooting

Backend can't connect to databases

• Verify all services are running: docker compose -f docker-compose.dev.yml ps
• Check logs: docker compose -f docker-compose.dev.yml logs backend
• Ensure environment variables match docker-compose.dev.yml

MTGJSON download fails

• Check network connectivity to mtgjson.com
• Verify DATA_DIR has write permissions
• Check disk space: df -h
• Manual download: python scripts/download_mtgjson.py

Database tables missing

• Run initialization: docker exec -i <mtgdata_container_id> psql -U mtgonline_user mtgdata < /path/to/scripts/init-mtgdata.sql
• Check tables: docker exec <mtgdata_container_id> psql -U mtgonline_user mtgdata -c "\dt"

CORS errors

• Check CORS_ORIGINS setting in app/core/settings.py
• Ensure frontend URL matches allowed origins

Next Phase: V1 Backend — Deck Building & Card Management (NEW SCOPE)

The v1 app function focuses on three core capabilities:

1. Per-user deck building with card search, deck precedents, and card suggestions
2. Card list import from spreadsheet/text files with fuzzy matching
3. Multiplayer gameplay (handled by a separate backend)

1. Per-User Deck Building

Database Schema

• user_decks table (per-user storage in primary mtgonline database):
  • deck_id (PK, auto-increment)
  • user_id (FK → users)
  • name (text)
  • status (ENUM: DRAFT, FINAL)
    • DRAFT = works in progress, can be edited freely
    • FINAL = user considers it complete, no further changes expected
  • cards (JSONB or separate junction table with card_id, quantity)
  • created_at, updated_at (timestamps)
  • folder_id (FK → user folders, optional)

API Endpoints to Implement

• POST /decks/ — Create new draft deck (auto status: DRAFT)
• GET /decks/ — List user's decks, filtered by status
• GET /decks/{deck_id} — Get full deck details
• PATCH /decks/{deck_id} — Update deck (name, status, card list)
• POST /decks/{deck_id}/finalize — Transition DRAFT → FINAL
• DELETE /decks/{deck_id} — Delete deck (only if FINAL, or admin override)
• GET /decks/{deck_id}/cards — Get cards in deck with quantity counts

Deck Building Features

• Card Search: Search the MTG card database by name, type, set, color, etc. Returns matching cards with full details.
• Deck Precedents: Preset/starting deck templates that users can use as a basis. Could be built-in (e.g., "Starter Deck") or user-saved as FINAL decks to be reused.
• Card Suggestion: Given a card already in the deck, suggest similar cards (same type, same color, same set, same mana cost, or cards often paired with the input card in existing decks).

2. Card Import from Files

Supported Formats

• XLSX (Excel)
• CSV
• JSON
• ODS (OpenDocument Spreadsheet)

Import Flow

1. User uploads a file (XLSX, CSV, JSON, or ODS)
2. Backend parses the file — each row is treated as one card entry
3. Duplicates within the file are allowed (each row → one card instance)
4. For each card name in the file, backend performs fuzzy matching against the Cards PSQL table
5. Matched cards are stored in a user-owned card table with metadata:
   • user_id (FK → users)
   • card_id (FK → mtg_cards from mtgdata, matched via fuzzy search)
   • raw_name (original name from file, for traceability)
   • confidence (match score from fuzzy search)
   • imported_at (timestamp)

Fuzzy Search Requirements

• Must handle spelling errors (e.g., "Wondrrland" → "Wonderland")
• Must handle American vs British English differences (e.g., "color" vs "colour", "armor" vs "armour")
• Use a fuzzy string matching library (e.g., python-Levenshtein, thefuzz/fuzzymatch)
• Confidence threshold to auto-accept vs flag for manual review
• Bulk matching: process all cards in the file in a single batch operation

API Endpoints to Implement

• POST /cards/import — Upload file for import
• GET /cards/import/{import_id}/status — Check import progress/status
• GET /cards/import/{import_id}/results — Get match results with confidence scores
• POST /cards/import/{import_id}/confirm — Confirm import (save to user card table)
• GET /user/cards — List user's imported/owned cards
• DELETE /user/cards/{card_import_id} — Remove from user card table

3. Multiplayer Play Feature (Separate Backend)

Architecture Decision

The multiplayer gameplay feature will live in a separate backend codebase to ensure smooth, independent development. This backend will communicate with the card backend via:

• API Calls: For authentication, user data, deck retrieval, card lookups
• Direct PSQL Queries: For card data and user deck data

Integration Points

• Card Backend API (http://backend:8000):

  • GET /api/cards/{card_id} — Get card details for in-game display
  • GET /api/cards/search?q=... — Search cards during gameplay
  • GET /api/sets/ — List available sets for game formatting

• PSQL Direct Access (via shared connection string):

  • mtgdata database — Read card data (cards, sets, etc.)
  • mtgonline database — Read user decks (for deck validation, game setup)

API Endpoints for Play Backend

• POST /play/decks/{deck_id}/validate — Validate deck against card database
• GET /play/users/{user_id}/decks — Get user's FINAL decks for selection
• GET /play/cards/{card_id} — Get card details for game board display

Note: The play backend is out of scope for this document. See separate codebase/repository when ready.

Summary of Work Required in This Backend

Feature	Database	API	Notes
User deck CRUD	mtgonline (new tables)	Full CRUD + finalize	DRAFT/FINAL status
Card search	mtgdata (existing)	Search endpoint	Leverages existing card DB
Card suggestions	mtgonline + mtgdata	Suggestion endpoint	Based on similar cards
File import (XLSX/CSV/JSON/ODS)	mtgonline (new user cards table)	Upload + confirm	Fuzzy match required
Fuzzy matching service	N/A	Internal service	Handles spelling + EN variants
Play backend integration	Read-only access	API consumer	Separate codebase

Files to Create/Modify

• New models: models/user_deck.py, models/user_card.py
• New migrations: Alembic migrations for new tables
• New router: routers/decks.py, routers/card_import.py
• New service: services/fuzzy_card_matcher.py
• New service: services/deck_suggestion.py
• New schema: schemas/deck.py, schemas/card_import.py
• Update core/database.py if new engine needed
• Update requirements.txt with fuzzy matching libraries

See ROADMAP.md Phase 2 for detailed task breakdown.

User Data Schema & API

Database Models (16 Tables)

• users - User accounts with authentication
• decks - User decks (DRAFT/FINAL status)
• cards - User card collections
• card_ownership - Card ownership tracking
• win_streaks - Win/loss statistics
• game_replays - Saved game replays (JSONB)
• groups - User groups
• group_members - Group membership
• networks - Network accounts (Twitch, X, YouTube)
• network_credentials - Network login info
• preferences - User preferences (JSONB)
• activity_log - User activity tracking (JSONB)
• suggested_cards - Card suggestions
• folders - Deck organization
• game_logs - Game audit trail
• user_decks - User deck storage (DRAFT/FINAL)

API Endpoints

Replays (/api/v1/user-data/replays)

• POST /api/v1/user-data/replays/ - Save replay
• GET /api/v1/user-data/replays/{replay_id} - Get replay
• DELETE /api/v1/user-data/replays/{replay_id} - Delete replay

Card Collection (/api/v1/user-data/cards)

• GET /api/v1/user-data/cards/ - List user's cards
• POST /api/v1/user-data/cards/ - Add card to collection
• DELETE /api/v1/user-data/cards/{card_id} - Remove card

Groups (/api/v1/user-data/groups)

• GET /api/v1/user-data/groups/ - List user's groups
• POST /api/v1/user-data/groups/ - Create group
• PATCH /api/v1/user-data/groups/{group_id} - Update group
• DELETE /api/v1/user-data/groups/{group_id} - Delete group

Networks (/api/v1/user-data/networks)

• GET /api/v1/user-data/networks/ - List network accounts
• POST /api/v1/user-data/networks/ - Add network
• PATCH /api/v1/user-data/networks/{network_id} - Update network
• DELETE /api/v1/user-data/networks/{network_id} - Remove network

Preferences (/api/v1/user-data/preferences)

• GET /api/v1/user-data/preferences/ - Get preferences
• PATCH /api/v1/user-data/preferences/ - Update preferences

Activity Log (/api/v1/user-data/activity)

• GET /api/v1/user-data/activity/ - List activity
• POST /api/v1/user-data/activity/ - Add activity entry

Alembic Migrations

• Async configuration with run_sync for database operations
• Initial migration: 001_initial_user_schema.py creates all 16 tables
• Migration script: scripts/run_migrations.sh runs on container startup
• JSONB columns used for flexible data storage (replay_data, activity_data, preferences)

Architecture Decisions

• CASCADE deletes for data integrity in related tables
• Composite unique constraints for card collection uniqueness
• RESTful API design with pagination support
• JWT authentication for all endpoints
• Permission checks for group/network management

State File

Current state saved at: /home/wall-o/projects/mtgonline/state.json

{
  "task_description": "Create Alembic migration setup for user data schema and complete API endpoints for all user data features",
  "current_step": "Phase 3 completed - All API endpoints created with comprehensive documentation",
  "commit_hash": "c42d7ca",
  "timestamp": "2026-07-22T23:23:00-04:00"
}

Access Information

• Backend API: http://localhost:5555
• User Data API: http://localhost:5555/api/v1/user-data
• Swagger Docs: http://localhost:5555/docs
• Health Check: http://localhost:5555/health
• Gitea: https://git.optimex.systems/admin/mtgonline

Key Files to Review

Current State

1. /home/wall-o/projects/mtgonline/state.json - Current project state
2. /home/wall-o/projects/mtgonline/README.md - Project documentation
3. /home/wall-o/projects/mtgonline/backend/README.md - Backend documentation

Architecture & Configuration

4. /home/wall-o/projects/mtgonline/docker-compose.dev.yml - Docker configuration
5. /home/wall-o/projects/mtgonline/backend/app/core/settings.py - Application settings
6. /home/wall-o/projects/mtgonline/backend/app/main.py - FastAPI entry point (user-data mounted at /api/v1/user-data)

User Data Models

7. /home/wall-o/projects/mtgonline/backend/app/models/user_data.py - All user data models (16 tables)
8. /home/wall-o/projects/mtgonline/backend/alembic/versions/001_initial_user_schema.py - Migration script

API Endpoints

9. /home/wall-o/projects/mtgonline/backend/app/routers/user_data.py - User data API endpoints
10. /home/wall-o/projects/mtgonline/backend/app/schemas/user_data_schemas.py - Pydantic schemas for user data

Documentation

11. /home/wall-o/projects/mtgonline/backend/API_DOCUMENTATION.md - Comprehensive API documentation
12. /home/wall-o/projects/mtgonline/backend/TEST_PLAN.md - Migration test plan
13. /home/wall-o/projects/mtgonline/ROADMAP.md - Complete feature roadmap and timeline
14. /home/wall-o/projects/mtgonline/STATEMENT_OF_INTENT.md - Project vision and objectives

Environment

• OS: Linux 6.8.0-136-generic (x86_64)
• Docker: Available
• Python: 3.12.3
• Working Directory: /home/wall-o/projects/mtgonline

---

Last Updated: 2026-07-22T23:23:00-04:00 Status: Phase 1-3 Complete. Ready for Phase 4: Testing and Deployment.

Next Action: Test migration execution in container, run API tests against all endpoints, add rate limiting, create integration tests, deploy to staging environment.

Timeline:

Phase	Duration	Status
Phase 1: Backend Foundation	2 weeks	✅ Complete
Phase 2: Frontend Development	4 weeks	Not Started
Phase 3: Integration & Polish	2 weeks	Not Started
Phase 4: Deployment & Production	1 week	Not Started
Phase 5: Advanced Features	Ongoing	Future