mtgonline/HANDOFF.md

MTG Online Backend - Handoff Document

Project Overview

Project Name: MTG Online Backend Location: /home/wall-o/projects/mtgonline Purpose: Python FastAPI application that processes Magic: The Gathering card data from MTGJSON and stores it in PostgreSQL databases.

Vision Statement (from STATEMENT_OF_INTENT.md)

MTG Online Web — A modern, web-based implementation of the MTG Online multiplayer Magic: The Gathering platform.

To build a fully-featured, open-source multiplayer Magic: The Gathering platform that runs entirely in modern web browsers, eliminating the need for desktop software installations while maintaining compatibility with the existing MTG Online ecosystem.

Core Objectives

User Experience

• Intuitive, modern interface that rivals native desktop applications
• Real-time multiplayer gameplay with minimal latency
• Seamless deck building with import/export from MTG Online
• Responsive design that works across all screen sizes

Technical Excellence

• Backend: Python 3.12 + FastAPI with async architecture
• Database: PostgreSQL with async SQLAlchemy ORM
• Real-time: WebSocket-based game server for live multiplayer
• Protocol: Full compatibility with MTG Online protocol buffer messages

Feature Parity with Desktop

• User authentication and account management
• Deck creation, editing, and storage (per-user)
• Multiplayer game rooms with real-time state sync
• Card game mechanics (mana, phases, priority, stack)
• Admin/moderation tools (ban, warn, log viewing)
• Card database integration with comprehensive card data

Performance Requirements

• API response times < 100ms for 95% of requests
• WebSocket latency < 50ms for game state updates
• Support 1000+ concurrent users
• Sub-second page loads with proper caching

Success Criteria

• Users can create accounts and authenticate securely
• Users can create, edit, and manage decks (per-user)
• Users can join and play multiplayer games in real-time
• Game state syncs correctly across all connected players
• Admin users can manage accounts and moderate games
• Deck formats are compatible with MTG Online desktop client

Current Status

All planned tasks have been completed:

• ✅ Documentation created (root README.md + backend/README.md)
• ✅ State.json consolidated to project root
• ✅ Commit pushed to Gitea (commit c42d7ca)
• ✅ Docker cleanup completed (all containers, images, volumes removed)
• ✅ User data schema implemented (16 tables with Alembic migrations)
• ✅ Comprehensive API endpoints created (7 routers covering all user data features)

Architecture Summary

Tech Stack

• Backend: Python 3.12, FastAPI, SQLAlchemy (async), asyncpg
• Databases: Dual PostgreSQL (14-alpine)
  • Primary: mtgonline database (users, decks, auth)
  • MTG Data: mtgdata database (card data, sets)
• Cache: Redis 7-alpine
• Protocol: Protocol buffer message compatibility

Service Architecture

mtgonline/
├── backend/                    # FastAPI application
│   ├── mtg_rules_engine/       # MTG rules engine (rules enforcement)
│   │   ├── engine.py           # Core game engine
│   │   ├── rules_engine.py     # Rules engine core
│   │   ├── keywords.py         # Card keywords
│   │   ├── keywords_db.py      # Keywords database
│   │   ├── keyword_validator.py # Keyword validation
│   │   ├── validator.py        # Card validation
│   │   ├── updater.py          # Engine updater
│   │   ├── update_check.py     # Update checker
│   │   ├── test_engine.py      # Engine tests
│   │   └── README.md           # Rules engine docs
│   ├── app/
│   │   ├── core/              # Settings, database engines, Redis client
│   │   ├── models/            # SQLAlchemy ORM models (user_data.py - 16 models)
│   │   ├── routers/           # API route modules
│   │   ├── schemas/           # Pydantic request/response schemas
│   │   ├── services/          # Business logic (MTGJSON manager, card DB, game server, deck parser)
│   │   └── main.py            # FastAPI app entry point
│   ├── alembic/               # Database migrations
│   ├── scripts/               # Utility scripts
│   ├── Dockerfile
│   ├── requirements.txt
│   └── .env.example
├── docker-compose.dev.yml     # Development stack
├── docker-compose.yml         # Production stack
└── README.md                  # Project documentation

Data Flow

1. Backend connects to PostgreSQL (both instances) and Redis on startup
2. Checks mtg_refresh_log in mtgdata database for existing data
3. If no data exists, downloads MTGJSON files from https://mtgjson.com/api/v5/
4. Upserts data into mtgdata tables (mtg_sets, mtg_cards, etc.)
5. Data available via REST endpoints

Key Database Connections

• Primary DB: postgresql+asyncpg://mtgonline_user:mtgonline_password@postgres:5432/mtgonline
• MTG DB: postgresql+asyncpg://mtgonline_user:mtgonline_password@mtgdata:5432/mtgdata
• Redis: redis://redis:6379

Environment Configuration

Docker Compose Dev Environment Variables

environment:
  DATABASE_URL: "postgresql+asyncpg://mtgonline_user:mtgonline_password@postgres:5432/mtgonline"
  MTG_DATABASE_URL: "postgresql+asyncpg://mtgonline_user:mtgonline_password@mtgdata:5432/mtgdata"
  REDIS_URL: "redis://redis:6379"

Service Ports (Host to Container)

• PostgreSQL: 5432:5432
• MTG Data PostgreSQL: 5433:5432
• Redis: 6379:6379
• Backend: 5555:8000

API Endpoints

Health & Status

• GET /health - Health check with MTGJSON status
• GET / - API info

Authentication

• POST /auth/login - User login
• POST /auth/register - User registration
• POST /auth/refresh - Refresh JWT
• GET /auth/me - Current user

Users

• GET /users/{user_id} - Get user
• PATCH /users/{user_id} - Update user
• POST /users/{user_id}/ban - Ban user (admin)

Decks

• GET /decks/ - List decks
• POST /decks/ - Create deck
• GET /decks/{deck_id} - Get deck
• PATCH /decks/{deck_id} - Update deck
• DELETE /decks/{deck_id} - Delete deck

MTG Cards

• GET /api/cards/ - Search cards
• GET /api/cards/{card_id} - Get card
• GET /api/sets/ - List sets

Admin

• GET /admin/users - List all users
• GET /admin/bans - List bans
• POST /admin/bans - Create ban

Data Management

• POST /refresh - Trigger MTGJSON refresh

WebSocket

• WS /ws/{room_id} - Real-time game communication

MTGJSON Data Pipeline

Downloaded Files

• AllPrintings.psql - Main card data (PostgreSQL format)
• AllIdentifiers.json - Card identifiers
• Keywords.json - Card keywords
• CardTypes.json - Card type definitions
• AllDeckFiles.zip - Deck files (unzipped on load)

Refresh Logic

1. On Startup: Checks mtg_refresh_log for existing data
2. If No Data: Downloads and loads all MTGJSON files (may take minutes)
3. Manual Refresh: POST /refresh triggers immediate reload
4. Logging: All refreshes logged to mtg_refresh_log with status, timing, and counts

Git Repository

Gitea Repository: https://git.optimex.systems/admin/mtgonline.git Credentials: Located at /home/wall-o/projects/gitea_credentials.txt Current Branch: main Last Commit: c42d7ca - "feat: implement user data schema and API endpoints"

Commit History

c42d7ca - feat: implement user data schema and API endpoints
  - Add Alembic migration setup with async configuration
  - Create 16 user data models (users, decks, cards, replays, etc.)
  - Implement comprehensive API endpoints with JWT auth
  - Add replay, card collection, group, network, preferences, and activity log routers
  - Include API documentation and migration test plan
46abfe5 - Add comprehensive documentation for MTG Online Backend
6f01e2d - Initial project setup

Running the Project

Start Services

cd /home/wall-o/projects/mtgonline
docker compose -f docker-compose.dev.yml up -d

Check Services

docker compose -f docker-compose.dev.yml ps
docker compose -f docker-compose.dev.yml logs -f backend

Stop Services

docker compose -f docker-compose.dev.yml down

Full Cleanup

# Stop and remove all containers
docker compose -f docker-compose.dev.yml down

# Remove images
docker images rm mtgonline-backend:latest postgres:14-alpine redis:7-alpine

# Clear build cache and prune
docker builder prune -af
docker system prune -af --volumes

Testing

Run Tests

cd /home/wall-o/projects/mtgonline/backend
docker exec -it <backend_container_id> pytest
# OR
cd /home/wall-o/projects/mtgonline/backend
pytest

Database Verification

# Check MTG data tables
docker exec <mtgdata_container_id> psql -U mtgonline_user mtgdata -c "\dt"

# Check refresh log
docker exec <mtgdata_container_id> psql -U mtgonline_user mtgdata -c "SELECT * FROM mtg_refresh_log ORDER BY refresh_time DESC LIMIT 5;"

Troubleshooting

Backend can't connect to databases

• Verify all services are running: docker compose -f docker-compose.dev.yml ps
• Check logs: docker compose -f docker-compose.dev.yml logs backend
• Ensure environment variables match docker-compose.dev.yml

MTGJSON download fails

• Check network connectivity to mtgjson.com
• Verify DATA_DIR has write permissions
• Check disk space: df -h
• Manual download: python scripts/download_mtgjson.py

Database tables missing

• Run initialization: docker exec -i <mtgdata_container_id> psql -U mtgonline_user mtgdata < /path/to/scripts/init-mtgdata.sql
• Check tables: docker exec <mtgdata_container_id> psql -U mtgonline_user mtgdata -c "\dt"

CORS errors

• Check CORS_ORIGINS setting in app/core/settings.py
• Ensure frontend URL matches allowed origins

Phase 2 Complete: Deck Building & Card Management

The v1 backend work has been completed. The following features were implemented:

Completed Features

1. Per-User Deck Building

• user_decks table with DRAFT/FINAL status tracking
• API endpoints: Create, list, get, update, finalize, delete decks
• Card search integrated with MTG card database
• Deck precedents and card suggestion features

2. Card Import from Files

• Supported formats: XLSX, CSV, JSON, ODS
• Fuzzy matching against MTG card database
• Import flow: Upload → Parse → Match → Confirm → Save
• API endpoints: Upload, status check, results, confirm, list user cards

3. Database Schema

• 16 user data tables with Alembic migrations
• Async SQLAlchemy with PostgreSQL
• JSONB columns for flexible data storage

4. API Endpoints

• Authentication: Login, register, refresh, current user
• Users: Get, update, ban (admin)
• Decks: CRUD operations with status management
• Cards: Search, import, user card management
• Admin: User list, ban management
• Data Management: MTGJSON refresh

5. MTG Rules Engine

• Integrated in backend/mtg_rules_engine/
• Core modules: engine.py, rules_engine.py, keywords.py, validator.py
• Supporting modules: keywords_db.py, keyword_validator.py, updater.py, update_check.py
• Test suite: test_engine.py
• Purpose: Enforces MTG game rules (mana, phases, priority, stack resolution, combat) for multiplayer server

Testing & Verification

• ✅ All API endpoints tested and working
• ✅ Database migrations applied successfully
• ✅ Docker deployment verified
• ✅ Integration tests passing
• ✅ Rules engine source files extracted and integrated

---

Session Summary: Phase 3 Architecture Planning

Work Completed

This session focused on planning Phase 3 (Multiplayer Game Server) architecture and updating documentation to reflect key decisions:

1. Architecture Decisions Documented

• Chat System: Pre-built Docker container (Tinode recommended)

  • Option A architecture (separate container)
  • WebSocket-based, frontend connects directly
  • No custom codebase needed

• Audio System: Pre-built Docker container (Kurento or Janus)

  • Option A architecture (separate container)
  • WebRTC-based, frontend connects directly
  • No custom codebase needed

• Game Engine: Python-based (not C++)

  • Codebase being developed separately
  • Will be integrated during Phase 3.1 (Core Game Server)
  • Python module with volume mount

2. Documentation Created

• PHASE_3_PLANNING.md: Comprehensive planning document with:
  • Architecture overview with modular design
  • Chat server solutions (Tinode, SimpleWebSocketChat, Zitadel)
  • Audio server solutions (Kurento, Janus)
  • Python game engine integration approach
  • Development timeline (9 weeks across 4 phases)
  • Docker configuration examples
  • Risk assessment
  • Frontend integration examples

3. HANDOFF.md Updates

• Removed "Next Phase: V1 Backend" section (work completed)
• Added "Phase 2 Complete" section summarizing completed work
• Updated Phase 3 section with new architecture decisions
• Added architecture diagram showing modular design
• Documented integration points for chat and audio servers
• Updated summary table to reflect pre-built solutions

Key Takeaways

1. No custom chat/audio codebase needed - using pre-built Docker solutions
2. Python game engine will be integrated as a module (not C++)
3. Modular architecture allows independent deployment of chat/audio
4. Phase 3.1 will focus on core game server with Python engine integration

Next Steps

1. Await Python game engine codebase delivery
2. Deploy pre-built chat server (Tinode)
3. Deploy pre-built audio server (Kurento or Janus)
4. Begin Phase 3.1: Core Game Server implementation

---

Phase 3: Multiplayer Game Server (In Progress)

3.1 Overview

The multiplayer game server provides a high-end graphic gameplay option for players within the same group (groups are set by admins). This is the core real-time gameplay system that replaces the legacy desktop client.

Architecture Decision: The multiplayer server uses a Python-based game engine (not C++). The game engine codebase is being developed separately and will be integrated during Phase 3.

3.2 Architecture

The multiplayer server follows a modular architecture with pre-built Docker containers for chat and audio:

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Frontend      │────▶│   Game Server   │────▶│   Card Backend  │
│   (React/TS)    │     │   (Python)      │     │   (FastAPI)     │
└─────────────────┘     └─────────────────┘     └─────────────────┘
         │                       │
         │                       │
         ▼                       ▼
┌─────────────────┐     ┌─────────────────┐
│   Chat Server   │     │   Audio Server  │
│   (Pre-built    │     │   (Pre-built    │
│    Docker)      │     │    Docker)      │
└─────────────────┘     └─────────────────┘

Key Decisions:

• Chat: Pre-built Docker container (Tinode recommended) - Option A architecture
• Audio: Pre-built Docker container (Kurento or Janus) - Option A architecture
• Game Engine: Python-based, codebase being developed separately

3.3 Game Engine

A Python-based game engine codes all Magic: The Gathering rules directly into the multiplayer system. This means:

• Cards being played automatically have the appropriate rules applied as they are played
• The engine enforces game mechanics (mana, phases, priority, stack resolution, combat)
• No manual rule implementation per card — the engine handles it all

Status: The game engine codebase is being developed separately and will be integrated during Phase 3.1 (Core Game Server).

Integration Approach:

# game_engine/engine.py
from game_engine.engine import GameEngine

engine = GameEngine()
engine.load_card('lightning-bolt')
result = engine.resolve_effect('lightning-bolt', target='player')

Docker Volume Mount:

volumes:
  - ./game-engine:/app/game_engine

3.4 Graphics

Graphics are handled in the front-end development (React/TypeScript with game board visualization). The multiplayer server provides:

• Game state synchronization
• Card data and metadata
• Real-time updates via WebSocket

The server does not handle rendering — that's the frontend's responsibility.

3.5 Chat System

The multiplayer server uses pre-built Docker containers for chat functionality:

Text-based Chat

• Solution: Tinode (recommended) or SimpleWebSocketChat
• Architecture: Option A (separate container)
• Integration: WebSocket-based, frontend connects directly
• Features: Room management, message persistence, moderation

Docker Deployment (Tinode):

docker run -d \
  --name tinode \
  -p 8080:8080 \
  -v ./tinode-data:/data \
  --restart unless-stopped \
  tinode/tinode

Frontend Integration:

import Tinode from 'tinode-sdk';

const tinode = new Tinode({ socket: 'ws://chat-server:8080' });
await tinode.connect();
await tinode.subscribe('game-room-123');
tinode.on('message', (topic, msg) => console.log('Chat:', msg));
tinode.sendMessage('game-room-123', { text: 'Hello game!' });

Audio-based Chat

• Solution: Kurento Media Server (recommended) or Janus Gateway
• Architecture: Option A (separate container)
• Integration: WebRTC-based, frontend connects directly
• Features: Low latency, scalable, open source

Docker Deployment (Kurento):

docker run -d \
  --name kurento \
  -p 8888:8888 \
  -p 8443:8443 \
  --restart unless-stopped \
  kurento/kurento-media-server:latest

No custom chat/audio codebase needed — both are pre-built solutions.

3.6 Integration with Card Backend

The multiplayer server communicates with the card backend (current FastAPI app) via:

• API Calls: For authentication, user data, deck retrieval, card lookups
• Direct PSQL Queries: For card data and user deck data

Integration Points

• Card Backend API (http://backend:8000):

  • GET /api/cards/{card_id} — Get card details for in-game display
  • GET /api/cards/search?q=... — Search cards during gameplay
  • GET /api/sets/ — List available sets for game formatting

• PSQL Direct Access (via shared connection string):

  • mtgdata database — Read card data (cards, sets, etc.)
  • mtgonline database — Read user decks (for deck validation, game setup)

API Endpoints for Play Backend

• POST /play/decks/{deck_id}/validate — Validate deck against card database
• GET /play/users/{user_id}/decks — Get user's FINAL decks for selection
• GET /play/cards/{card_id} — Get card details for game board display
• POST /play/chat/text — Send text message in game/room (via Tinode)
• POST /play/chat/audio — Manage audio chat sessions (via Kurento)

Summary of Work Required in Phase 3: Multiplayer Game Server

Feature	Database	API	Notes
Game engine integration	N/A (Python codebase)	N/A	Being developed separately, integrate in Phase 3.1
Multiplayer WebSocket server	mtgonline (rooms, games)	WebSocket hub	Python/FastAPI
Game state management	In-memory + DB persistence	State sync	Server-authoritative
MTG rule enforcement	N/A (game engine)	Automatic	Cards auto-apply rules
Text chat backend	N/A (Tinode)	WebSocket	Pre-built Docker container
Audio chat backend	N/A (Kurento)	WebRTC	Pre-built Docker container
Group-based access	mtgonline (groups)	Group validation	Admin-configured groups
Deck validation	mtgdata + mtgonline	Validation endpoint	Against card database
Card backend integration	Read-only	API consumer	Shared DB + REST API

See PHASE_3_PLANNING.md for detailed task breakdown.

User Data Schema & API

Database Models (16 Tables)

• users - User accounts with authentication
• decks - User decks (DRAFT/FINAL status)
• cards - User card collections
• card_ownership - Card ownership tracking
• win_streaks - Win/loss statistics
• game_replays - Saved game replays (JSONB)
• groups - User groups
• group_members - Group membership
• networks - Network accounts (Twitch, X, YouTube)
• network_credentials - Network login info
• preferences - User preferences (JSONB)
• activity_log - User activity tracking (JSONB)
• suggested_cards - Card suggestions
• folders - Deck organization
• game_logs - Game audit trail
• user_decks - User deck storage (DRAFT/FINAL)

API Endpoints

Replays (/api/v1/user-data/replays)

• POST /api/v1/user-data/replays/ - Save replay
• GET /api/v1/user-data/replays/{replay_id} - Get replay
• DELETE /api/v1/user-data/replays/{replay_id} - Delete replay

Card Collection (/api/v1/user-data/cards)

• GET /api/v1/user-data/cards/ - List user's cards
• POST /api/v1/user-data/cards/ - Add card to collection
• DELETE /api/v1/user-data/cards/{card_id} - Remove card

Groups (/api/v1/user-data/groups)

• GET /api/v1/user-data/groups/ - List user's groups
• POST /api/v1/user-data/groups/ - Create group
• PATCH /api/v1/user-data/groups/{group_id} - Update group
• DELETE /api/v1/user-data/groups/{group_id} - Delete group

Networks (/api/v1/user-data/networks)

• GET /api/v1/user-data/networks/ - List network accounts
• POST /api/v1/user-data/networks/ - Add network
• PATCH /api/v1/user-data/networks/{network_id} - Update network
• DELETE /api/v1/user-data/networks/{network_id} - Remove network

Preferences (/api/v1/user-data/preferences)

• GET /api/v1/user-data/preferences/ - Get preferences
• PATCH /api/v1/user-data/preferences/ - Update preferences

Activity Log (/api/v1/user-data/activity)

• GET /api/v1/user-data/activity/ - List activity
• POST /api/v1/user-data/activity/ - Add activity entry

Alembic Migrations

• Async configuration with run_sync for database operations
• Initial migration: 001_initial_user_schema.py creates all 16 tables
• Migration script: scripts/run_migrations.sh runs on container startup
• JSONB columns used for flexible data storage (replay_data, activity_data, preferences)

Architecture Decisions

• CASCADE deletes for data integrity in related tables
• Composite unique constraints for card collection uniqueness
• RESTful API design with pagination support
• JWT authentication for all endpoints
• Permission checks for group/network management

State File

Current state saved at: /home/wall-o/projects/mtgonline/state.json

{
  "project_summary": "MTG Online Backend API with PostgreSQL database. Implements card game platform with deck management, card import, and user data features. Phase 3 (multiplayer game server) is in progress with game engine code incoming.",
  "task_description": "Phase 3: Multiplayer game server with Cockatrice-inspired architecture, game engine integration, text/audio chat, and group-based gameplay",
  "current_step": "Preparation phase - reviewing Cockatrice architecture analysis, awaiting game engine code delivery, planning server scaffolding",
  "commit_hash": "1e7c762",
  "timestamp": "2026-07-24T04:35:00-04:00"
}

Access Information

• Backend API: http://localhost:5555
• User Data API: http://localhost:5555/api/v1/user-data
• Swagger Docs: http://localhost:5555/docs
• Health Check: http://localhost:5555/health
• Gitea: https://git.optimex.systems/admin/mtgonline

Key Files to Review

Current State

1. /home/wall-o/projects/mtgonline/state.json - Current project state
2. /home/wall-o/projects/mtgonline/README.md - Project documentation
3. /home/wall-o/projects/mtgonline/backend/README.md - Backend documentation

Architecture & Configuration

4. /home/wall-o/projects/mtgonline/docker-compose.dev.yml - Docker configuration
5. /home/wall-o/projects/mtgonline/backend/app/core/settings.py - Application settings
6. /home/wall-o/projects/mtgonline/backend/app/main.py - FastAPI entry point (user-data mounted at /api/v1/user-data)

User Data Models

7. /home/wall-o/projects/mtgonline/backend/app/models/user_data.py - All user data models (16 tables)
8. /home/wall-o/projects/mtgonline/backend/alembic/versions/001_initial_user_schema.py - Migration script

API Endpoints

9. /home/wall-o/projects/mtgonline/backend/app/routers/user_data.py - User data API endpoints
10. /home/wall-o/projects/mtgonline/backend/app/schemas/user_data_schemas.py - Pydantic schemas for user data

Documentation

11. /home/wall-o/projects/mtgonline/backend/API_DOCUMENTATION.md - Comprehensive API documentation
12. /home/wall-o/projects/mtgonline/backend/TEST_PLAN.md - Migration test plan
13. /home/wall-o/projects/mtgonline/ROADMAP.md - Complete feature roadmap and timeline
14. /home/wall-o/projects/mtgonline/STATEMENT_OF_INTENT.md - Project vision and objectives

Environment

• OS: Linux 6.8.0-136-generic (x86_64)
• Docker: Available
• Python: 3.12.3
• Working Directory: /home/wall-o/projects/mtgonline

---

Last Updated: 2026-07-25T19:58:00-04:00 Status: Phase 1-2 Complete. Phase 3 (Multiplayer Game Server) in progress. Rules engine integrated in backend/mtg_rules_engine/.

Next Action: Begin Phase 3 — set up multiplayer server scaffolding, integrate rules engine, implement WebSocket hub with chat support.

Timeline:

Phase	Duration	Status
Phase 1: Backend Foundation	2 weeks	✅ Complete
Phase 2: Card Import & Deck Building	2 weeks	✅ Complete
Phase 3: Multiplayer Game Server	TBD	🔄 In Progress
Phase 4: Frontend Integration	TBD	Pending
Phase 5: Advanced Features	Ongoing	Future