mtgonline/backend

MTG Online Backend

Python FastAPI application for processing MTGJSON card data and managing the MTG Online platform backend.

Architecture

Core Components

The backend consists of several key layers:

Core Layer (app/core/)

• settings.py — Application configuration using pydantic-settings with environment variable overrides
• database.py — Dual PostgreSQL engine setup (mtgonline app DB + mtgdata MTG cards DB)
• redis_client.py — Redis connection and caching utilities

Models (app/models/)

• models.py — SQLAlchemy ORM models for application data (users, decks, auth)
• mtg_models.py — ORM models for MTG card data

Services (app/services/)

• mtgjson_manager.py — Primary MTGJSON data pipeline (download, unzip, upsert)
• mtgjson_downloader.py — HTTP client for MTGJSON API
• mtgjson_loader.py — Data loading and transformation
• mtgjson_uploader.py — Database upsert operations
• card_database.py — Card data access layer
• game_server.py — Game state management
• deck_parser.py — Deck list parsing and validation

Routers (app/routers/)

• auth.py — JWT authentication endpoints
• users.py — User management
• decks.py — Deck CRUD operations
• rooms.py — Game room management
• games.py — Game state endpoints
• admin.py — Admin tools
• card_router.py — MTG card data API
• interactions.py — Card interaction engine
• refresh.py — MTGJSON data refresh endpoint
• ws.py — WebSocket support

Schemas (app/schemas/)

• schemas.py — Pydantic models for request/response validation
• proto_messages.py — Protocol buffer message definitions
• protocol_constants.py — MTG protocol constants

Data Flow

MTGJSON API (https://mtgjson.com/api/v5/)
    ↓ download
MTGJSON files (AllPrintings.psql, AllIdentifiers.json, etc.)
    ↓ load/transform
PostgreSQL (mtgdata database)
    ↓ query
REST API / Swagger Docs

Database Setup

Primary Database (mtgonline)

• Purpose: Application data (users, decks, auth tokens)
• Connection: postgresql+asyncpg://mtgonline_user:mtgonline_password@postgres:5432/mtgonline
• Port: 5432 (internal), 5432:5432 (host)

MTG Data Database (mtgdata)

• Purpose: MTG card data, sets, refresh logs
• Connection: postgresql+asyncpg://mtgonline_user:mtgonline_password@mtgdata:5432/mtgdata
• Port: 5432 (internal), 5433:5432 (host)
• Tables:
  • mtg_sets — Card sets metadata
  • mtg_cards — Individual card data
  • mtg_refresh_log — Refresh history and status

Redis

• Purpose: Caching, session management
• Connection: redis://redis:6379
• Port: 6379 (internal), 6379:6379 (host)

MTGJSON Data Pipeline

Downloaded Files

The backend downloads these files from MTGJSON v5:

• AllPrintings.psql — Main card data (PostgreSQL format)
• AllIdentifiers.json — Card identifiers (Multiverse, Scryfall, etc.)
• Keywords.json — Card keywords
• CardTypes.json — Card type definitions
• AllDeckFiles.zip — Deck files (must be unzipped)

Refresh Logic

1. On Startup: Checks mtg_refresh_log for existing data
2. If No Data: Downloads and loads all MTGJSON files (may take minutes)
3. Manual Refresh: POST /refresh endpoint triggers immediate reload
4. Logging: All refreshes logged to mtg_refresh_log with status, timing, and counts

Environment Variables

Variable	Default	Description
DATABASE_URL	postgresql+asyncpg://mtgonline_user:mtgonline_password@postgres:5432/mtgonline	Primary database
MTG_DATABASE_URL	postgresql+asyncpg://mtgonline_user:mtgonline_password@mtgdata:5432/mtgdata	MTG data database
REDIS_URL	redis://redis:6379	Redis connection
DATA_DIR	/app/data	MTGJSON files directory
UPLOAD_DIR	/app/uploads	User uploads directory
DEBUG	False	Enable debug logging
LOG_LEVEL	INFO	Logging level
SECRET_KEY	change-me-in-production	JWT secret
JWT_SECRET_KEY	change-me-in-production	JWT signing key

Running the Backend

Docker (Recommended)

# Build image
cd backend
docker build -t mtgonline-backend:latest .

# Run with dependencies
docker compose -f ../docker-compose.dev.yml up -d backend

Local Development

cd backend

# Create venv
python -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Set environment variables
export DATABASE_URL="postgresql+asyncpg://mtgonline_user:mtgonline_password@localhost:5432/mtgonline"
export MTG_DATABASE_URL="postgresql+asyncpg://mtgonline_user:mtgonline_password@localhost:5433/mtgdata"
export REDIS_URL="redis://localhost:6379"

# Run server
python -m uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

API Endpoints

Health & Status

• GET /health — Health check with MTGJSON status
• GET / — API info

Authentication

• POST /auth/login — User login
• POST /auth/register — User registration
• POST /auth/refresh — Refresh JWT
• GET /auth/me — Current user

Users

• GET /users/{user_id} — Get user
• PATCH /users/{user_id} — Update user
• POST /users/{user_id}/ban — Ban user (admin)

Decks

• GET /decks/ — List decks
• POST /decks/ — Create deck
• GET /decks/{deck_id} — Get deck
• PATCH /decks/{deck_id} — Update deck
• DELETE /decks/{deck_id} — Delete deck

MTG Cards

• GET /api/cards/ — Search cards
• GET /api/cards/{card_id} — Get card
• GET /api/sets/ — List sets

Admin

• GET /admin/users — List all users
• GET /admin/bans — List bans
• POST /admin/bans — Create ban

Data Management

• POST /refresh — Trigger MTGJSON refresh

WebSocket

• WS /ws/{room_id} — Real-time game communication

Utility Scripts

Located in scripts/:

• download_mtgjson.py — Manual MTGJSON download
• check_mtgjson_status.py — Verify data freshness
• verify_mtgjson_data.py — Data validation
• inspect_db.py — Database inspection
• load_mtgjson_data.py — Data loading

Testing

cd backend

# Run tests
pytest

# Run with coverage
pytest --cov=app --cov-report=html

Project Structure

backend/
├── app/
│   ├── __init__.py
│   ├── main.py                 # FastAPI app entry
│   ├── core/
│   │   ├── settings.py         # Configuration
│   │   ├── database.py         # Database engines
│   │   └── redis_client.py     # Redis utilities
│   ├── models/
│   │   ├── models.py           # App models
│   │   └── mtg_models.py       # MTG models
│   ├── routers/
│   │   ├── auth.py             # Auth endpoints
│   │   ├── users.py            # User endpoints
│   │   ├── decks.py            # Deck endpoints
│   │   ├── rooms.py            # Room endpoints
│   │   ├── games.py            # Game endpoints
│   │   ├── admin.py            # Admin endpoints
│   │   ├── card_router.py      # Card API
│   │   ├── interactions.py     # Card interactions
│   │   ├── refresh.py          # Data refresh
│   │   └── ws.py               # WebSocket
│   ├── schemas/
│   │   ├── schemas.py          # Pydantic models
│   │   ├── proto_messages.py   # Protocol messages
│   │   └── protocol_constants.py
│   └── services/
│       ├── mtgjson_manager.py  # MTGJSON pipeline
│       ├── mtgjson_downloader.py
│       ├── mtgjson_loader.py
│       ├── mtgjson_uploader.py
│       ├── card_database.py    # Card data access
│       ├── game_server.py      # Game logic
│       └── deck_parser.py      # Deck parsing
├── scripts/                    # Utility scripts
├── tests/                      # Test suite
├── Dockerfile                  # Container build
├── requirements.txt            # Python dependencies
├── pyproject.toml              # Ruff config
├── .env.example                # Environment template
└── setup_db.py                 # Database setup script

Troubleshooting

Backend can't connect to databases

• Verify all services are running: docker compose -f ../docker-compose.dev.yml ps
• Check logs: docker compose -f ../docker-compose.dev.yml logs backend
• Ensure environment variables match docker-compose.dev.yml

MTGJSON download fails

• Check network connectivity to mtgjson.com
• Verify DATA_DIR has write permissions
• Check disk space: df -h
• Manual download: python scripts/download_mtgjson.py

Database tables missing

• Run initialization: docker exec -i mtgdata psql -U mtgonline_user mtgdata < /path/to/scripts/init-mtgdata.sql
• Check tables: docker exec mtgdata psql -U mtgonline_user mtgdata -c "\dt"

License

MIT