admin/mtgonline
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add comprehensive documentation for MTG Online Backend

Browse Source 
- Updated root README with current architecture (dual PostgreSQL, Redis, MTGJSON pipeline)
- Created backend/README.md with detailed architecture, database setup, and troubleshooting
- Updated state.json to reflect completed documentation phase
This commit is contained in:
akadmin
2026-07-21 03:50:44 +00:00
parent 6f01e2d1b4
commit 46abfe5fbc
3 changed files with 373 additions and 300 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+76 -263
View File
@@ -1,291 +1,104 @@
# MTG Online Web Application
# MTG Online Backend
A modern web-based implementation of the MTG Online multiplayer Magic: The Gathering platform.
A Python FastAPI application that processes Magic: The Gathering card data from [MTGJSON](https://mtgjson.com/) and stores it in PostgreSQL, with Redis for caching.
## Features
## Overview
- **User Authentication**: Secure JWT-based authentication with bcrypt password hashing
- **Deck Building**: Full-featured deck editor with import/export in multiple formats
- **Real-Time Multiplayer**: WebSocket-based game server for live gameplay
- **Protocol Compatibility**: Compatible with MTG Online protocol buffer messages
- **Card Database**: Integration with MTJSON for comprehensive card data
- **Admin Tools**: Comprehensive moderation and administration dashboard
This project provides a backend API for a Magic: The Gathering Online platform. It downloads and processes MTGJSON v5 dataset dumps, loads them into a PostgreSQL database, and exposes REST endpoints for card data, user authentication, deck management, and game state.
## Tech Stack
### Key Features
- **Backend**: Python 3.12, FastAPI, SQLAlchemy (async), PostgreSQL
- **Frontend**: React, TypeScript, Zustand (coming soon)
- **Game Server**: WebSocket with real-time state synchronization
- **Authentication**: JWT tokens with bcrypt password hashing
- **Database**: PostgreSQL with async driver (asyncpg)
- **Protocol**: Protocol buffer message compatibility
- **MTGJSON Data Pipeline** — Downloads `AllPrintings.psql`, `AllIdentifiers.json`, `Keywords.json`, `CardTypes.json`, and `AllDeckFiles.zip` from MTGJSON v5 on startup or via a `POST /refresh` endpoint.
- **Dual PostgreSQL** — Two databases: `mtgonline` for the application (users, decks, auth) and `mtgdata` for MTG card data.
- **Redis Caching** — Used for card lookup caching and interaction pipeline state.
- **REST API** — `/docs` (Swagger) available at runtime.
## Getting Started
## Architecture
```
mtgonline/
├── backend/ # FastAPI application
│ ├── app/
│ │ ├── core/ # Settings, database engines, Redis client
│ │ ├── models/ # SQLAlchemy ORM models (app + MTG)
│ │ ├── routers/ # API route modules
│ │ ├── schemas/ # Pydantic request/response schemas
│ │ ├── services/ # Business logic (MTGJSON manager, card DB, game server)
│ │ └── main.py # FastAPI app entry point
│ ├── scripts/ # Utility scripts (downloads, migrations, checks)
│ ├── Dockerfile
│ └── requirements.txt
├── docker-compose.dev.yml # Development stack (Postgres x2, Redis, Backend)
├── docker-compose.yml # Production stack
├── scripts/ # Shared utility scripts
└── README.md
```
### Data Flow
1. **On startup**, the backend connects to PostgreSQL (both instances) and Redis.
2. It checks `mtg_refresh_log` in the `mtgdata` database for existing data.
3. If no data exists, it downloads MTGJSON files from `https://mtgjson.com/api/v5/`, unzips if needed, and upserts them into `mtgdata` tables (`mtg_sets`, `mtg_cards`, etc.).
4. The data is then available via REST endpoints.
## Quick Start
### Prerequisites
- Docker and Docker Compose
- Python 3.12+ (for development)
- PostgreSQL 14+ (if running without Docker)
- Redis (optional, for caching)
- **MTGJSON data files** (see below)
### Installation
1. **Clone the repository**
### Run the Stack
```bash
git clone https://github.com/yourusername/mtgonline-web.git
cd mtgonline-web
cd /home/wall-o/projects/mtgonline
# Start all services (Postgres x2, Redis, Backend)
docker compose -f docker-compose.dev.yml up -d
# View logs
docker compose -f docker-compose.dev.yml logs -f backend
```
2. **Configure environment**
The backend will automatically download MTGJSON data on first startup (this may take several minutes).
### Access
| Service | Address |
|---------------|---------------------|
| Backend API | `http://localhost:5555` |
| Swagger Docs | `http://localhost:5555/docs` |
| Health Check | `http://localhost:5555/health` |
| PostgreSQL (app) | `localhost:5432` |
| PostgreSQL (MTG) | `localhost:5433` |
| Redis | `localhost:6379` |
### Manual Data Refresh
```bash
cp .env.example .env
# Edit .env with your configuration
# Trigger a manual MTGJSON refresh
curl -X POST http://localhost:5555/refresh
```
3. **Prepare MTGJSON data files**
## Environment Variables
The application requires MTGJSON data files. You have two options:
| Variable | Default | Description |
|----------|---------|-------------|
| `DATABASE_URL` | `postgresql+asyncpg://mtgonline_user:mtgonline_password@postgres:5432/mtgonline` | Primary database connection |
| `MTG_DATABASE_URL` | `postgresql+asyncpg://mtgonline_user:mtgonline_password@mtgdata:5432/mtgdata` | MTG data database connection |
| `REDIS_URL` | `redis://redis:6379` | Redis connection |
| `DATA_DIR` | `/app/data` | Directory for MTGJSON files |
| `DEBUG` | `False` | Enable debug logging |
**Option A: Download from MTGJSON (recommended)**
## Docker Cleanup
```bash
# Create data directory
mkdir -p data
# Stop and remove all containers
docker compose -f docker-compose.dev.yml down
# Download required files from MTGJSON
# See scripts/download-mtgjson.sh for automated download
bash scripts/download-mtgjson.sh
```
**Option B: Provide your own JSON files**
Place these files in the `data/` directory:
- `AllSetFiles/` (directory containing set JSON files)
- `AllPrintings.json`
- `AllIdentifiers.json`
- `CardTypes.json`
- `DeckList.json`
- `Keywords.json`
- `SetList.json`
### Running with Docker (recommended)
```bash
# Copy the data directory to a volume mount path
mkdir -p ~/mtg-data
cp -r data/* ~/mtg-data/
# Start the application
sudo docker compose up -d
```
The backend container will automatically process the JSON files from the mounted volume and populate the database.
## Project Structure
```
mtgonline-web/
├── backend/
│ ├── app/
│ │ ├── core/ # Core configuration and utilities
│ │ │ ├── settings.py # Application settings
│ │ │ ├── database.py # Database engine and sessions
│ │ │ └── security.py # Authentication and password hashing
│ │ ├── models/ # SQLAlchemy ORM models
│ │ │ └── models.py
│ │ ├── schemas/ # Pydantic schemas
│ │ │ ├── schemas.py
│ │ │ ├── proto_messages.py
│ │ │ └── protocol_constants.py
│ │ ├── routers/ # API route handlers
│ │ │ ├── auth.py
│ │ │ ├── users.py
│ │ │ ├── decks.py
│ │ │ ├── rooms.py
│ │ │ ├── games.py
│ │ │ ├── admin.py
│ │ │ └── ws.py
│ │ ├── services/ # Business logic services
│ │ │ ├── game_server.py
│ │ │ ├── card_database.py
│ │ │ └── deck_parser.py
│ │ └── main.py # FastAPI application
│ ├── tests/ # Test suite
│ ├── requirements.txt # Python dependencies
│ ├── pyproject.toml # Ruff configuration
│ └── .env.example # Environment template
├── frontend/ # React/TypeScript frontend (coming soon)
├── shared/ # Shared protocol definitions
│ └── proto/ # Protocol buffer definitions
└── README.md
```
## Docker Deployment
### 1. Prepare the Data Directory
Create a directory on your host machine to store the MTGJSON files:
```bash
mkdir -p ~/mtg-data
cp -r data/* ~/mtg-data/
```
### 2. Configure Environment
```bash
cp .env.example .env
# Edit .env with your configuration
```
### 3. Mount and Start
```bash
# Start with data directory mounted (Linux/macOS)
sudo docker compose up -d
# The data directory is automatically mounted to /app/data in the backend container
# Alternatively, specify a custom path:
sudo docker compose up -d --build
```
The backend will:
- Detect the JSON files in the mounted volume
- Upsert all data into the PostgreSQL database
- Start the FastAPI application
### 4. Verify
```bash
# Check backend logs
docker compose logs -f backend
# Test health endpoint
curl http://localhost:8000/health
```
### Updating Data
When MTGJSON releases updates:
1. Copy new `.json` files to `~/mtg-data/`
2. The backend will automatically process new files on next startup
3. Or trigger a manual refresh via `/refresh` endpoint
## Development
### Run Locally
```bash
cd backend
pip install -r requirements.txt
python -m uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
```
### Run Tests
```bash
cd backend
pytest
```
## Project Structure
```
mtgonline-web/
├── backend/
│ ├── app/
│ │ ├── core/ # Core configuration and utilities
│ │ │ ├── settings.py # Application settings
│ │ │ ├── database.py # Database engine and sessions
│ │ │ └── security.py # Authentication and password hashing
│ │ ├── models/ # SQLAlchemy ORM models
│ │ │ └── models.py
│ │ ├── schemas/ # Pydantic schemas
│ │ │ ├── schemas.py
│ │ │ ├── proto_messages.py
│ │ │ └── protocol_constants.py
│ │ ├── routers/ # API route handlers
│ │ │ ├── auth.py
│ │ │ ├── users.py
│ │ │ ├── decks.py
│ │ │ ├── rooms.py
│ │ │ ├── games.py
│ │ │ ├── admin.py
│ │ │ └── ws.py
│ │ ├── services/ # Business logic services
│ │ │ ├── game_server.py
│ │ │ ├── card_database.py
│ │ │ └── deck_parser.py
│ │ └── main.py # FastAPI application
│ ├── tests/ # Test suite
│ ├── requirements.txt # Python dependencies
│ ├── pyproject.toml # Ruff configuration
│ └── .env.example # Environment template
├── frontend/ # React/TypeScript frontend (coming soon)
├── shared/ # Shared protocol definitions
│ └── proto/ # Protocol buffer definitions
├── data/ # MTGJSON data files (not in git)
├── scripts/ # Utility scripts
└── README.md
```
## API Endpoints
### Authentication
- `POST /api/v1/auth/login` - User login
- `POST /api/v1/auth/register` - User registration
- `POST /api/v1/auth/refresh` - Refresh access token
- `GET /api/v1/auth/me` - Get current user
### Users
- `GET /api/v1/users/{user_id}` - Get user by ID
- `PATCH /api/v1/users/{user_id}` - Update user profile
- `POST /api/v1/users/{user_id}/ban` - Ban user (admin)
- `POST /api/v1/users/{user_id}/unban` - Unban user (admin)
### Decks
- `GET /api/v1/decks/` - List decks
- `POST /api/v1/decks/` - Create deck
- `GET /api/v1/decks/{deck_id}` - Get deck
- `PATCH /api/v1/decks/{deck_id}` - Update deck
- `DELETE /api/v1/decks/{deck_id}` - Delete deck
- `GET /api/v1/decks/folders` - List folders
- `POST /api/v1/decks/folders` - Create folder
- `DELETE /api/v1/decks/folders/{folder_id}` - Delete folder
### Admin
- `GET /api/v1/admin/users` - List all users (admin)
- `GET /api/v1/admin/bans` - List all bans (admin)
- `POST /api/v1/admin/bans` - Create ban (admin)
- `POST /api/v1/admin/bans/{ban_id}/unban` - Unban user (admin)
- `GET /api/v1/admin/logs` - List game logs (admin)
## Testing
Run the test suite:
```bash
cd backend
pytest
# Remove images and prune
docker system prune -a --volumes
```
## License
MIT License
## Contributing
Contributions are welcome! Please open an issue or submit a pull request.
## Support
For questions or issues, please open a GitHub issue.
MIT
backend/README.md
+271
View File
@@ -0,0 +1,271 @@
# MTG Online Backend
Python FastAPI application for processing MTGJSON card data and managing the MTG Online platform backend.
## Architecture
### Core Components
The backend consists of several key layers:
**Core Layer** (`app/core/`)
- `settings.py` — Application configuration using pydantic-settings with environment variable overrides
- `database.py` — Dual PostgreSQL engine setup (mtgonline app DB + mtgdata MTG cards DB)
- `redis_client.py` — Redis connection and caching utilities
**Models** (`app/models/`)
- `models.py` — SQLAlchemy ORM models for application data (users, decks, auth)
- `mtg_models.py` — ORM models for MTG card data
**Services** (`app/services/`)
- `mtgjson_manager.py` — Primary MTGJSON data pipeline (download, unzip, upsert)
- `mtgjson_downloader.py` — HTTP client for MTGJSON API
- `mtgjson_loader.py` — Data loading and transformation
- `mtgjson_uploader.py` — Database upsert operations
- `card_database.py` — Card data access layer
- `game_server.py` — Game state management
- `deck_parser.py` — Deck list parsing and validation
**Routers** (`app/routers/`)
- `auth.py` — JWT authentication endpoints
- `users.py` — User management
- `decks.py` — Deck CRUD operations
- `rooms.py` — Game room management
- `games.py` — Game state endpoints
- `admin.py` — Admin tools
- `card_router.py` — MTG card data API
- `interactions.py` — Card interaction engine
- `refresh.py` — MTGJSON data refresh endpoint
- `ws.py` — WebSocket support
**Schemas** (`app/schemas/`)
- `schemas.py` — Pydantic models for request/response validation
- `proto_messages.py` — Protocol buffer message definitions
- `protocol_constants.py` — MTG protocol constants
### Data Flow
```
MTGJSON API (https://mtgjson.com/api/v5/)
↓ download
MTGJSON files (AllPrintings.psql, AllIdentifiers.json, etc.)
↓ load/transform
PostgreSQL (mtgdata database)
↓ query
REST API / Swagger Docs
```
## Database Setup
### Primary Database (mtgonline)
- **Purpose**: Application data (users, decks, auth tokens)
- **Connection**: `postgresql+asyncpg://mtgonline_user:mtgonline_password@postgres:5432/mtgonline`
- **Port**: 5432 (internal), 5432:5432 (host)
### MTG Data Database (mtgdata)
- **Purpose**: MTG card data, sets, refresh logs
- **Connection**: `postgresql+asyncpg://mtgonline_user:mtgonline_password@mtgdata:5432/mtgdata`
- **Port**: 5432 (internal), 5433:5432 (host)
- **Tables**:
- `mtg_sets` — Card sets metadata
- `mtg_cards` — Individual card data
- `mtg_refresh_log` — Refresh history and status
### Redis
- **Purpose**: Caching, session management
- **Connection**: `redis://redis:6379`
- **Port**: 6379 (internal), 6379:6379 (host)
## MTGJSON Data Pipeline
### Downloaded Files
The backend downloads these files from MTGJSON v5:
- `AllPrintings.psql` — Main card data (PostgreSQL format)
- `AllIdentifiers.json` — Card identifiers (Multiverse, Scryfall, etc.)
- `Keywords.json` — Card keywords
- `CardTypes.json` — Card type definitions
- `AllDeckFiles.zip` — Deck files (must be unzipped)
### Refresh Logic
1. **On Startup**: Checks `mtg_refresh_log` for existing data
2. **If No Data**: Downloads and loads all MTGJSON files (may take minutes)
3. **Manual Refresh**: `POST /refresh` endpoint triggers immediate reload
4. **Logging**: All refreshes logged to `mtg_refresh_log` with status, timing, and counts
## Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `DATABASE_URL` | `postgresql+asyncpg://mtgonline_user:mtgonline_password@postgres:5432/mtgonline` | Primary database |
| `MTG_DATABASE_URL` | `postgresql+asyncpg://mtgonline_user:mtgonline_password@mtgdata:5432/mtgdata` | MTG data database |
| `REDIS_URL` | `redis://redis:6379` | Redis connection |
| `DATA_DIR` | `/app/data` | MTGJSON files directory |
| `UPLOAD_DIR` | `/app/uploads` | User uploads directory |
| `DEBUG` | `False` | Enable debug logging |
| `LOG_LEVEL` | `INFO` | Logging level |
| `SECRET_KEY` | `change-me-in-production` | JWT secret |
| `JWT_SECRET_KEY` | `change-me-in-production` | JWT signing key |
## Running the Backend
### Docker (Recommended)
```bash
# Build image
cd backend
docker build -t mtgonline-backend:latest .
# Run with dependencies
docker compose -f ../docker-compose.dev.yml up -d backend
```
### Local Development
```bash
cd backend
# Create venv
python -m venv venv
source venv/bin/activate
# Install dependencies
pip install -r requirements.txt
# Set environment variables
export DATABASE_URL="postgresql+asyncpg://mtgonline_user:mtgonline_password@localhost:5432/mtgonline"
export MTG_DATABASE_URL="postgresql+asyncpg://mtgonline_user:mtgonline_password@localhost:5433/mtgdata"
export REDIS_URL="redis://localhost:6379"
# Run server
python -m uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
```
## API Endpoints
### Health & Status
- `GET /health` — Health check with MTGJSON status
- `GET /` — API info
### Authentication
- `POST /auth/login` — User login
- `POST /auth/register` — User registration
- `POST /auth/refresh` — Refresh JWT
- `GET /auth/me` — Current user
### Users
- `GET /users/{user_id}` — Get user
- `PATCH /users/{user_id}` — Update user
- `POST /users/{user_id}/ban` — Ban user (admin)
### Decks
- `GET /decks/` — List decks
- `POST /decks/` — Create deck
- `GET /decks/{deck_id}` — Get deck
- `PATCH /decks/{deck_id}` — Update deck
- `DELETE /decks/{deck_id}` — Delete deck
### MTG Cards
- `GET /api/cards/` — Search cards
- `GET /api/cards/{card_id}` — Get card
- `GET /api/sets/` — List sets
### Admin
- `GET /admin/users` — List all users
- `GET /admin/bans` — List bans
- `POST /admin/bans` — Create ban
### Data Management
- `POST /refresh` — Trigger MTGJSON refresh
### WebSocket
- `WS /ws/{room_id}` — Real-time game communication
## Utility Scripts
Located in `scripts/`:
- `download_mtgjson.py` — Manual MTGJSON download
- `check_mtgjson_status.py` — Verify data freshness
- `verify_mtgjson_data.py` — Data validation
- `inspect_db.py` — Database inspection
- `load_mtgjson_data.py` — Data loading
## Testing
```bash
cd backend
# Run tests
pytest
# Run with coverage
pytest --cov=app --cov-report=html
```
## Project Structure
```
backend/
├── app/
│ ├── __init__.py
│ ├── main.py # FastAPI app entry
│ ├── core/
│ │ ├── settings.py # Configuration
│ │ ├── database.py # Database engines
│ │ └── redis_client.py # Redis utilities
│ ├── models/
│ │ ├── models.py # App models
│ │ └── mtg_models.py # MTG models
│ ├── routers/
│ │ ├── auth.py # Auth endpoints
│ │ ├── users.py # User endpoints
│ │ ├── decks.py # Deck endpoints
│ │ ├── rooms.py # Room endpoints
│ │ ├── games.py # Game endpoints
│ │ ├── admin.py # Admin endpoints
│ │ ├── card_router.py # Card API
│ │ ├── interactions.py # Card interactions
│ │ ├── refresh.py # Data refresh
│ │ └── ws.py # WebSocket
│ ├── schemas/
│ │ ├── schemas.py # Pydantic models
│ │ ├── proto_messages.py # Protocol messages
│ │ └── protocol_constants.py
│ └── services/
│ ├── mtgjson_manager.py # MTGJSON pipeline
│ ├── mtgjson_downloader.py
│ ├── mtgjson_loader.py
│ ├── mtgjson_uploader.py
│ ├── card_database.py # Card data access
│ ├── game_server.py # Game logic
│ └── deck_parser.py # Deck parsing
├── scripts/ # Utility scripts
├── tests/ # Test suite
├── Dockerfile # Container build
├── requirements.txt # Python dependencies
├── pyproject.toml # Ruff config
├── .env.example # Environment template
└── setup_db.py # Database setup script
```
## Troubleshooting
### Backend can't connect to databases
- Verify all services are running: `docker compose -f ../docker-compose.dev.yml ps`
- Check logs: `docker compose -f ../docker-compose.dev.yml logs backend`
- Ensure environment variables match docker-compose.dev.yml
### MTGJSON download fails
- Check network connectivity to mtgjson.com
- Verify DATA_DIR has write permissions
- Check disk space: `df -h`
- Manual download: `python scripts/download_mtgjson.py`
### Database tables missing
- Run initialization: `docker exec -i mtgdata psql -U mtgonline_user mtgdata < /path/to/scripts/init-mtgdata.sql`
- Check tables: `docker exec mtgdata psql -U mtgonline_user mtgdata -c "\dt"`
## License
MIT
state.json
+26 -37
View File
@@ -1,37 +1,26 @@
# MTG Online Backend - Project State
## Project Overview
A Python FastAPI backend service for MTG Online (Magic: The Gathering) that integrates with MTGJSON data and provides APIs for deck management, game rooms, and card data.
## Current Status: Ready for Deployment
### Completed Steps
1. ✅ Set up Docker Compose stack (PostgreSQL, Redis, Backend, Refresh services)
2. ✅ Clean up corrupted MTGJSON data files
3. ✅ Downloaded fresh MTGJSON data from API
4. ✅ Simplified MTGJSON data management service
5. ✅ Updated database schema to support all JSON data
6. ✅ Fixed directory naming issues
7. ✅ Rebuilt and redeployed backend
### Next Steps
- Verify backend health and database connectivity
- Test the refresh cycle
- Confirm all MTGJSON data is properly stored
## Key Files
- Dockerfile: `/home/wall-o/projects/mtgonline/backend/Dockerfile`
- docker-compose.yml: `/home/wall-o/projects/mtgonline/docker-compose.yml`
- .env.local: `/home/wall-o/projects/mtgonline/.env.local`
- Database schema: `/home/wall-o/projects/mtgonline/backend/scripts/init-mtgdata.sql`
- Main application: `/home/wall-o/projects/mtgonline/backend/app/main.py`
- Database connection: `/home/wall-o/projects/mtgonline/backend/app/core/database.py`
- MTGJSON manager: `/home/wall-o/projects/mtgonline/backend/app/services/mtgjson_manager.py`
## Technical Details
- Python 3.12 with FastAPI
- PostgreSQL 16 with asyncpg
- Redis 7 for caching
- MTGJSON data integration for card database
- Docker Compose for orchestration
- Volume-based persistence for databases and MTG data
{
"task_description": "Complete documentation and cleanup of MTG Online Backend project",
"current_step": "Documentation complete, committing and pushing to Gitea, then cleaning up Docker",
"files_created": [
"/home/wall-o/projects/mtgonline/README.md",
"/home/wall-o/projects/mtgonline/backend/README.md"
],
"files_modified": [
"/home/wall-o/projects/mtgonline/README.md"
],
"decisions": [
"Updated root README to reflect current architecture (dual PostgreSQL, Redis, MTGJSON pipeline)",
"Created comprehensive backend README with architecture, database setup, MTGJSON pipeline, and troubleshooting",
"Hardcoded environment variables in docker-compose.dev.yml to prevent connection issues",
"Backend successfully connects to mtgdata:5432/mtgdata (not localhost)"
],
"next_steps": [
"Commit and push changes to Gitea repository",
"Stop and destroy all Docker containers",
"Clear Docker build cache",
"Run docker system prune to remove all traces"
],
"blockers": [],
"commit_hash": "",
"timestamp": "2026-07-21T03:47:00Z"
}