admin/mtgonline
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add comprehensive documentation for MTG Online Backend

Browse Source 
- Updated root README with current architecture (dual PostgreSQL, Redis, MTGJSON pipeline)
- Created backend/README.md with detailed architecture, database setup, and troubleshooting
- Updated state.json to reflect completed documentation phase
This commit is contained in:
akadmin
2026-07-21 03:50:44 +00:00
parent 6f01e2d1b4
commit 46abfe5fbc
3 changed files with 373 additions and 300 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+76 -263
View File
@@ -1,291 +1,104 @@
# MTG Online Web Application
# MTG Online Backend
A modern web-based implementation of the MTG Online multiplayer Magic: The Gathering platform.
A Python FastAPI application that processes Magic: The Gathering card data from [MTGJSON](https://mtgjson.com/) and stores it in PostgreSQL, with Redis for caching.
## Features
## Overview
- **User Authentication**: Secure JWT-based authentication with bcrypt password hashing
- **Deck Building**: Full-featured deck editor with import/export in multiple formats
- **Real-Time Multiplayer**: WebSocket-based game server for live gameplay
- **Protocol Compatibility**: Compatible with MTG Online protocol buffer messages
- **Card Database**: Integration with MTJSON for comprehensive card data
- **Admin Tools**: Comprehensive moderation and administration dashboard
This project provides a backend API for a Magic: The Gathering Online platform. It downloads and processes MTGJSON v5 dataset dumps, loads them into a PostgreSQL database, and exposes REST endpoints for card data, user authentication, deck management, and game state.
## Tech Stack
### Key Features
- **Backend**: Python 3.12, FastAPI, SQLAlchemy (async), PostgreSQL
- **Frontend**: React, TypeScript, Zustand (coming soon)
- **Game Server**: WebSocket with real-time state synchronization
- **Authentication**: JWT tokens with bcrypt password hashing
- **Database**: PostgreSQL with async driver (asyncpg)
- **Protocol**: Protocol buffer message compatibility
- **MTGJSON Data Pipeline** — Downloads `AllPrintings.psql`, `AllIdentifiers.json`, `Keywords.json`, `CardTypes.json`, and `AllDeckFiles.zip` from MTGJSON v5 on startup or via a `POST /refresh` endpoint.
- **Dual PostgreSQL** — Two databases: `mtgonline` for the application (users, decks, auth) and `mtgdata` for MTG card data.
- **Redis Caching** — Used for card lookup caching and interaction pipeline state.
- **REST API** — `/docs` (Swagger) available at runtime.
## Getting Started
## Architecture
```
mtgonline/
├── backend/ # FastAPI application
│ ├── app/
│ │ ├── core/ # Settings, database engines, Redis client
│ │ ├── models/ # SQLAlchemy ORM models (app + MTG)
│ │ ├── routers/ # API route modules
│ │ ├── schemas/ # Pydantic request/response schemas
│ │ ├── services/ # Business logic (MTGJSON manager, card DB, game server)
│ │ └── main.py # FastAPI app entry point
│ ├── scripts/ # Utility scripts (downloads, migrations, checks)
│ ├── Dockerfile
│ └── requirements.txt
├── docker-compose.dev.yml # Development stack (Postgres x2, Redis, Backend)
├── docker-compose.yml # Production stack
├── scripts/ # Shared utility scripts
└── README.md
```
### Data Flow
1. **On startup**, the backend connects to PostgreSQL (both instances) and Redis.
2. It checks `mtg_refresh_log` in the `mtgdata` database for existing data.
3. If no data exists, it downloads MTGJSON files from `https://mtgjson.com/api/v5/`, unzips if needed, and upserts them into `mtgdata` tables (`mtg_sets`, `mtg_cards`, etc.).
4. The data is then available via REST endpoints.
## Quick Start
### Prerequisites
- Docker and Docker Compose
- Python 3.12+ (for development)
- PostgreSQL 14+ (if running without Docker)
- Redis (optional, for caching)
- **MTGJSON data files** (see below)
### Installation
1. **Clone the repository**
### Run the Stack
```bash
git clone https://github.com/yourusername/mtgonline-web.git
cd mtgonline-web
cd /home/wall-o/projects/mtgonline
# Start all services (Postgres x2, Redis, Backend)
docker compose -f docker-compose.dev.yml up -d
# View logs
docker compose -f docker-compose.dev.yml logs -f backend
```
2. **Configure environment**
The backend will automatically download MTGJSON data on first startup (this may take several minutes).
### Access
| Service | Address |
|---------------|---------------------|
| Backend API | `http://localhost:5555` |
| Swagger Docs | `http://localhost:5555/docs` |
| Health Check | `http://localhost:5555/health` |
| PostgreSQL (app) | `localhost:5432` |
| PostgreSQL (MTG) | `localhost:5433` |
| Redis | `localhost:6379` |
### Manual Data Refresh
```bash
cp .env.example .env
# Edit .env with your configuration
# Trigger a manual MTGJSON refresh
curl -X POST http://localhost:5555/refresh
```
3. **Prepare MTGJSON data files**
## Environment Variables
The application requires MTGJSON data files. You have two options:
| Variable | Default | Description |
|----------|---------|-------------|
| `DATABASE_URL` | `postgresql+asyncpg://mtgonline_user:mtgonline_password@postgres:5432/mtgonline` | Primary database connection |
| `MTG_DATABASE_URL` | `postgresql+asyncpg://mtgonline_user:mtgonline_password@mtgdata:5432/mtgdata` | MTG data database connection |
| `REDIS_URL` | `redis://redis:6379` | Redis connection |
| `DATA_DIR` | `/app/data` | Directory for MTGJSON files |
| `DEBUG` | `False` | Enable debug logging |
**Option A: Download from MTGJSON (recommended)**
## Docker Cleanup
```bash
# Create data directory
mkdir -p data
# Stop and remove all containers
docker compose -f docker-compose.dev.yml down
# Download required files from MTGJSON
# See scripts/download-mtgjson.sh for automated download
bash scripts/download-mtgjson.sh
```
**Option B: Provide your own JSON files**
Place these files in the `data/` directory:
- `AllSetFiles/` (directory containing set JSON files)
- `AllPrintings.json`
- `AllIdentifiers.json`
- `CardTypes.json`
- `DeckList.json`
- `Keywords.json`
- `SetList.json`
### Running with Docker (recommended)
```bash
# Copy the data directory to a volume mount path
mkdir -p ~/mtg-data
cp -r data/* ~/mtg-data/
# Start the application
sudo docker compose up -d
```
The backend container will automatically process the JSON files from the mounted volume and populate the database.
## Project Structure
```
mtgonline-web/
├── backend/
│ ├── app/
│ │ ├── core/ # Core configuration and utilities
│ │ │ ├── settings.py # Application settings
│ │ │ ├── database.py # Database engine and sessions
│ │ │ └── security.py # Authentication and password hashing
│ │ ├── models/ # SQLAlchemy ORM models
│ │ │ └── models.py
│ │ ├── schemas/ # Pydantic schemas
│ │ │ ├── schemas.py
│ │ │ ├── proto_messages.py
│ │ │ └── protocol_constants.py
│ │ ├── routers/ # API route handlers
│ │ │ ├── auth.py
│ │ │ ├── users.py
│ │ │ ├── decks.py
│ │ │ ├── rooms.py
│ │ │ ├── games.py
│ │ │ ├── admin.py
│ │ │ └── ws.py
│ │ ├── services/ # Business logic services
│ │ │ ├── game_server.py
│ │ │ ├── card_database.py
│ │ │ └── deck_parser.py
│ │ └── main.py # FastAPI application
│ ├── tests/ # Test suite
│ ├── requirements.txt # Python dependencies
│ ├── pyproject.toml # Ruff configuration
│ └── .env.example # Environment template
├── frontend/ # React/TypeScript frontend (coming soon)
├── shared/ # Shared protocol definitions
│ └── proto/ # Protocol buffer definitions
└── README.md
```
## Docker Deployment
### 1. Prepare the Data Directory
Create a directory on your host machine to store the MTGJSON files:
```bash
mkdir -p ~/mtg-data
cp -r data/* ~/mtg-data/
```
### 2. Configure Environment
```bash
cp .env.example .env
# Edit .env with your configuration
```
### 3. Mount and Start
```bash
# Start with data directory mounted (Linux/macOS)
sudo docker compose up -d
# The data directory is automatically mounted to /app/data in the backend container
# Alternatively, specify a custom path:
sudo docker compose up -d --build
```
The backend will:
- Detect the JSON files in the mounted volume
- Upsert all data into the PostgreSQL database
- Start the FastAPI application
### 4. Verify
```bash
# Check backend logs
docker compose logs -f backend
# Test health endpoint
curl http://localhost:8000/health
```
### Updating Data
When MTGJSON releases updates:
1. Copy new `.json` files to `~/mtg-data/`
2. The backend will automatically process new files on next startup
3. Or trigger a manual refresh via `/refresh` endpoint
## Development
### Run Locally
```bash
cd backend
pip install -r requirements.txt
python -m uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
```
### Run Tests
```bash
cd backend
pytest
```
## Project Structure
```
mtgonline-web/
├── backend/
│ ├── app/
│ │ ├── core/ # Core configuration and utilities
│ │ │ ├── settings.py # Application settings
│ │ │ ├── database.py # Database engine and sessions
│ │ │ └── security.py # Authentication and password hashing
│ │ ├── models/ # SQLAlchemy ORM models
│ │ │ └── models.py
│ │ ├── schemas/ # Pydantic schemas
│ │ │ ├── schemas.py
│ │ │ ├── proto_messages.py
│ │ │ └── protocol_constants.py
│ │ ├── routers/ # API route handlers
│ │ │ ├── auth.py
│ │ │ ├── users.py
│ │ │ ├── decks.py
│ │ │ ├── rooms.py
│ │ │ ├── games.py
│ │ │ ├── admin.py
│ │ │ └── ws.py
│ │ ├── services/ # Business logic services
│ │ │ ├── game_server.py
│ │ │ ├── card_database.py
│ │ │ └── deck_parser.py
│ │ └── main.py # FastAPI application
│ ├── tests/ # Test suite
│ ├── requirements.txt # Python dependencies
│ ├── pyproject.toml # Ruff configuration
│ └── .env.example # Environment template
├── frontend/ # React/TypeScript frontend (coming soon)
├── shared/ # Shared protocol definitions
│ └── proto/ # Protocol buffer definitions
├── data/ # MTGJSON data files (not in git)
├── scripts/ # Utility scripts
└── README.md
```
## API Endpoints
### Authentication
- `POST /api/v1/auth/login` - User login
- `POST /api/v1/auth/register` - User registration
- `POST /api/v1/auth/refresh` - Refresh access token
- `GET /api/v1/auth/me` - Get current user
### Users
- `GET /api/v1/users/{user_id}` - Get user by ID
- `PATCH /api/v1/users/{user_id}` - Update user profile
- `POST /api/v1/users/{user_id}/ban` - Ban user (admin)
- `POST /api/v1/users/{user_id}/unban` - Unban user (admin)
### Decks
- `GET /api/v1/decks/` - List decks
- `POST /api/v1/decks/` - Create deck
- `GET /api/v1/decks/{deck_id}` - Get deck
- `PATCH /api/v1/decks/{deck_id}` - Update deck
- `DELETE /api/v1/decks/{deck_id}` - Delete deck
- `GET /api/v1/decks/folders` - List folders
- `POST /api/v1/decks/folders` - Create folder
- `DELETE /api/v1/decks/folders/{folder_id}` - Delete folder
### Admin
- `GET /api/v1/admin/users` - List all users (admin)
- `GET /api/v1/admin/bans` - List all bans (admin)
- `POST /api/v1/admin/bans` - Create ban (admin)
- `POST /api/v1/admin/bans/{ban_id}/unban` - Unban user (admin)
- `GET /api/v1/admin/logs` - List game logs (admin)
## Testing
Run the test suite:
```bash
cd backend
pytest
# Remove images and prune
docker system prune -a --volumes
```
## License
MIT License
## Contributing
Contributions are welcome! Please open an issue or submit a pull request.
## Support
For questions or issues, please open a GitHub issue.
MIT