Add comprehensive documentation for MTG Online Backend

- Updated root README with current architecture (dual PostgreSQL, Redis, MTGJSON pipeline) - Created backend/README.md with detailed architecture, database setup, and troubleshooting - Updated state.json to reflect completed documentation phase
2026-07-21 03:50:44 +00:00
parent 6f01e2d1b4
commit 46abfe5fbc
3 changed files with 373 additions and 300 deletions
@@ -0,0 +1,271 @@
+# MTG Online Backend
+
+Python FastAPI application for processing MTGJSON card data and managing the MTG Online platform backend.
+
+## Architecture
+
+### Core Components
+
+The backend consists of several key layers:
+
+**Core Layer** (`app/core/`)
+- `settings.py` — Application configuration using pydantic-settings with environment variable overrides
+- `database.py` — Dual PostgreSQL engine setup (mtgonline app DB + mtgdata MTG cards DB)
+- `redis_client.py` — Redis connection and caching utilities
+
+**Models** (`app/models/`)
+- `models.py` — SQLAlchemy ORM models for application data (users, decks, auth)
+- `mtg_models.py` — ORM models for MTG card data
+
+**Services** (`app/services/`)
+- `mtgjson_manager.py` — Primary MTGJSON data pipeline (download, unzip, upsert)
+- `mtgjson_downloader.py` — HTTP client for MTGJSON API
+- `mtgjson_loader.py` — Data loading and transformation
+- `mtgjson_uploader.py` — Database upsert operations
+- `card_database.py` — Card data access layer
+- `game_server.py` — Game state management
+- `deck_parser.py` — Deck list parsing and validation
+
+**Routers** (`app/routers/`)
+- `auth.py` — JWT authentication endpoints
+- `users.py` — User management
+- `decks.py` — Deck CRUD operations
+- `rooms.py` — Game room management
+- `games.py` — Game state endpoints
+- `admin.py` — Admin tools
+- `card_router.py` — MTG card data API
+- `interactions.py` — Card interaction engine
+- `refresh.py` — MTGJSON data refresh endpoint
+- `ws.py` — WebSocket support
+
+**Schemas** (`app/schemas/`)
+- `schemas.py` — Pydantic models for request/response validation
+- `proto_messages.py` — Protocol buffer message definitions
+- `protocol_constants.py` — MTG protocol constants
+
+### Data Flow
+
+```
+MTGJSON API (https://mtgjson.com/api/v5/)
+    ↓ download
+MTGJSON files (AllPrintings.psql, AllIdentifiers.json, etc.)
+    ↓ load/transform
+PostgreSQL (mtgdata database)
+    ↓ query
+REST API / Swagger Docs
+```
+
+## Database Setup
+
+### Primary Database (mtgonline)
+- **Purpose**: Application data (users, decks, auth tokens)
+- **Connection**: `postgresql+asyncpg://mtgonline_user:mtgonline_password@postgres:5432/mtgonline`
+- **Port**: 5432 (internal), 5432:5432 (host)
+
+### MTG Data Database (mtgdata)
+- **Purpose**: MTG card data, sets, refresh logs
+- **Connection**: `postgresql+asyncpg://mtgonline_user:mtgonline_password@mtgdata:5432/mtgdata`
+- **Port**: 5432 (internal), 5433:5432 (host)
+- **Tables**:
+  - `mtg_sets` — Card sets metadata
+  - `mtg_cards` — Individual card data
+  - `mtg_refresh_log` — Refresh history and status
+
+### Redis
+- **Purpose**: Caching, session management
+- **Connection**: `redis://redis:6379`
+- **Port**: 6379 (internal), 6379:6379 (host)
+
+## MTGJSON Data Pipeline
+
+### Downloaded Files
+
+The backend downloads these files from MTGJSON v5:
+- `AllPrintings.psql` — Main card data (PostgreSQL format)
+- `AllIdentifiers.json` — Card identifiers (Multiverse, Scryfall, etc.)
+- `Keywords.json` — Card keywords
+- `CardTypes.json` — Card type definitions
+- `AllDeckFiles.zip` — Deck files (must be unzipped)
+
+### Refresh Logic
+
+1. **On Startup**: Checks `mtg_refresh_log` for existing data
+2. **If No Data**: Downloads and loads all MTGJSON files (may take minutes)
+3. **Manual Refresh**: `POST /refresh` endpoint triggers immediate reload
+4. **Logging**: All refreshes logged to `mtg_refresh_log` with status, timing, and counts
+
+## Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `DATABASE_URL` | `postgresql+asyncpg://mtgonline_user:mtgonline_password@postgres:5432/mtgonline` | Primary database |
+| `MTG_DATABASE_URL` | `postgresql+asyncpg://mtgonline_user:mtgonline_password@mtgdata:5432/mtgdata` | MTG data database |
+| `REDIS_URL` | `redis://redis:6379` | Redis connection |
+| `DATA_DIR` | `/app/data` | MTGJSON files directory |
+| `UPLOAD_DIR` | `/app/uploads` | User uploads directory |
+| `DEBUG` | `False` | Enable debug logging |
+| `LOG_LEVEL` | `INFO` | Logging level |
+| `SECRET_KEY` | `change-me-in-production` | JWT secret |
+| `JWT_SECRET_KEY` | `change-me-in-production` | JWT signing key |
+
+## Running the Backend
+
+### Docker (Recommended)
+
+```bash
+# Build image
+cd backend
+docker build -t mtgonline-backend:latest .
+
+# Run with dependencies
+docker compose -f ../docker-compose.dev.yml up -d backend
+```
+
+### Local Development
+
+```bash
+cd backend
+
+# Create venv
+python -m venv venv
+source venv/bin/activate
+
+# Install dependencies
+pip install -r requirements.txt
+
+# Set environment variables
+export DATABASE_URL="postgresql+asyncpg://mtgonline_user:mtgonline_password@localhost:5432/mtgonline"
+export MTG_DATABASE_URL="postgresql+asyncpg://mtgonline_user:mtgonline_password@localhost:5433/mtgdata"
+export REDIS_URL="redis://localhost:6379"
+
+# Run server
+python -m uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
+```
+
+## API Endpoints
+
+### Health & Status
+- `GET /health` — Health check with MTGJSON status
+- `GET /` — API info
+
+### Authentication
+- `POST /auth/login` — User login
+- `POST /auth/register` — User registration
+- `POST /auth/refresh` — Refresh JWT
+- `GET /auth/me` — Current user
+
+### Users
+- `GET /users/{user_id}` — Get user
+- `PATCH /users/{user_id}` — Update user
+- `POST /users/{user_id}/ban` — Ban user (admin)
+
+### Decks
+- `GET /decks/` — List decks
+- `POST /decks/` — Create deck
+- `GET /decks/{deck_id}` — Get deck
+- `PATCH /decks/{deck_id}` — Update deck
+- `DELETE /decks/{deck_id}` — Delete deck
+
+### MTG Cards
+- `GET /api/cards/` — Search cards
+- `GET /api/cards/{card_id}` — Get card
+- `GET /api/sets/` — List sets
+
+### Admin
+- `GET /admin/users` — List all users
+- `GET /admin/bans` — List bans
+- `POST /admin/bans` — Create ban
+
+### Data Management
+- `POST /refresh` — Trigger MTGJSON refresh
+
+### WebSocket
+- `WS /ws/{room_id}` — Real-time game communication
+
+## Utility Scripts
+
+Located in `scripts/`:
+- `download_mtgjson.py` — Manual MTGJSON download
+- `check_mtgjson_status.py` — Verify data freshness
+- `verify_mtgjson_data.py` — Data validation
+- `inspect_db.py` — Database inspection
+- `load_mtgjson_data.py` — Data loading
+
+## Testing
+
+```bash
+cd backend
+
+# Run tests
+pytest
+
+# Run with coverage
+pytest --cov=app --cov-report=html
+```
+
+## Project Structure
+
+```
+backend/
+├── app/
+│   ├── __init__.py
+│   ├── main.py                 # FastAPI app entry
+│   ├── core/
+│   │   ├── settings.py         # Configuration
+│   │   ├── database.py         # Database engines
+│   │   └── redis_client.py     # Redis utilities
+│   ├── models/
+│   │   ├── models.py           # App models
+│   │   └── mtg_models.py       # MTG models
+│   ├── routers/
+│   │   ├── auth.py             # Auth endpoints
+│   │   ├── users.py            # User endpoints
+│   │   ├── decks.py            # Deck endpoints
+│   │   ├── rooms.py            # Room endpoints
+│   │   ├── games.py            # Game endpoints
+│   │   ├── admin.py            # Admin endpoints
+│   │   ├── card_router.py      # Card API
+│   │   ├── interactions.py     # Card interactions
+│   │   ├── refresh.py          # Data refresh
+│   │   └── ws.py               # WebSocket
+│   ├── schemas/
+│   │   ├── schemas.py          # Pydantic models
+│   │   ├── proto_messages.py   # Protocol messages
+│   │   └── protocol_constants.py
+│   └── services/
+│       ├── mtgjson_manager.py  # MTGJSON pipeline
+│       ├── mtgjson_downloader.py
+│       ├── mtgjson_loader.py
+│       ├── mtgjson_uploader.py
+│       ├── card_database.py    # Card data access
+│       ├── game_server.py      # Game logic
+│       └── deck_parser.py      # Deck parsing
+├── scripts/                    # Utility scripts
+├── tests/                      # Test suite
+├── Dockerfile                  # Container build
+├── requirements.txt            # Python dependencies
+├── pyproject.toml              # Ruff config
+├── .env.example                # Environment template
+└── setup_db.py                 # Database setup script
+```
+
+## Troubleshooting
+
+### Backend can't connect to databases
+- Verify all services are running: `docker compose -f ../docker-compose.dev.yml ps`
+- Check logs: `docker compose -f ../docker-compose.dev.yml logs backend`
+- Ensure environment variables match docker-compose.dev.yml
+
+### MTGJSON download fails
+- Check network connectivity to mtgjson.com
+- Verify DATA_DIR has write permissions
+- Check disk space: `df -h`
+- Manual download: `python scripts/download_mtgjson.py`
+
+### Database tables missing
+- Run initialization: `docker exec -i mtgdata psql -U mtgonline_user mtgdata < /path/to/scripts/init-mtgdata.sql`
+- Check tables: `docker exec mtgdata psql -U mtgonline_user mtgdata -c "\dt"`
+
+## License
+
+MIT