admin/mtgonline
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Clean up backend folder - remove obsolete scripts and test files

Browse Source 
Removed obsolete scripts that were not imported or used:
- card_interaction_rule_engine.py
- card_profile_extractor.py
- create_card_interaction_graph.py
- interaction_determinator.py
- interaction_pipeline.py
- interaction_recommender.py
- interaction_schema.py
- recommendation_engine.py
- migrate_complete.py
- migrate_schema.py
- test_interaction_determinator.py
- check_mtgjson_full.py
- check_mtgjson_status.py
- verify_integration.py
- verify_mtgjson_data.py
- sanity_check_mtgjson.py
- investigate_sets.py
- inspect_db.py
- code_review.md
- monitor/mtg_monitor.py

Removed test artifacts:
- test.db
- test_download.py
- test_system.py
- setup_db.py
- BACKEND_TESTING_SUMMARY.md
- CHAT_PROMPT_TEST.md
- CONTINUATION_PROMPT.md
- PORTED_STATE.md
- SPEC_synergy-mapping-engine.md
- STATE.md
- SUPPORTED_FILE_TYPES.md

Removed sensitive/environment files:
- .env.local
- state.json (backend)

Cleaned up:
- __pycache__ directories
- venv directory

Backend scripts/ directory now contains only essential data loading and maintenance scripts.
This commit is contained in:
akadmin
2026-07-22 03:00:04 +00:00
parent b4a0b1f8be
commit a01e33eb5e
35 changed files with 1093 additions and 9215 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/README.md
+74 -241
View File
@@ -1,271 +1,104 @@
# MTG Online Backend
Python FastAPI application for processing MTGJSON card data and managing the MTG Online platform backend.
A Python FastAPI application that processes Magic: The Gathering card data from [MTGJSON](https://mtgjson.com/) and stores it in PostgreSQL, with Redis for caching.
## Overview
This project provides a backend API for a Magic: The Gathering Online platform. It downloads and processes MTGJSON v5 dataset dumps, loads them into a PostgreSQL database, and exposes REST endpoints for card data, user authentication, deck management, and game state.
### Key Features
- **MTGJSON Data Pipeline** — Downloads `AllPrintings.psql`, `AllIdentifiers.json`, `Keywords.json`, `CardTypes.json`, and `AllDeckFiles.zip` from MTGJSON v5 on startup or via a `POST /refresh` endpoint.
- **Dual PostgreSQL** — Two databases: `mtgonline` for the application (users, decks, auth) and `mtgdata` for MTG card data.
- **Redis Caching** — Used for card lookup caching and interaction pipeline state.
- **REST API** — `/docs` (Swagger) available at runtime.
## Architecture
### Core Components
The backend consists of several key layers:
**Core Layer** (`app/core/`)
- `settings.py` — Application configuration using pydantic-settings with environment variable overrides
- `database.py` — Dual PostgreSQL engine setup (mtgonline app DB + mtgdata MTG cards DB)
- `redis_client.py` — Redis connection and caching utilities
**Models** (`app/models/`)
- `models.py` — SQLAlchemy ORM models for application data (users, decks, auth)
- `mtg_models.py` — ORM models for MTG card data
**Services** (`app/services/`)
- `mtgjson_manager.py` — Primary MTGJSON data pipeline (download, unzip, upsert)
- `mtgjson_downloader.py` — HTTP client for MTGJSON API
- `mtgjson_loader.py` — Data loading and transformation
- `mtgjson_uploader.py` — Database upsert operations
- `card_database.py` — Card data access layer
- `game_server.py` — Game state management
- `deck_parser.py` — Deck list parsing and validation
**Routers** (`app/routers/`)
- `auth.py` — JWT authentication endpoints
- `users.py` — User management
- `decks.py` — Deck CRUD operations
- `rooms.py` — Game room management
- `games.py` — Game state endpoints
- `admin.py` — Admin tools
- `card_router.py` — MTG card data API
- `interactions.py` — Card interaction engine
- `refresh.py` — MTGJSON data refresh endpoint
- `ws.py` — WebSocket support
**Schemas** (`app/schemas/`)
- `schemas.py` — Pydantic models for request/response validation
- `proto_messages.py` — Protocol buffer message definitions
- `protocol_constants.py` — MTG protocol constants
```
mtgonline/
├── backend/ # FastAPI application
│ ├── app/
│ │ ├── core/ # Settings, database engines, Redis client
│ │ ├── models/ # SQLAlchemy ORM models (app + MTG)
│ │ ├── routers/ # API route modules
│ │ ├── schemas/ # Pydantic request/response schemas
│ │ ├── services/ # Business logic (MTGJSON manager, card DB, game server)
│ │ └── main.py # FastAPI app entry point
│ ├── scripts/ # Utility scripts (downloads, migrations, checks)
│ ├── Dockerfile
│ └── requirements.txt
├── docker-compose.dev.yml # Development stack (Postgres x2, Redis, Backend)
├── docker-compose.yml # Production stack
├── scripts/ # Shared utility scripts
└── README.md
```
### Data Flow
```
MTGJSON API (https://mtgjson.com/api/v5/)
↓ download
MTGJSON files (AllPrintings.psql, AllIdentifiers.json, etc.)
↓ load/transform
PostgreSQL (mtgdata database)
↓ query
REST API / Swagger Docs
1. **On startup**, the backend connects to PostgreSQL (both instances) and Redis.
2. It checks `mtg_refresh_log` in the `mtgdata` database for existing data.
3. If no data exists, it downloads MTGJSON files from `https://mtgjson.com/api/v5/`, unzips if needed, and upserts them into `mtgdata` tables (`mtg_sets`, `mtg_cards`, etc.).
4. The data is then available via REST endpoints.
## Quick Start
### Prerequisites
- Docker and Docker Compose
### Run the Stack
```bash
cd /home/wall-o/projects/mtgonline
# Start all services (Postgres x2, Redis, Backend)
docker compose -f docker-compose.dev.yml up -d
# View logs
docker compose -f docker-compose.dev.yml logs -f backend
```
## Database Setup
The backend will automatically download MTGJSON data on first startup (this may take several minutes).
### Primary Database (mtgonline)
- **Purpose**: Application data (users, decks, auth tokens)
- **Connection**: `postgresql+asyncpg://mtgonline_user:mtgonline_password@postgres:5432/mtgonline`
- **Port**: 5432 (internal), 5432:5432 (host)
### Access
### MTG Data Database (mtgdata)
- **Purpose**: MTG card data, sets, refresh logs
- **Connection**: `postgresql+asyncpg://mtgonline_user:mtgonline_password@mtgdata:5432/mtgdata`
- **Port**: 5432 (internal), 5433:5432 (host)
- **Tables**:
- `mtg_sets` — Card sets metadata
- `mtg_cards` — Individual card data
- `mtg_refresh_log` — Refresh history and status
| Service | Address |
|---------------|---------------------|
| Backend API | `http://localhost:5555` |
| Swagger Docs | `http://localhost:5555/docs` |
| Health Check | `http://localhost:5555/health` |
| PostgreSQL (app) | `localhost:5432` |
| PostgreSQL (MTG) | `localhost:5433` |
| Redis | `localhost:6379` |
### Redis
- **Purpose**: Caching, session management
- **Connection**: `redis://redis:6379`
- **Port**: 6379 (internal), 6379:6379 (host)
### Manual Data Refresh
## MTGJSON Data Pipeline
### Downloaded Files
The backend downloads these files from MTGJSON v5:
- `AllPrintings.psql` — Main card data (PostgreSQL format)
- `AllIdentifiers.json` — Card identifiers (Multiverse, Scryfall, etc.)
- `Keywords.json` — Card keywords
- `CardTypes.json` — Card type definitions
- `AllDeckFiles.zip` — Deck files (must be unzipped)
### Refresh Logic
1. **On Startup**: Checks `mtg_refresh_log` for existing data
2. **If No Data**: Downloads and loads all MTGJSON files (may take minutes)
3. **Manual Refresh**: `POST /refresh` endpoint triggers immediate reload
4. **Logging**: All refreshes logged to `mtg_refresh_log` with status, timing, and counts
```bash
# Trigger a manual MTGJSON refresh
curl -X POST http://localhost:5555/refresh
```
## Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `DATABASE_URL` | `postgresql+asyncpg://mtgonline_user:mtgonline_password@postgres:5432/mtgonline` | Primary database |
| `MTG_DATABASE_URL` | `postgresql+asyncpg://mtgonline_user:mtgonline_password@mtgdata:5432/mtgdata` | MTG data database |
| `DATABASE_URL` | `postgresql+asyncpg://mtgonline_user:mtgonline_password@postgres:5432/mtgonline` | Primary database connection |
| `MTG_DATABASE_URL` | `postgresql+asyncpg://mtgonline_user:mtgonline_password@mtgdata:5432/mtgdata` | MTG data database connection |
| `REDIS_URL` | `redis://redis:6379` | Redis connection |
| `DATA_DIR` | `/app/data` | MTGJSON files directory |
| `UPLOAD_DIR` | `/app/uploads` | User uploads directory |
| `DATA_DIR` | `/app/data` | Directory for MTGJSON files |
| `DEBUG` | `False` | Enable debug logging |
| `LOG_LEVEL` | `INFO` | Logging level |
| `SECRET_KEY` | `change-me-in-production` | JWT secret |
| `JWT_SECRET_KEY` | `change-me-in-production` | JWT signing key |
## Running the Backend
### Docker (Recommended)
## Docker Cleanup
```bash
# Build image
cd backend
docker build -t mtgonline-backend:latest .
# Stop and remove all containers
docker compose -f docker-compose.dev.yml down
# Run with dependencies
docker compose -f ../docker-compose.dev.yml up -d backend
# Remove images and prune
docker system prune -a --volumes
```
### Local Development
```bash
cd backend
# Create venv
python -m venv venv
source venv/bin/activate
# Install dependencies
pip install -r requirements.txt
# Set environment variables
export DATABASE_URL="postgresql+asyncpg://mtgonline_user:mtgonline_password@localhost:5432/mtgonline"
export MTG_DATABASE_URL="postgresql+asyncpg://mtgonline_user:mtgonline_password@localhost:5433/mtgdata"
export REDIS_URL="redis://localhost:6379"
# Run server
python -m uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
```
## API Endpoints
### Health & Status
- `GET /health` — Health check with MTGJSON status
- `GET /` — API info
### Authentication
- `POST /auth/login` — User login
- `POST /auth/register` — User registration
- `POST /auth/refresh` — Refresh JWT
- `GET /auth/me` — Current user
### Users
- `GET /users/{user_id}` — Get user
- `PATCH /users/{user_id}` — Update user
- `POST /users/{user_id}/ban` — Ban user (admin)
### Decks
- `GET /decks/` — List decks
- `POST /decks/` — Create deck
- `GET /decks/{deck_id}` — Get deck
- `PATCH /decks/{deck_id}` — Update deck
- `DELETE /decks/{deck_id}` — Delete deck
### MTG Cards
- `GET /api/cards/` — Search cards
- `GET /api/cards/{card_id}` — Get card
- `GET /api/sets/` — List sets
### Admin
- `GET /admin/users` — List all users
- `GET /admin/bans` — List bans
- `POST /admin/bans` — Create ban
### Data Management
- `POST /refresh` — Trigger MTGJSON refresh
### WebSocket
- `WS /ws/{room_id}` — Real-time game communication
## Utility Scripts
Located in `scripts/`:
- `download_mtgjson.py` — Manual MTGJSON download
- `check_mtgjson_status.py` — Verify data freshness
- `verify_mtgjson_data.py` — Data validation
- `inspect_db.py` — Database inspection
- `load_mtgjson_data.py` — Data loading
## Testing
```bash
cd backend
# Run tests
pytest
# Run with coverage
pytest --cov=app --cov-report=html
```
## Project Structure
```
backend/
├── app/
│ ├── __init__.py
│ ├── main.py # FastAPI app entry
│ ├── core/
│ │ ├── settings.py # Configuration
│ │ ├── database.py # Database engines
│ │ └── redis_client.py # Redis utilities
│ ├── models/
│ │ ├── models.py # App models
│ │ └── mtg_models.py # MTG models
│ ├── routers/
│ │ ├── auth.py # Auth endpoints
│ │ ├── users.py # User endpoints
│ │ ├── decks.py # Deck endpoints
│ │ ├── rooms.py # Room endpoints
│ │ ├── games.py # Game endpoints
│ │ ├── admin.py # Admin endpoints
│ │ ├── card_router.py # Card API
│ │ ├── interactions.py # Card interactions
│ │ ├── refresh.py # Data refresh
│ │ └── ws.py # WebSocket
│ ├── schemas/
│ │ ├── schemas.py # Pydantic models
│ │ ├── proto_messages.py # Protocol messages
│ │ └── protocol_constants.py
│ └── services/
│ ├── mtgjson_manager.py # MTGJSON pipeline
│ ├── mtgjson_downloader.py
│ ├── mtgjson_loader.py
│ ├── mtgjson_uploader.py
│ ├── card_database.py # Card data access
│ ├── game_server.py # Game logic
│ └── deck_parser.py # Deck parsing
├── scripts/ # Utility scripts
├── tests/ # Test suite
├── Dockerfile # Container build
├── requirements.txt # Python dependencies
├── pyproject.toml # Ruff config
├── .env.example # Environment template
└── setup_db.py # Database setup script
```
## Troubleshooting
### Backend can't connect to databases
- Verify all services are running: `docker compose -f ../docker-compose.dev.yml ps`
- Check logs: `docker compose -f ../docker-compose.dev.yml logs backend`
- Ensure environment variables match docker-compose.dev.yml
### MTGJSON download fails
- Check network connectivity to mtgjson.com
- Verify DATA_DIR has write permissions
- Check disk space: `df -h`
- Manual download: `python scripts/download_mtgjson.py`
### Database tables missing
- Run initialization: `docker exec -i mtgdata psql -U mtgonline_user mtgdata < /path/to/scripts/init-mtgdata.sql`
- Check tables: `docker exec mtgdata psql -U mtgonline_user mtgdata -c "\dt"`
## License
MIT