admin/mtgonline
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update HANDOFF.md with roadmap and statement of intent

Browse Source 
- Add Vision Statement and Core Objectives from STATEMENT_OF_INTENT.md
- Add Primary Focus section for backend expansion
- Add Database Schema Expansion requirements
- Add API Endpoints to Implement
- Add Frontend Features to Support (from ROADMAP Phase 2)
- Add Integration Requirements (from ROADMAP Phase 3)
- Add Deployment & Production requirements (from ROADMAP Phase 4)
- Expand Key Files to Review to include strategic documents
- Add Timeline table from ROADMAP
- Update status to Phase 1 Complete
This commit is contained in:
akadmin
2026-07-22 00:07:49 +00:00
parent 2872c562ac
commit b4a0b1f8be
1 changed files with 334 additions and 191 deletions
Download Patch File Download Diff File Expand all files Collapse all files
HANDOFF.md
+334 -191
View File
@@ -1,239 +1,382 @@
# MTG Online Backend - Handoff Document
## 🎯 Project Overview
A Python FastAPI backend service that integrates with MTGJSON API to provide comprehensive Magic: The Gathering card data. The system downloads, processes, and stores MTGJSON datasets in PostgreSQL with a weekly refresh cycle.
## Project Overview
**Location**: `/home/wall-o/projects/mtgonline`
**Last Commit**: `ad2742c`
**Status**: ✅ Code complete, tested, ready for deployment
**Project Name**: MTG Online Backend
**Location**: `/home/wall-o/projects/mtgonline`
**Purpose**: Python FastAPI application that processes Magic: The Gathering card data from MTGJSON and stores it in PostgreSQL databases.
---
## Vision Statement (from STATEMENT_OF_INTENT.md)
## 📋 Quick Start for New Assistant
**MTG Online Web** — A modern, web-based implementation of the MTG Online multiplayer Magic: The Gathering platform.
```bash
cd /home/wall-o/projects/mtgonline
To build a fully-featured, open-source multiplayer Magic: The Gathering platform that runs entirely in modern web browsers, eliminating the need for desktop software installations while maintaining compatibility with the existing MTG Online ecosystem.
# 1. Read this document first
# 2. Build the backend image
docker build -t mtgonline_backend backend/
## Core Objectives
# 3. Deploy the stack
docker compose -p mtgonline up -d
### User Experience
- Intuitive, modern interface that rivals native desktop applications
- Real-time multiplayer gameplay with minimal latency
- Seamless deck building with import/export from MTG Online
- Responsive design that works across all screen sizes
# 4. Monitor startup (MTGJSON download takes ~10-15 minutes)
docker logs -f mtgonline_backend
### Technical Excellence
- **Backend**: Python 3.12 + FastAPI with async architecture
- **Database**: PostgreSQL with async SQLAlchemy ORM
- **Real-time**: WebSocket-based game server for live multiplayer
- **Protocol**: Full compatibility with MTG Online protocol buffer messages
### Feature Parity with Desktop
- User authentication and account management
- Deck creation, editing, and storage (per-user)
- Multiplayer game rooms with real-time state sync
- Card game mechanics (mana, phases, priority, stack)
- Admin/moderation tools (ban, warn, log viewing)
- Card database integration with comprehensive card data
### Performance Requirements
- API response times < 100ms for 95% of requests
- WebSocket latency < 50ms for game state updates
- Support 1000+ concurrent users
- Sub-second page loads with proper caching
### Success Criteria
- [ ] Users can create accounts and authenticate securely
- [ ] Users can create, edit, and manage decks (per-user)
- [ ] Users can join and play multiplayer games in real-time
- [ ] Game state syncs correctly across all connected players
- [ ] Admin users can manage accounts and moderate games
- [ ] Deck formats are compatible with MTG Online desktop client
## Current Status
All planned tasks have been completed:
- ✅ Documentation created (root README.md + backend/README.md)
- ✅ State.json updated
- ✅ Commit pushed to Gitea (commit `46abfe5`)
- ✅ Docker cleanup completed (all containers, images, volumes removed)
## Architecture Summary
### Tech Stack
- **Backend**: Python 3.12, FastAPI, SQLAlchemy (async), asyncpg
- **Databases**: Dual PostgreSQL (14-alpine)
- Primary: `mtgonline` database (users, decks, auth)
- MTG Data: `mtgdata` database (card data, sets)
- **Cache**: Redis 7-alpine
- **Protocol**: Protocol buffer message compatibility
### Service Architecture
```
mtgonline/
├── backend/ # FastAPI application
│ ├── app/
│ │ ├── core/ # Settings, database engines, Redis client
│ │ ├── models/ # SQLAlchemy ORM models
│ │ ├── routers/ # API route modules (auth, users, decks, rooms, games, admin, cards, interactions, refresh, ws)
│ │ ├── schemas/ # Pydantic request/response schemas
│ │ ├── services/ # Business logic (MTGJSON manager, card DB, game server, deck parser)
│ │ └── main.py # FastAPI app entry point
│ ├── scripts/ # Utility scripts (downloads, migrations, checks)
│ ├── Dockerfile
│ ├── requirements.txt
│ └── .env.example
├── docker-compose.dev.yml # Development stack
├── docker-compose.yml # Production stack
└── README.md # Project documentation
```
---
## 🏗️ Architecture
### Services
- **Backend** (FastAPI + MTGJSON integration)
- **PostgreSQL** (main + mtgdata - separate database for MTG data)
- **Redis** (caching)
- **Refresh** (weekly MTGJSON sync job)
### Data Flow
```
MTGJSON API (https://mtgjson.com/api/v5)
↓
Download AllPrintings.json.gz (500MB+)
↓
Unpack & Validate
↓
Upsert to PostgreSQL (ON CONFLICT DO UPDATE)
↓
Health Check verifies data exists
1. Backend connects to PostgreSQL (both instances) and Redis on startup
2. Checks `mtg_refresh_log` in `mtgdata` database for existing data
3. If no data exists, downloads MTGJSON files from `https://mtgjson.com/api/v5/`
4. Upserts data into `mtgdata` tables (`mtg_sets`, `mtg_cards`, etc.)
5. Data available via REST endpoints
### Key Database Connections
- **Primary DB**: `postgresql+asyncpg://mtgonline_user:mtgonline_password@postgres:5432/mtgonline`
- **MTG DB**: `postgresql+asyncpg://mtgonline_user:mtgonline_password@mtgdata:5432/mtgdata`
- **Redis**: `redis://redis:6379`
## Environment Configuration
### Docker Compose Dev Environment Variables
```yaml
environment:
DATABASE_URL: "postgresql+asyncpg://mtgonline_user:mtgonline_password@postgres:5432/mtgonline"
MTG_DATABASE_URL: "postgresql+asyncpg://mtgonline_user:mtgonline_password@mtgdata:5432/mtgdata"
REDIS_URL: "redis://redis:6379"
```
---
### Service Ports (Host to Container)
- PostgreSQL: `5432:5432`
- MTG Data PostgreSQL: `5433:5432`
- Redis: `6379:6379`
- Backend: `5555:8000`
## 🔑 Key Files
## API Endpoints
### Core Integration
- `backend/app/services/mtgjson_manager.py` - **Main MTGJSON service**
- Handles download, unpack, validate, upsert
- Weekly refresh logic
- Data integrity checks
### Health & Status
- `GET /health` - Health check with MTGJSON status
- `GET /` - API info
### Entry Points
- `backend/app/main.py` - FastAPI app
- `backend/scripts/refresh_mtg.py` - Refresh script
- `backend/scripts/sanity_check_mtgjson.py` - Validation script
### Authentication
- `POST /auth/login` - User login
- `POST /auth/register` - User registration
- `POST /auth/refresh` - Refresh JWT
- `GET /auth/me` - Current user
### Configuration
- `docker-compose.yml` - Production compose file
- `docker-compose.dev.yml` - Development compose
- `.env` - Environment variables
- `state.json` - Project state tracking
### Users
- `GET /users/{user_id}` - Get user
- `PATCH /users/{user_id}` - Update user
- `POST /users/{user_id}/ban` - Ban user (admin)
---
### Decks
- `GET /decks/` - List decks
- `POST /decks/` - Create deck
- `GET /decks/{deck_id}` - Get deck
- `PATCH /decks/{deck_id}` - Update deck
- `DELETE /decks/{deck_id}` - Delete deck
## ⚙️ Important Configuration
### MTG Cards
- `GET /api/cards/` - Search cards
- `GET /api/cards/{card_id}` - Get card
- `GET /api/sets/` - List sets
### Environment Variables (.env)
```bash
MTGJSON_BASE_URL=https://mtgjson.com/api/v5
MTG_REFRESH_INTERVAL_DAYS=7
MTG_DOWNLOAD_TIMEOUT=3600 # 60 minutes
MTG_UPSERT_TIMEOUT=3600
### Admin
- `GET /admin/users` - List all users
- `GET /admin/bans` - List bans
- `POST /admin/bans` - Create ban
### Data Management
- `POST /refresh` - Trigger MTGJSON refresh
### WebSocket
- `WS /ws/{room_id}` - Real-time game communication
## MTGJSON Data Pipeline
### Downloaded Files
- `AllPrintings.psql` - Main card data (PostgreSQL format)
- `AllIdentifiers.json` - Card identifiers
- `Keywords.json` - Card keywords
- `CardTypes.json` - Card type definitions
- `AllDeckFiles.zip` - Deck files (unzipped on load)
### Refresh Logic
1. **On Startup**: Checks `mtg_refresh_log` for existing data
2. **If No Data**: Downloads and loads all MTGJSON files (may take minutes)
3. **Manual Refresh**: `POST /refresh` triggers immediate reload
4. **Logging**: All refreshes logged to `mtg_refresh_log` with status, timing, and counts
## Git Repository
**Gitea Repository**: `https://git.optimex.systems/admin/mtgonline.git`
**Credentials**: Located at `/home/wall-o/projects/gitea_credentials.txt`
**Current Branch**: `main`
**Last Commit**: `46abfe5` - "Add comprehensive documentation for MTG Online Backend"
### Commit History
```
46abfe5 - Add comprehensive documentation for MTG Online Backend
6f01e2d - Initial project setup
```
### Docker Compose
- **Start Period**: 600s (10 minutes for download)
- **Volumes**:
- `mtgonline_postgres_mtgdata_data` - MTG database
- `mtgonline_mtg_data` - Downloaded files
- `mtgonline_mtg_logs` - Logs
## Running the Project
---
## 🐛 Known Issues & Fixes
### Issue 1: MTGJSON URL Scheme Change
**Problem**: MTGJSON changed from `.json` to `.json.gz` URLs
**Fix**: Updated URLs in `mtgjson_manager.py`:
```python
MTGJSON_BASE_URL = "https://mtgjson.com/api/v5"
REQUIRED_FILES = {
"AllPrintings.json.gz": MTGJSON_BASE_URL + "/AllPrintings.json.gz",
# ...
}
```
### Issue 2: Pre-uncompressed Files
**Problem**: Some `.gz` files are actually plain JSON
**Fix**: Check magic bytes before decompression:
```python
with open(gz_file, 'rb') as f:
magic = f.read(2)
if magic == b'\x1f\x8b': # gzip
# decompress
else:
# just rename
```
### Issue 3: Large File Download Times
**Solution**:
- Timeout set to 60 minutes
- Start period set to 10 minutes
- Files cached in volume
---
## 🧪 Testing Results (Last Test)
✅ **All containers healthy**
- `mtgonline_backend` - Healthy
- `mtgonline_postgres` - Healthy
- `mtgonline_postgres_mtgdata` - Healthy
- `mtgonline_redis` - Healthy
- `mtgonline_refresh` - Healthy
✅ **Data loaded successfully**
- 5,434,222 cards
- 5,398 sets
- 43 card types
- 205 keywords
---
## 📦 Downloaded Files
| File | Size | Description |
|------|------|-------------|
| `AllPrintings.json.gz` | 500MB+ | Complete card data |
| `AllSetFiles.zip` | 10MB+ | Set metadata |
| `AllIdentifiers.json.gz` | 100MB+ | Card identifiers |
| `CardTypes.json.gz` | 1MB+ | Card type definitions |
| `Keywords.json.gz` | 0.5MB+ | Game keywords |
| `SetList.json.gz` | 50MB+ | Set information |
---
## 🔄 Refresh Cycle
- **Frequency**: Weekly (7 days)
- **Method**: Full re-download and upsert
- **Trigger**: Cron job or backend startup
- **Idempotent**: Safe to run multiple times
---
## 🛠️ Manual Operations
### Refresh Data
### Start Services
```bash
cd /home/wall-o/projects/mtgonline
bash backend/scripts/refresh_mtg.py
docker compose -f docker-compose.dev.yml up -d
```
### Run Sanity Check
### Check Services
```bash
python backend/scripts/sanity_check_mtgjson.py
docker compose -f docker-compose.dev.yml ps
docker compose -f docker-compose.dev.yml logs -f backend
```
### Verify Data
### Stop Services
```bash
docker exec -it mtgonline_postgres_mtgdata psql -U postgres -c "SELECT count(*) FROM mtg_cards;"
docker compose -f docker-compose.dev.yml down
```
---
### Full Cleanup
```bash
# Stop and remove all containers
docker compose -f docker-compose.dev.yml down
## 📝 State Tracking
# Remove images
docker images rm mtgonline-backend:latest postgres:14-alpine redis:7-alpine
# Clear build cache and prune
docker builder prune -af
docker system prune -af --volumes
```
## Testing
### Run Tests
```bash
cd /home/wall-o/projects/mtgonline/backend
docker exec -it <backend_container_id> pytest
# OR
cd /home/wall-o/projects/mtgonline/backend
pytest
```
### Database Verification
```bash
# Check MTG data tables
docker exec <mtgdata_container_id> psql -U mtgonline_user mtgdata -c "\dt"
# Check refresh log
docker exec <mtgdata_container_id> psql -U mtgonline_user mtgdata -c "SELECT * FROM mtg_refresh_log ORDER BY refresh_time DESC LIMIT 5;"
```
## Troubleshooting
### Backend can't connect to databases
- Verify all services are running: `docker compose -f docker-compose.dev.yml ps`
- Check logs: `docker compose -f docker-compose.dev.yml logs backend`
- Ensure environment variables match docker-compose.dev.yml
### MTGJSON download fails
- Check network connectivity to mtgjson.com
- Verify DATA_DIR has write permissions
- Check disk space: `df -h`
- Manual download: `python scripts/download_mtgjson.py`
### Database tables missing
- Run initialization: `docker exec -i <mtgdata_container_id> psql -U mtgonline_user mtgdata < /path/to/scripts/init-mtgdata.sql`
- Check tables: `docker exec <mtgdata_container_id> psql -U mtgonline_user mtgdata -c "\dt"`
### CORS errors
- Check CORS_ORIGINS setting in app/core/settings.py
- Ensure frontend URL matches allowed origins
## Next Phase: Backend Expansion for Frontend Support
**Primary Focus**: Expand the PostgreSQL database schema and API endpoints to support frontend deckbuilding and gameplay features on a per-user basis.
### Database Schema Expansion
- Per-user deck storage (decks, folders, custom card sets)
- Game state persistence (saved games, match history, game logs)
- User card collection tracking (owned cards, favorites)
- Game room state management (active games, waiting lists)
- Tournament and custom rule support
### API Endpoints to Implement
- Deck CRUD with user ownership and sharing
- Game room creation, joining, and state management
- Real-time WebSocket endpoints for multiplayer gameplay
- Card collection APIs (search, filter, organize)
- Game history and replay APIs
- Admin tools for game monitoring and moderation
### Frontend Features to Support (from ROADMAP.md Phase 2)
- **Deck Builder**: Card search with filters (name, color, type, set), drag-and-drop editor, import/export formats
- **Game Interface**: Game board visualization, player zones (hand, library, graveyard, exile, command), real-time updates
- **Chat System**: Room chat, game chat, player list, moderator tools
- **Admin Dashboard**: User management, ban/unban controls, game logs, system statistics
### Integration Requirements (from ROADMAP.md Phase 3)
- WebSocket client with reconnection logic
- Card database caching and search functionality
- Game logic implementation (turn-based state, mana tracking, stack resolution)
- Performance optimization (virtual scrolling, memoization, code splitting)
### Deployment & Production (from ROADMAP.md Phase 4)
- Docker Compose for development and production
- CI/CD pipeline with GitHub Actions
- Security hardening (rate limiting, input validation, HTTPS)
- Monitoring and alerting (structured logging, error tracking)
See **ROADMAP.md** for complete feature specifications and timeline.
## State File
Current state saved at: `/home/wall-o/projects/mtgonline/state.json`
Update `state.json` after each significant change:
```json
{
"task_description": "MTG Online Backend - MTGJSON Integration",
"current_step": "...",
"files_created": [...],
"files_modified": [...],
"decisions": [...],
"next_steps": [...],
"blockers": [...],
"commit_hash": "...",
"timestamp": ...
"task_description": "Complete documentation and cleanup of MTG Online Backend project",
"current_step": "All tasks completed: documentation, commit/push to Gitea, Docker cleanup",
"files_created": [
"/home/wall-o/projects/mtgonline/README.md",
"/home/wall-o/projects/mtgonline/backend/README.md"
],
"files_modified": [
"/home/wall-o/projects/mtgonline/README.md",
"/home/wall-o/projects/mtgonline/state.json"
],
"decisions": [
"Updated root README with current architecture (dual PostgreSQL, Redis, MTGJSON pipeline)",
"Created comprehensive backend README with architecture, database setup, and troubleshooting",
"Hardcoded environment variables in docker-compose.dev.yml to prevent connection issues",
"Backend successfully connects to mtgdata:5432/mtgdata (not localhost)",
"All Docker resources cleaned up: containers, images, volumes, networks"
],
"next_steps": [],
"blockers": [],
"commit_hash": "46abfe5",
"timestamp": "2026-07-21T03:56:00Z"
}
```
---
## Access Information
## 🚀 Next Steps
- **Backend API**: `http://localhost:5555`
- **Swagger Docs**: `http://localhost:5555/docs`
- **Health Check**: `http://localhost:5555/health`
- **Gitea**: `https://git.optimex.systems/admin/mtgonline`
1. **Deploy and test** the stack
2. **Monitor** MTGJSON download progress
3. **Verify** all containers are healthy
4. **Consider improvements**:
- Incremental updates (vs full refresh)
- Better error handling
- Database optimization
- Monitoring and alerting
### Key Files to Review
#### Current State
1. `/home/wall-o/projects/mtgonline/state.json` - Current project state
2. `/home/wall-o/projects/mtgonline/README.md` - Project documentation
3. `/home/wall-o/projects/mtgonline/backend/README.md` - Backend documentation
#### Architecture & Configuration
4. `/home/wall-o/projects/mtgonline/docker-compose.dev.yml` - Docker configuration
5. `/home/wall-o/projects/mtgonline/backend/app/core/settings.py` - Application settings
6. `/home/wall-o/projects/mtgonline/backend/app/main.py` - FastAPI entry point
#### MTGJSON Integration
7. `/home/wall-o/projects/mtgonline/backend/app/services/mtgjson_manager.py` - MTGJSON pipeline
#### Strategic Documents
8. `/home/wall-o/projects/mtgonline/ROADMAP.md` - Complete feature roadmap and timeline
9. `/home/wall-o/projects/mtgonline/STATEMENT_OF_INTENT.md` - Project vision and objectives
#### Database & Models
10. `/home/wall-o/projects/mtgonline/backend/app/models/models.py` - SQLAlchemy ORM models
11. `/home/wall-o/projects/mtgonline/backend/app/models/mtg_models.py` - MTG-specific models
#### APIs
12. `/home/wall-o/projects/mtgonline/backend/app/routers/` - All API route modules
13. `/home/wall-o/projects/mtgonline/backend/app/schemas/` - Pydantic request/response schemas
## Environment
- **OS**: Linux 6.8.0-136-generic (x86_64)
- **Docker**: Available
- **Python**: 3.12.3
- **Working Directory**: `/home/wall-o/projects/mtgonline`
---
## 📚 Additional Documentation
**Last Updated**: 2026-07-21T03:56:00Z
**Status**: Phase 1 Complete. Ready for Phase 2: Backend Expansion for Frontend Support.
- `README.md` - Project overview
- `ROADMAP.md` - Development roadmap
- `STATEMENT_OF_INTENT.md` - Project goals
- `HYBRID_SETUP.md` - Hybrid deployment guide
- `DOCKER_MIGRATION_PLAN.md` - Docker migration
**Next Action**: Begin backend database schema expansion and API development for deckbuilding and gameplay features.
---
## 🆘 Support
- **Git credentials**: `/home/wall-o/projects/gitea_credentials.txt`
- **Python venv**: `/home/wall-o/workspace/venv`
- **Docker**: User is in docker group
---
*Handoff created: 2026-07-20*
*Last working commit: `ad2742c`*
*Status: Ready for deployment*
**Timeline**:
| Phase | Duration | Status |
|-------|----------|--------|
| Phase 1: Backend Foundation | 2 weeks | ✅ Complete |
| Phase 2: Frontend Development | 4 weeks | Not Started |
| Phase 3: Integration & Polish | 2 weeks | Not Started |
| Phase 4: Deployment & Production | 1 week | Not Started |
| Phase 5: Advanced Features | Ongoing | Future |