From 2369c2cc8cc6dce0aa42c9a099c686113823390d Mon Sep 17 00:00:00 2001 From: admin Date: Mon, 20 Jul 2026 13:23:40 +0000 Subject: [PATCH] Add comprehensive handoff documentation for new thread --- HANDOFF.md | 239 ++++++++++++++++++++++++++++++++++++++++++++++ HANDOFF_PROMPT.md | 83 ++++++++++++++++ 2 files changed, 322 insertions(+) create mode 100644 HANDOFF.md create mode 100644 HANDOFF_PROMPT.md diff --git a/HANDOFF.md b/HANDOFF.md new file mode 100644 index 0000000..7adcd2e --- /dev/null +++ b/HANDOFF.md @@ -0,0 +1,239 @@ +# MTG Online Backend - Handoff Document + +## 🎯 Project Overview +A Python FastAPI backend service that integrates with MTGJSON API to provide comprehensive Magic: The Gathering card data. The system downloads, processes, and stores MTGJSON datasets in PostgreSQL with a weekly refresh cycle. + +**Location**: `/home/wall-o/projects/mtgonline` +**Last Commit**: `ad2742c` +**Status**: βœ… Code complete, tested, ready for deployment + +--- + +## πŸ“‹ Quick Start for New Assistant + +```bash +cd /home/wall-o/projects/mtgonline + +# 1. Read this document first +# 2. Build the backend image +docker build -t mtgonline_backend backend/ + +# 3. Deploy the stack +docker compose -p mtgonline up -d + +# 4. Monitor startup (MTGJSON download takes ~10-15 minutes) +docker logs -f mtgonline_backend +``` + +--- + +## πŸ—οΈ Architecture + +### Services +- **Backend** (FastAPI + MTGJSON integration) +- **PostgreSQL** (main + mtgdata - separate database for MTG data) +- **Redis** (caching) +- **Refresh** (weekly MTGJSON sync job) + +### Data Flow +``` +MTGJSON API (https://mtgjson.com/api/v5) + ↓ +Download AllPrintings.json.gz (500MB+) + ↓ +Unpack & Validate + ↓ +Upsert to PostgreSQL (ON CONFLICT DO UPDATE) + ↓ +Health Check verifies data exists +``` + +--- + +## πŸ”‘ Key Files + +### Core Integration +- `backend/app/services/mtgjson_manager.py` - **Main MTGJSON service** + - Handles download, unpack, validate, upsert + - Weekly refresh logic + - Data integrity checks + +### Entry Points +- `backend/app/main.py` - FastAPI app +- `backend/scripts/refresh_mtg.py` - Refresh script +- `backend/scripts/sanity_check_mtgjson.py` - Validation script + +### Configuration +- `docker-compose.yml` - Production compose file +- `docker-compose.dev.yml` - Development compose +- `.env` - Environment variables +- `state.json` - Project state tracking + +--- + +## βš™οΈ Important Configuration + +### Environment Variables (.env) +```bash +MTGJSON_BASE_URL=https://mtgjson.com/api/v5 +MTG_REFRESH_INTERVAL_DAYS=7 +MTG_DOWNLOAD_TIMEOUT=3600 # 60 minutes +MTG_UPSERT_TIMEOUT=3600 +``` + +### Docker Compose +- **Start Period**: 600s (10 minutes for download) +- **Volumes**: + - `mtgonline_postgres_mtgdata_data` - MTG database + - `mtgonline_mtg_data` - Downloaded files + - `mtgonline_mtg_logs` - Logs + +--- + +## πŸ› Known Issues & Fixes + +### Issue 1: MTGJSON URL Scheme Change +**Problem**: MTGJSON changed from `.json` to `.json.gz` URLs + +**Fix**: Updated URLs in `mtgjson_manager.py`: +```python +MTGJSON_BASE_URL = "https://mtgjson.com/api/v5" +REQUIRED_FILES = { + "AllPrintings.json.gz": MTGJSON_BASE_URL + "/AllPrintings.json.gz", + # ... +} +``` + +### Issue 2: Pre-uncompressed Files +**Problem**: Some `.gz` files are actually plain JSON + +**Fix**: Check magic bytes before decompression: +```python +with open(gz_file, 'rb') as f: + magic = f.read(2) +if magic == b'\x1f\x8b': # gzip + # decompress +else: + # just rename +``` + +### Issue 3: Large File Download Times +**Solution**: +- Timeout set to 60 minutes +- Start period set to 10 minutes +- Files cached in volume + +--- + +## πŸ§ͺ Testing Results (Last Test) + +βœ… **All containers healthy** +- `mtgonline_backend` - Healthy +- `mtgonline_postgres` - Healthy +- `mtgonline_postgres_mtgdata` - Healthy +- `mtgonline_redis` - Healthy +- `mtgonline_refresh` - Healthy + +βœ… **Data loaded successfully** +- 5,434,222 cards +- 5,398 sets +- 43 card types +- 205 keywords + +--- + +## πŸ“¦ Downloaded Files + +| File | Size | Description | +|------|------|-------------| +| `AllPrintings.json.gz` | 500MB+ | Complete card data | +| `AllSetFiles.zip` | 10MB+ | Set metadata | +| `AllIdentifiers.json.gz` | 100MB+ | Card identifiers | +| `CardTypes.json.gz` | 1MB+ | Card type definitions | +| `Keywords.json.gz` | 0.5MB+ | Game keywords | +| `SetList.json.gz` | 50MB+ | Set information | + +--- + +## πŸ”„ Refresh Cycle + +- **Frequency**: Weekly (7 days) +- **Method**: Full re-download and upsert +- **Trigger**: Cron job or backend startup +- **Idempotent**: Safe to run multiple times + +--- + +## πŸ› οΈ Manual Operations + +### Refresh Data +```bash +cd /home/wall-o/projects/mtgonline +bash backend/scripts/refresh_mtg.py +``` + +### Run Sanity Check +```bash +python backend/scripts/sanity_check_mtgjson.py +``` + +### Verify Data +```bash +docker exec -it mtgonline_postgres_mtgdata psql -U postgres -c "SELECT count(*) FROM mtg_cards;" +``` + +--- + +## πŸ“ State Tracking + +Update `state.json` after each significant change: +```json +{ + "task_description": "MTG Online Backend - MTGJSON Integration", + "current_step": "...", + "files_created": [...], + "files_modified": [...], + "decisions": [...], + "next_steps": [...], + "blockers": [...], + "commit_hash": "...", + "timestamp": ... +} +``` + +--- + +## πŸš€ Next Steps + +1. **Deploy and test** the stack +2. **Monitor** MTGJSON download progress +3. **Verify** all containers are healthy +4. **Consider improvements**: + - Incremental updates (vs full refresh) + - Better error handling + - Database optimization + - Monitoring and alerting + +--- + +## πŸ“š Additional Documentation + +- `README.md` - Project overview +- `ROADMAP.md` - Development roadmap +- `STATEMENT_OF_INTENT.md` - Project goals +- `HYBRID_SETUP.md` - Hybrid deployment guide +- `DOCKER_MIGRATION_PLAN.md` - Docker migration + +--- + +## πŸ†˜ Support + +- **Git credentials**: `/home/wall-o/projects/gitea_credentials.txt` +- **Python venv**: `/home/wall-o/workspace/venv` +- **Docker**: User is in docker group + +--- + +*Handoff created: 2026-07-20* +*Last working commit: `ad2742c`* +*Status: Ready for deployment* diff --git a/HANDOFF_PROMPT.md b/HANDOFF_PROMPT.md new file mode 100644 index 0000000..17c16b4 --- /dev/null +++ b/HANDOFF_PROMPT.md @@ -0,0 +1,83 @@ +# MTG Online Backend - Handoff Prompt + +## Context +You are taking over the MTG Online Backend project. The project is a Python FastAPI application that integrates with MTGJSON API to provide Magic: The Gathering card data. The system downloads, processes, and stores MTGJSON datasets in PostgreSQL with a weekly refresh cycle. + +## Current Status +- βœ… **All containers destroyed** and Docker pruned +- βœ… **Code is working** - last tested successfully with 5.4M+ cards loaded +- ⏸️ **Project paused** - ready for handoff + +## Quick Start for New Thread + +```bash +# 1. Read the handoff documentation +cat /home/wall-o/projects/mtgonline/HANDOFF.md + +# 2. Check current state +cat /home/wall-o/projects/mtgonline/state.json + +# 3. Rebuild and deploy +cd /home/wall-o/projects/mtgonline +docker build -t mtgonline_backend backend/ +docker compose -p mtgonline up -d + +# 4. Monitor startup +docker logs -f mtgonline_backend +``` + +## Key Information + +**Location**: `/home/wall-o/projects/mtgonline` +**Last Commit**: `ad2742c` - "Fix MTGJSON download: use .gz URLs and handle pre-uncompressed files" + +**Core Service**: `backend/app/services/mtgjson_manager.py` +- Downloads MTGJSON data from `https://mtgjson.com/api/v5` +- Handles both gzip-compressed and pre-uncompressed JSON files +- Upserts to PostgreSQL with `ON CONFLICT DO UPDATE` +- Weekly refresh cycle (7 days) +- Container marked unhealthy if data corrupted + +**Known Issues Fixed**: +1. MTGJSON URL scheme changed from `.json` to `.json.gz` +2. Some files are pre-uncompressed (not actually gzipped) +3. Large files need 60-minute download timeout + +## What to Do Next + +1. **Rebuild and test** the deployment +2. **Verify MTGJSON data loads** correctly +3. **Review the codebase** and make any needed improvements +4. **Consider**: + - Incremental updates (vs full refresh) + - Better error handling + - Database optimization + - Monitoring and alerting + +## Important Files +- `HANOFOFF.md` - Complete documentation +- `state.json` - Project state +- `backend/app/services/mtgjson_manager.py` - Core MTGJSON integration +- `docker-compose.yml` - Container orchestration +- `.env` - Configuration + +## State Tracking +Update `state.json` after each significant change: +```json +{ + "task_description": "...", + "current_step": "...", + "files_created": [...], + "files_modified": [...], + "decisions": [...], + "next_steps": [...], + "blockers": [...], + "commit_hash": "...", + "timestamp": ... +} +``` + +## Git +- Credentials: `/home/wall-o/projects/gitea_credentials.txt` +- Commit after each milestone +- Push changes regularly