Add comprehensive handoff documentation for new thread
This commit is contained in:
+239
@@ -0,0 +1,239 @@
|
|||||||
|
# MTG Online Backend - Handoff Document
|
||||||
|
|
||||||
|
## 🎯 Project Overview
|
||||||
|
A Python FastAPI backend service that integrates with MTGJSON API to provide comprehensive Magic: The Gathering card data. The system downloads, processes, and stores MTGJSON datasets in PostgreSQL with a weekly refresh cycle.
|
||||||
|
|
||||||
|
**Location**: `/home/wall-o/projects/mtgonline`
|
||||||
|
**Last Commit**: `ad2742c`
|
||||||
|
**Status**: ✅ Code complete, tested, ready for deployment
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📋 Quick Start for New Assistant
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /home/wall-o/projects/mtgonline
|
||||||
|
|
||||||
|
# 1. Read this document first
|
||||||
|
# 2. Build the backend image
|
||||||
|
docker build -t mtgonline_backend backend/
|
||||||
|
|
||||||
|
# 3. Deploy the stack
|
||||||
|
docker compose -p mtgonline up -d
|
||||||
|
|
||||||
|
# 4. Monitor startup (MTGJSON download takes ~10-15 minutes)
|
||||||
|
docker logs -f mtgonline_backend
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🏗️ Architecture
|
||||||
|
|
||||||
|
### Services
|
||||||
|
- **Backend** (FastAPI + MTGJSON integration)
|
||||||
|
- **PostgreSQL** (main + mtgdata - separate database for MTG data)
|
||||||
|
- **Redis** (caching)
|
||||||
|
- **Refresh** (weekly MTGJSON sync job)
|
||||||
|
|
||||||
|
### Data Flow
|
||||||
|
```
|
||||||
|
MTGJSON API (https://mtgjson.com/api/v5)
|
||||||
|
↓
|
||||||
|
Download AllPrintings.json.gz (500MB+)
|
||||||
|
↓
|
||||||
|
Unpack & Validate
|
||||||
|
↓
|
||||||
|
Upsert to PostgreSQL (ON CONFLICT DO UPDATE)
|
||||||
|
↓
|
||||||
|
Health Check verifies data exists
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔑 Key Files
|
||||||
|
|
||||||
|
### Core Integration
|
||||||
|
- `backend/app/services/mtgjson_manager.py` - **Main MTGJSON service**
|
||||||
|
- Handles download, unpack, validate, upsert
|
||||||
|
- Weekly refresh logic
|
||||||
|
- Data integrity checks
|
||||||
|
|
||||||
|
### Entry Points
|
||||||
|
- `backend/app/main.py` - FastAPI app
|
||||||
|
- `backend/scripts/refresh_mtg.py` - Refresh script
|
||||||
|
- `backend/scripts/sanity_check_mtgjson.py` - Validation script
|
||||||
|
|
||||||
|
### Configuration
|
||||||
|
- `docker-compose.yml` - Production compose file
|
||||||
|
- `docker-compose.dev.yml` - Development compose
|
||||||
|
- `.env` - Environment variables
|
||||||
|
- `state.json` - Project state tracking
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ⚙️ Important Configuration
|
||||||
|
|
||||||
|
### Environment Variables (.env)
|
||||||
|
```bash
|
||||||
|
MTGJSON_BASE_URL=https://mtgjson.com/api/v5
|
||||||
|
MTG_REFRESH_INTERVAL_DAYS=7
|
||||||
|
MTG_DOWNLOAD_TIMEOUT=3600 # 60 minutes
|
||||||
|
MTG_UPSERT_TIMEOUT=3600
|
||||||
|
```
|
||||||
|
|
||||||
|
### Docker Compose
|
||||||
|
- **Start Period**: 600s (10 minutes for download)
|
||||||
|
- **Volumes**:
|
||||||
|
- `mtgonline_postgres_mtgdata_data` - MTG database
|
||||||
|
- `mtgonline_mtg_data` - Downloaded files
|
||||||
|
- `mtgonline_mtg_logs` - Logs
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🐛 Known Issues & Fixes
|
||||||
|
|
||||||
|
### Issue 1: MTGJSON URL Scheme Change
|
||||||
|
**Problem**: MTGJSON changed from `.json` to `.json.gz` URLs
|
||||||
|
|
||||||
|
**Fix**: Updated URLs in `mtgjson_manager.py`:
|
||||||
|
```python
|
||||||
|
MTGJSON_BASE_URL = "https://mtgjson.com/api/v5"
|
||||||
|
REQUIRED_FILES = {
|
||||||
|
"AllPrintings.json.gz": MTGJSON_BASE_URL + "/AllPrintings.json.gz",
|
||||||
|
# ...
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Issue 2: Pre-uncompressed Files
|
||||||
|
**Problem**: Some `.gz` files are actually plain JSON
|
||||||
|
|
||||||
|
**Fix**: Check magic bytes before decompression:
|
||||||
|
```python
|
||||||
|
with open(gz_file, 'rb') as f:
|
||||||
|
magic = f.read(2)
|
||||||
|
if magic == b'\x1f\x8b': # gzip
|
||||||
|
# decompress
|
||||||
|
else:
|
||||||
|
# just rename
|
||||||
|
```
|
||||||
|
|
||||||
|
### Issue 3: Large File Download Times
|
||||||
|
**Solution**:
|
||||||
|
- Timeout set to 60 minutes
|
||||||
|
- Start period set to 10 minutes
|
||||||
|
- Files cached in volume
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🧪 Testing Results (Last Test)
|
||||||
|
|
||||||
|
✅ **All containers healthy**
|
||||||
|
- `mtgonline_backend` - Healthy
|
||||||
|
- `mtgonline_postgres` - Healthy
|
||||||
|
- `mtgonline_postgres_mtgdata` - Healthy
|
||||||
|
- `mtgonline_redis` - Healthy
|
||||||
|
- `mtgonline_refresh` - Healthy
|
||||||
|
|
||||||
|
✅ **Data loaded successfully**
|
||||||
|
- 5,434,222 cards
|
||||||
|
- 5,398 sets
|
||||||
|
- 43 card types
|
||||||
|
- 205 keywords
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📦 Downloaded Files
|
||||||
|
|
||||||
|
| File | Size | Description |
|
||||||
|
|------|------|-------------|
|
||||||
|
| `AllPrintings.json.gz` | 500MB+ | Complete card data |
|
||||||
|
| `AllSetFiles.zip` | 10MB+ | Set metadata |
|
||||||
|
| `AllIdentifiers.json.gz` | 100MB+ | Card identifiers |
|
||||||
|
| `CardTypes.json.gz` | 1MB+ | Card type definitions |
|
||||||
|
| `Keywords.json.gz` | 0.5MB+ | Game keywords |
|
||||||
|
| `SetList.json.gz` | 50MB+ | Set information |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔄 Refresh Cycle
|
||||||
|
|
||||||
|
- **Frequency**: Weekly (7 days)
|
||||||
|
- **Method**: Full re-download and upsert
|
||||||
|
- **Trigger**: Cron job or backend startup
|
||||||
|
- **Idempotent**: Safe to run multiple times
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🛠️ Manual Operations
|
||||||
|
|
||||||
|
### Refresh Data
|
||||||
|
```bash
|
||||||
|
cd /home/wall-o/projects/mtgonline
|
||||||
|
bash backend/scripts/refresh_mtg.py
|
||||||
|
```
|
||||||
|
|
||||||
|
### Run Sanity Check
|
||||||
|
```bash
|
||||||
|
python backend/scripts/sanity_check_mtgjson.py
|
||||||
|
```
|
||||||
|
|
||||||
|
### Verify Data
|
||||||
|
```bash
|
||||||
|
docker exec -it mtgonline_postgres_mtgdata psql -U postgres -c "SELECT count(*) FROM mtg_cards;"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📝 State Tracking
|
||||||
|
|
||||||
|
Update `state.json` after each significant change:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"task_description": "MTG Online Backend - MTGJSON Integration",
|
||||||
|
"current_step": "...",
|
||||||
|
"files_created": [...],
|
||||||
|
"files_modified": [...],
|
||||||
|
"decisions": [...],
|
||||||
|
"next_steps": [...],
|
||||||
|
"blockers": [...],
|
||||||
|
"commit_hash": "...",
|
||||||
|
"timestamp": ...
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🚀 Next Steps
|
||||||
|
|
||||||
|
1. **Deploy and test** the stack
|
||||||
|
2. **Monitor** MTGJSON download progress
|
||||||
|
3. **Verify** all containers are healthy
|
||||||
|
4. **Consider improvements**:
|
||||||
|
- Incremental updates (vs full refresh)
|
||||||
|
- Better error handling
|
||||||
|
- Database optimization
|
||||||
|
- Monitoring and alerting
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📚 Additional Documentation
|
||||||
|
|
||||||
|
- `README.md` - Project overview
|
||||||
|
- `ROADMAP.md` - Development roadmap
|
||||||
|
- `STATEMENT_OF_INTENT.md` - Project goals
|
||||||
|
- `HYBRID_SETUP.md` - Hybrid deployment guide
|
||||||
|
- `DOCKER_MIGRATION_PLAN.md` - Docker migration
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🆘 Support
|
||||||
|
|
||||||
|
- **Git credentials**: `/home/wall-o/projects/gitea_credentials.txt`
|
||||||
|
- **Python venv**: `/home/wall-o/workspace/venv`
|
||||||
|
- **Docker**: User is in docker group
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*Handoff created: 2026-07-20*
|
||||||
|
*Last working commit: `ad2742c`*
|
||||||
|
*Status: Ready for deployment*
|
||||||
@@ -0,0 +1,83 @@
|
|||||||
|
# MTG Online Backend - Handoff Prompt
|
||||||
|
|
||||||
|
## Context
|
||||||
|
You are taking over the MTG Online Backend project. The project is a Python FastAPI application that integrates with MTGJSON API to provide Magic: The Gathering card data. The system downloads, processes, and stores MTGJSON datasets in PostgreSQL with a weekly refresh cycle.
|
||||||
|
|
||||||
|
## Current Status
|
||||||
|
- ✅ **All containers destroyed** and Docker pruned
|
||||||
|
- ✅ **Code is working** - last tested successfully with 5.4M+ cards loaded
|
||||||
|
- ⏸️ **Project paused** - ready for handoff
|
||||||
|
|
||||||
|
## Quick Start for New Thread
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Read the handoff documentation
|
||||||
|
cat /home/wall-o/projects/mtgonline/HANDOFF.md
|
||||||
|
|
||||||
|
# 2. Check current state
|
||||||
|
cat /home/wall-o/projects/mtgonline/state.json
|
||||||
|
|
||||||
|
# 3. Rebuild and deploy
|
||||||
|
cd /home/wall-o/projects/mtgonline
|
||||||
|
docker build -t mtgonline_backend backend/
|
||||||
|
docker compose -p mtgonline up -d
|
||||||
|
|
||||||
|
# 4. Monitor startup
|
||||||
|
docker logs -f mtgonline_backend
|
||||||
|
```
|
||||||
|
|
||||||
|
## Key Information
|
||||||
|
|
||||||
|
**Location**: `/home/wall-o/projects/mtgonline`
|
||||||
|
**Last Commit**: `ad2742c` - "Fix MTGJSON download: use .gz URLs and handle pre-uncompressed files"
|
||||||
|
|
||||||
|
**Core Service**: `backend/app/services/mtgjson_manager.py`
|
||||||
|
- Downloads MTGJSON data from `https://mtgjson.com/api/v5`
|
||||||
|
- Handles both gzip-compressed and pre-uncompressed JSON files
|
||||||
|
- Upserts to PostgreSQL with `ON CONFLICT DO UPDATE`
|
||||||
|
- Weekly refresh cycle (7 days)
|
||||||
|
- Container marked unhealthy if data corrupted
|
||||||
|
|
||||||
|
**Known Issues Fixed**:
|
||||||
|
1. MTGJSON URL scheme changed from `.json` to `.json.gz`
|
||||||
|
2. Some files are pre-uncompressed (not actually gzipped)
|
||||||
|
3. Large files need 60-minute download timeout
|
||||||
|
|
||||||
|
## What to Do Next
|
||||||
|
|
||||||
|
1. **Rebuild and test** the deployment
|
||||||
|
2. **Verify MTGJSON data loads** correctly
|
||||||
|
3. **Review the codebase** and make any needed improvements
|
||||||
|
4. **Consider**:
|
||||||
|
- Incremental updates (vs full refresh)
|
||||||
|
- Better error handling
|
||||||
|
- Database optimization
|
||||||
|
- Monitoring and alerting
|
||||||
|
|
||||||
|
## Important Files
|
||||||
|
- `HANOFOFF.md` - Complete documentation
|
||||||
|
- `state.json` - Project state
|
||||||
|
- `backend/app/services/mtgjson_manager.py` - Core MTGJSON integration
|
||||||
|
- `docker-compose.yml` - Container orchestration
|
||||||
|
- `.env` - Configuration
|
||||||
|
|
||||||
|
## State Tracking
|
||||||
|
Update `state.json` after each significant change:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"task_description": "...",
|
||||||
|
"current_step": "...",
|
||||||
|
"files_created": [...],
|
||||||
|
"files_modified": [...],
|
||||||
|
"decisions": [...],
|
||||||
|
"next_steps": [...],
|
||||||
|
"blockers": [...],
|
||||||
|
"commit_hash": "...",
|
||||||
|
"timestamp": ...
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Git
|
||||||
|
- Credentials: `/home/wall-o/projects/gitea_credentials.txt`
|
||||||
|
- Commit after each milestone
|
||||||
|
- Push changes regularly
|
||||||
Reference in New Issue
Block a user