Add comprehensive handoff documentation for new thread
This commit is contained in:
+239
@@ -0,0 +1,239 @@
|
||||
# MTG Online Backend - Handoff Document
|
||||
|
||||
## 🎯 Project Overview
|
||||
A Python FastAPI backend service that integrates with MTGJSON API to provide comprehensive Magic: The Gathering card data. The system downloads, processes, and stores MTGJSON datasets in PostgreSQL with a weekly refresh cycle.
|
||||
|
||||
**Location**: `/home/wall-o/projects/mtgonline`
|
||||
**Last Commit**: `ad2742c`
|
||||
**Status**: ✅ Code complete, tested, ready for deployment
|
||||
|
||||
---
|
||||
|
||||
## 📋 Quick Start for New Assistant
|
||||
|
||||
```bash
|
||||
cd /home/wall-o/projects/mtgonline
|
||||
|
||||
# 1. Read this document first
|
||||
# 2. Build the backend image
|
||||
docker build -t mtgonline_backend backend/
|
||||
|
||||
# 3. Deploy the stack
|
||||
docker compose -p mtgonline up -d
|
||||
|
||||
# 4. Monitor startup (MTGJSON download takes ~10-15 minutes)
|
||||
docker logs -f mtgonline_backend
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Architecture
|
||||
|
||||
### Services
|
||||
- **Backend** (FastAPI + MTGJSON integration)
|
||||
- **PostgreSQL** (main + mtgdata - separate database for MTG data)
|
||||
- **Redis** (caching)
|
||||
- **Refresh** (weekly MTGJSON sync job)
|
||||
|
||||
### Data Flow
|
||||
```
|
||||
MTGJSON API (https://mtgjson.com/api/v5)
|
||||
↓
|
||||
Download AllPrintings.json.gz (500MB+)
|
||||
↓
|
||||
Unpack & Validate
|
||||
↓
|
||||
Upsert to PostgreSQL (ON CONFLICT DO UPDATE)
|
||||
↓
|
||||
Health Check verifies data exists
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔑 Key Files
|
||||
|
||||
### Core Integration
|
||||
- `backend/app/services/mtgjson_manager.py` - **Main MTGJSON service**
|
||||
- Handles download, unpack, validate, upsert
|
||||
- Weekly refresh logic
|
||||
- Data integrity checks
|
||||
|
||||
### Entry Points
|
||||
- `backend/app/main.py` - FastAPI app
|
||||
- `backend/scripts/refresh_mtg.py` - Refresh script
|
||||
- `backend/scripts/sanity_check_mtgjson.py` - Validation script
|
||||
|
||||
### Configuration
|
||||
- `docker-compose.yml` - Production compose file
|
||||
- `docker-compose.dev.yml` - Development compose
|
||||
- `.env` - Environment variables
|
||||
- `state.json` - Project state tracking
|
||||
|
||||
---
|
||||
|
||||
## ⚙️ Important Configuration
|
||||
|
||||
### Environment Variables (.env)
|
||||
```bash
|
||||
MTGJSON_BASE_URL=https://mtgjson.com/api/v5
|
||||
MTG_REFRESH_INTERVAL_DAYS=7
|
||||
MTG_DOWNLOAD_TIMEOUT=3600 # 60 minutes
|
||||
MTG_UPSERT_TIMEOUT=3600
|
||||
```
|
||||
|
||||
### Docker Compose
|
||||
- **Start Period**: 600s (10 minutes for download)
|
||||
- **Volumes**:
|
||||
- `mtgonline_postgres_mtgdata_data` - MTG database
|
||||
- `mtgonline_mtg_data` - Downloaded files
|
||||
- `mtgonline_mtg_logs` - Logs
|
||||
|
||||
---
|
||||
|
||||
## 🐛 Known Issues & Fixes
|
||||
|
||||
### Issue 1: MTGJSON URL Scheme Change
|
||||
**Problem**: MTGJSON changed from `.json` to `.json.gz` URLs
|
||||
|
||||
**Fix**: Updated URLs in `mtgjson_manager.py`:
|
||||
```python
|
||||
MTGJSON_BASE_URL = "https://mtgjson.com/api/v5"
|
||||
REQUIRED_FILES = {
|
||||
"AllPrintings.json.gz": MTGJSON_BASE_URL + "/AllPrintings.json.gz",
|
||||
# ...
|
||||
}
|
||||
```
|
||||
|
||||
### Issue 2: Pre-uncompressed Files
|
||||
**Problem**: Some `.gz` files are actually plain JSON
|
||||
|
||||
**Fix**: Check magic bytes before decompression:
|
||||
```python
|
||||
with open(gz_file, 'rb') as f:
|
||||
magic = f.read(2)
|
||||
if magic == b'\x1f\x8b': # gzip
|
||||
# decompress
|
||||
else:
|
||||
# just rename
|
||||
```
|
||||
|
||||
### Issue 3: Large File Download Times
|
||||
**Solution**:
|
||||
- Timeout set to 60 minutes
|
||||
- Start period set to 10 minutes
|
||||
- Files cached in volume
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Testing Results (Last Test)
|
||||
|
||||
✅ **All containers healthy**
|
||||
- `mtgonline_backend` - Healthy
|
||||
- `mtgonline_postgres` - Healthy
|
||||
- `mtgonline_postgres_mtgdata` - Healthy
|
||||
- `mtgonline_redis` - Healthy
|
||||
- `mtgonline_refresh` - Healthy
|
||||
|
||||
✅ **Data loaded successfully**
|
||||
- 5,434,222 cards
|
||||
- 5,398 sets
|
||||
- 43 card types
|
||||
- 205 keywords
|
||||
|
||||
---
|
||||
|
||||
## 📦 Downloaded Files
|
||||
|
||||
| File | Size | Description |
|
||||
|------|------|-------------|
|
||||
| `AllPrintings.json.gz` | 500MB+ | Complete card data |
|
||||
| `AllSetFiles.zip` | 10MB+ | Set metadata |
|
||||
| `AllIdentifiers.json.gz` | 100MB+ | Card identifiers |
|
||||
| `CardTypes.json.gz` | 1MB+ | Card type definitions |
|
||||
| `Keywords.json.gz` | 0.5MB+ | Game keywords |
|
||||
| `SetList.json.gz` | 50MB+ | Set information |
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Refresh Cycle
|
||||
|
||||
- **Frequency**: Weekly (7 days)
|
||||
- **Method**: Full re-download and upsert
|
||||
- **Trigger**: Cron job or backend startup
|
||||
- **Idempotent**: Safe to run multiple times
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ Manual Operations
|
||||
|
||||
### Refresh Data
|
||||
```bash
|
||||
cd /home/wall-o/projects/mtgonline
|
||||
bash backend/scripts/refresh_mtg.py
|
||||
```
|
||||
|
||||
### Run Sanity Check
|
||||
```bash
|
||||
python backend/scripts/sanity_check_mtgjson.py
|
||||
```
|
||||
|
||||
### Verify Data
|
||||
```bash
|
||||
docker exec -it mtgonline_postgres_mtgdata psql -U postgres -c "SELECT count(*) FROM mtg_cards;"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📝 State Tracking
|
||||
|
||||
Update `state.json` after each significant change:
|
||||
```json
|
||||
{
|
||||
"task_description": "MTG Online Backend - MTGJSON Integration",
|
||||
"current_step": "...",
|
||||
"files_created": [...],
|
||||
"files_modified": [...],
|
||||
"decisions": [...],
|
||||
"next_steps": [...],
|
||||
"blockers": [...],
|
||||
"commit_hash": "...",
|
||||
"timestamp": ...
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Next Steps
|
||||
|
||||
1. **Deploy and test** the stack
|
||||
2. **Monitor** MTGJSON download progress
|
||||
3. **Verify** all containers are healthy
|
||||
4. **Consider improvements**:
|
||||
- Incremental updates (vs full refresh)
|
||||
- Better error handling
|
||||
- Database optimization
|
||||
- Monitoring and alerting
|
||||
|
||||
---
|
||||
|
||||
## 📚 Additional Documentation
|
||||
|
||||
- `README.md` - Project overview
|
||||
- `ROADMAP.md` - Development roadmap
|
||||
- `STATEMENT_OF_INTENT.md` - Project goals
|
||||
- `HYBRID_SETUP.md` - Hybrid deployment guide
|
||||
- `DOCKER_MIGRATION_PLAN.md` - Docker migration
|
||||
|
||||
---
|
||||
|
||||
## 🆘 Support
|
||||
|
||||
- **Git credentials**: `/home/wall-o/projects/gitea_credentials.txt`
|
||||
- **Python venv**: `/home/wall-o/workspace/venv`
|
||||
- **Docker**: User is in docker group
|
||||
|
||||
---
|
||||
|
||||
*Handoff created: 2026-07-20*
|
||||
*Last working commit: `ad2742c`*
|
||||
*Status: Ready for deployment*
|
||||
@@ -0,0 +1,83 @@
|
||||
# MTG Online Backend - Handoff Prompt
|
||||
|
||||
## Context
|
||||
You are taking over the MTG Online Backend project. The project is a Python FastAPI application that integrates with MTGJSON API to provide Magic: The Gathering card data. The system downloads, processes, and stores MTGJSON datasets in PostgreSQL with a weekly refresh cycle.
|
||||
|
||||
## Current Status
|
||||
- ✅ **All containers destroyed** and Docker pruned
|
||||
- ✅ **Code is working** - last tested successfully with 5.4M+ cards loaded
|
||||
- ⏸️ **Project paused** - ready for handoff
|
||||
|
||||
## Quick Start for New Thread
|
||||
|
||||
```bash
|
||||
# 1. Read the handoff documentation
|
||||
cat /home/wall-o/projects/mtgonline/HANDOFF.md
|
||||
|
||||
# 2. Check current state
|
||||
cat /home/wall-o/projects/mtgonline/state.json
|
||||
|
||||
# 3. Rebuild and deploy
|
||||
cd /home/wall-o/projects/mtgonline
|
||||
docker build -t mtgonline_backend backend/
|
||||
docker compose -p mtgonline up -d
|
||||
|
||||
# 4. Monitor startup
|
||||
docker logs -f mtgonline_backend
|
||||
```
|
||||
|
||||
## Key Information
|
||||
|
||||
**Location**: `/home/wall-o/projects/mtgonline`
|
||||
**Last Commit**: `ad2742c` - "Fix MTGJSON download: use .gz URLs and handle pre-uncompressed files"
|
||||
|
||||
**Core Service**: `backend/app/services/mtgjson_manager.py`
|
||||
- Downloads MTGJSON data from `https://mtgjson.com/api/v5`
|
||||
- Handles both gzip-compressed and pre-uncompressed JSON files
|
||||
- Upserts to PostgreSQL with `ON CONFLICT DO UPDATE`
|
||||
- Weekly refresh cycle (7 days)
|
||||
- Container marked unhealthy if data corrupted
|
||||
|
||||
**Known Issues Fixed**:
|
||||
1. MTGJSON URL scheme changed from `.json` to `.json.gz`
|
||||
2. Some files are pre-uncompressed (not actually gzipped)
|
||||
3. Large files need 60-minute download timeout
|
||||
|
||||
## What to Do Next
|
||||
|
||||
1. **Rebuild and test** the deployment
|
||||
2. **Verify MTGJSON data loads** correctly
|
||||
3. **Review the codebase** and make any needed improvements
|
||||
4. **Consider**:
|
||||
- Incremental updates (vs full refresh)
|
||||
- Better error handling
|
||||
- Database optimization
|
||||
- Monitoring and alerting
|
||||
|
||||
## Important Files
|
||||
- `HANOFOFF.md` - Complete documentation
|
||||
- `state.json` - Project state
|
||||
- `backend/app/services/mtgjson_manager.py` - Core MTGJSON integration
|
||||
- `docker-compose.yml` - Container orchestration
|
||||
- `.env` - Configuration
|
||||
|
||||
## State Tracking
|
||||
Update `state.json` after each significant change:
|
||||
```json
|
||||
{
|
||||
"task_description": "...",
|
||||
"current_step": "...",
|
||||
"files_created": [...],
|
||||
"files_modified": [...],
|
||||
"decisions": [...],
|
||||
"next_steps": [...],
|
||||
"blockers": [...],
|
||||
"commit_hash": "...",
|
||||
"timestamp": ...
|
||||
}
|
||||
```
|
||||
|
||||
## Git
|
||||
- Credentials: `/home/wall-o/projects/gitea_credentials.txt`
|
||||
- Commit after each milestone
|
||||
- Push changes regularly
|
||||
Reference in New Issue
Block a user