mtgonline/HANDOFF.md

# MTG Online Backend - Handoff Document

## 🎯 Project Overview
A Python FastAPI backend service that integrates with MTGJSON API to provide comprehensive Magic: The Gathering card data. The system downloads, processes, and stores MTGJSON datasets in PostgreSQL with a weekly refresh cycle.

**Location**: `/home/wall-o/projects/mtgonline`  
**Last Commit**: `ad2742c`  
**Status**: ✅ Code complete, tested, ready for deployment

---

## 📋 Quick Start for New Assistant

```bash
cd /home/wall-o/projects/mtgonline

# 1. Read this document first
# 2. Build the backend image
docker build -t mtgonline_backend backend/

# 3. Deploy the stack
docker compose -p mtgonline up -d

# 4. Monitor startup (MTGJSON download takes ~10-15 minutes)
docker logs -f mtgonline_backend
```

---

## 🏗️ Architecture

### Services
- **Backend** (FastAPI + MTGJSON integration)
- **PostgreSQL** (main + mtgdata - separate database for MTG data)
- **Redis** (caching)
- **Refresh** (weekly MTGJSON sync job)

### Data Flow
```
MTGJSON API (https://mtgjson.com/api/v5)
    ↓
Download AllPrintings.json.gz (500MB+)
    ↓
Unpack & Validate
    ↓
Upsert to PostgreSQL (ON CONFLICT DO UPDATE)
    ↓
Health Check verifies data exists
```

---

## 🔑 Key Files

### Core Integration
- `backend/app/services/mtgjson_manager.py` - **Main MTGJSON service**
  - Handles download, unpack, validate, upsert
  - Weekly refresh logic
  - Data integrity checks

### Entry Points
- `backend/app/main.py` - FastAPI app
- `backend/scripts/refresh_mtg.py` - Refresh script
- `backend/scripts/sanity_check_mtgjson.py` - Validation script

### Configuration
- `docker-compose.yml` - Production compose file
- `docker-compose.dev.yml` - Development compose
- `.env` - Environment variables
- `state.json` - Project state tracking

---

## ⚙️ Important Configuration

### Environment Variables (.env)
```bash
MTGJSON_BASE_URL=https://mtgjson.com/api/v5
MTG_REFRESH_INTERVAL_DAYS=7
MTG_DOWNLOAD_TIMEOUT=3600  # 60 minutes
MTG_UPSERT_TIMEOUT=3600
```

### Docker Compose
- **Start Period**: 600s (10 minutes for download)
- **Volumes**:
  - `mtgonline_postgres_mtgdata_data` - MTG database
  - `mtgonline_mtg_data` - Downloaded files
  - `mtgonline_mtg_logs` - Logs

---

## 🐛 Known Issues & Fixes

### Issue 1: MTGJSON URL Scheme Change
**Problem**: MTGJSON changed from `.json` to `.json.gz` URLs

**Fix**: Updated URLs in `mtgjson_manager.py`:
```python
MTGJSON_BASE_URL = "https://mtgjson.com/api/v5"
REQUIRED_FILES = {
    "AllPrintings.json.gz": MTGJSON_BASE_URL + "/AllPrintings.json.gz",
    # ...
}
```

### Issue 2: Pre-uncompressed Files
**Problem**: Some `.gz` files are actually plain JSON

**Fix**: Check magic bytes before decompression:
```python
with open(gz_file, 'rb') as f:
    magic = f.read(2)
if magic == b'\x1f\x8b':  # gzip
    # decompress
else:
    # just rename
```

### Issue 3: Large File Download Times
**Solution**: 
- Timeout set to 60 minutes
- Start period set to 10 minutes
- Files cached in volume

---

## 🧪 Testing Results (Last Test)

✅ **All containers healthy**
- `mtgonline_backend` - Healthy
- `mtgonline_postgres` - Healthy  
- `mtgonline_postgres_mtgdata` - Healthy
- `mtgonline_redis` - Healthy
- `mtgonline_refresh` - Healthy

✅ **Data loaded successfully**
- 5,434,222 cards
- 5,398 sets
- 43 card types
- 205 keywords

---

## 📦 Downloaded Files

| File | Size | Description |
|------|------|-------------|
| `AllPrintings.json.gz` | 500MB+ | Complete card data |
| `AllSetFiles.zip` | 10MB+ | Set metadata |
| `AllIdentifiers.json.gz` | 100MB+ | Card identifiers |
| `CardTypes.json.gz` | 1MB+ | Card type definitions |
| `Keywords.json.gz` | 0.5MB+ | Game keywords |
| `SetList.json.gz` | 50MB+ | Set information |

---

## 🔄 Refresh Cycle

- **Frequency**: Weekly (7 days)
- **Method**: Full re-download and upsert
- **Trigger**: Cron job or backend startup
- **Idempotent**: Safe to run multiple times

---

## 🛠️ Manual Operations

### Refresh Data
```bash
cd /home/wall-o/projects/mtgonline
bash backend/scripts/refresh_mtg.py
```

### Run Sanity Check
```bash
python backend/scripts/sanity_check_mtgjson.py
```

### Verify Data
```bash
docker exec -it mtgonline_postgres_mtgdata psql -U postgres -c "SELECT count(*) FROM mtg_cards;"
```

---

## 📝 State Tracking

Update `state.json` after each significant change:
```json
{
  "task_description": "MTG Online Backend - MTGJSON Integration",
  "current_step": "...",
  "files_created": [...],
  "files_modified": [...],
  "decisions": [...],
  "next_steps": [...],
  "blockers": [...],
  "commit_hash": "...",
  "timestamp": ...
}
```

---

## 🚀 Next Steps

1. **Deploy and test** the stack
2. **Monitor** MTGJSON download progress
3. **Verify** all containers are healthy
4. **Consider improvements**:
   - Incremental updates (vs full refresh)
   - Better error handling
   - Database optimization
   - Monitoring and alerting

---

## 📚 Additional Documentation

- `README.md` - Project overview
- `ROADMAP.md` - Development roadmap
- `STATEMENT_OF_INTENT.md` - Project goals
- `HYBRID_SETUP.md` - Hybrid deployment guide
- `DOCKER_MIGRATION_PLAN.md` - Docker migration

---

## 🆘 Support

- **Git credentials**: `/home/wall-o/projects/gitea_credentials.txt`
- **Python venv**: `/home/wall-o/workspace/venv`
- **Docker**: User is in docker group

---

*Handoff created: 2026-07-20*  
*Last working commit: `ad2742c`*  
*Status: Ready for deployment*