mtgonline/HANDOFF.md

MTG Online Backend - Handoff Document

🎯 Project Overview

A Python FastAPI backend service that integrates with MTGJSON API to provide comprehensive Magic: The Gathering card data. The system downloads, processes, and stores MTGJSON datasets in PostgreSQL with a weekly refresh cycle.

Location: /home/wall-o/projects/mtgonline
Last Commit: ad2742c
Status: ✅ Code complete, tested, ready for deployment

---

📋 Quick Start for New Assistant

cd /home/wall-o/projects/mtgonline

# 1. Read this document first
# 2. Build the backend image
docker build -t mtgonline_backend backend/

# 3. Deploy the stack
docker compose -p mtgonline up -d

# 4. Monitor startup (MTGJSON download takes ~10-15 minutes)
docker logs -f mtgonline_backend

---

🏗️ Architecture

Services

• Backend (FastAPI + MTGJSON integration)
• PostgreSQL (main + mtgdata - separate database for MTG data)
• Redis (caching)
• Refresh (weekly MTGJSON sync job)

Data Flow

MTGJSON API (https://mtgjson.com/api/v5)
    ↓
Download AllPrintings.json.gz (500MB+)
    ↓
Unpack & Validate
    ↓
Upsert to PostgreSQL (ON CONFLICT DO UPDATE)
    ↓
Health Check verifies data exists

---

🔑 Key Files

Core Integration

• backend/app/services/mtgjson_manager.py - Main MTGJSON service
  • Handles download, unpack, validate, upsert
  • Weekly refresh logic
  • Data integrity checks

Entry Points

• backend/app/main.py - FastAPI app
• backend/scripts/refresh_mtg.py - Refresh script
• backend/scripts/sanity_check_mtgjson.py - Validation script

Configuration

• docker-compose.yml - Production compose file
• docker-compose.dev.yml - Development compose
• .env - Environment variables
• state.json - Project state tracking

---

⚙️ Important Configuration

Environment Variables (.env)

MTGJSON_BASE_URL=https://mtgjson.com/api/v5
MTG_REFRESH_INTERVAL_DAYS=7
MTG_DOWNLOAD_TIMEOUT=3600  # 60 minutes
MTG_UPSERT_TIMEOUT=3600

Docker Compose

• Start Period: 600s (10 minutes for download)
• Volumes:
  • mtgonline_postgres_mtgdata_data - MTG database
  • mtgonline_mtg_data - Downloaded files
  • mtgonline_mtg_logs - Logs

---

🐛 Known Issues & Fixes

Issue 1: MTGJSON URL Scheme Change

Problem: MTGJSON changed from .json to .json.gz URLs

Fix: Updated URLs in mtgjson_manager.py:

MTGJSON_BASE_URL = "https://mtgjson.com/api/v5"
REQUIRED_FILES = {
    "AllPrintings.json.gz": MTGJSON_BASE_URL + "/AllPrintings.json.gz",
    # ...
}

Issue 2: Pre-uncompressed Files

Problem: Some .gz files are actually plain JSON

Fix: Check magic bytes before decompression:

with open(gz_file, 'rb') as f:
    magic = f.read(2)
if magic == b'\x1f\x8b':  # gzip
    # decompress
else:
    # just rename

Issue 3: Large File Download Times

Solution:

• Timeout set to 60 minutes
• Start period set to 10 minutes
• Files cached in volume

---

🧪 Testing Results (Last Test)

✅ All containers healthy

• mtgonline_backend - Healthy
• mtgonline_postgres - Healthy
• mtgonline_postgres_mtgdata - Healthy
• mtgonline_redis - Healthy
• mtgonline_refresh - Healthy

✅ Data loaded successfully

• 5,434,222 cards
• 5,398 sets
• 43 card types
• 205 keywords

---

📦 Downloaded Files

File	Size	Description
AllPrintings.json.gz	500MB+	Complete card data
AllSetFiles.zip	10MB+	Set metadata
AllIdentifiers.json.gz	100MB+	Card identifiers
CardTypes.json.gz	1MB+	Card type definitions
Keywords.json.gz	0.5MB+	Game keywords
SetList.json.gz	50MB+	Set information

---

🔄 Refresh Cycle

• Frequency: Weekly (7 days)
• Method: Full re-download and upsert
• Trigger: Cron job or backend startup
• Idempotent: Safe to run multiple times

---

🛠️ Manual Operations

Refresh Data

cd /home/wall-o/projects/mtgonline
bash backend/scripts/refresh_mtg.py

Run Sanity Check

python backend/scripts/sanity_check_mtgjson.py

Verify Data

docker exec -it mtgonline_postgres_mtgdata psql -U postgres -c "SELECT count(*) FROM mtg_cards;"

---

📝 State Tracking

Update state.json after each significant change:

{
  "task_description": "MTG Online Backend - MTGJSON Integration",
  "current_step": "...",
  "files_created": [...],
  "files_modified": [...],
  "decisions": [...],
  "next_steps": [...],
  "blockers": [...],
  "commit_hash": "...",
  "timestamp": ...
}

---

🚀 Next Steps

1. Deploy and test the stack
2. Monitor MTGJSON download progress
3. Verify all containers are healthy
4. Consider improvements:
   • Incremental updates (vs full refresh)
   • Better error handling
   • Database optimization
   • Monitoring and alerting

---

📚 Additional Documentation

• README.md - Project overview
• ROADMAP.md - Development roadmap
• STATEMENT_OF_INTENT.md - Project goals
• HYBRID_SETUP.md - Hybrid deployment guide
• DOCKER_MIGRATION_PLAN.md - Docker migration

---

🆘 Support

• Git credentials: /home/wall-o/projects/gitea_credentials.txt
• Python venv: /home/wall-o/workspace/venv
• Docker: User is in docker group

---

Handoff created: 2026-07-20
Last working commit: ad2742c
Status: Ready for deployment