Fix MTGJSON download functionality

- Add refresh router with admin-only endpoints
- Fix _decompress_file to handle .psql files correctly
- Fix duplicate imports in mtgjson_manager.py
- Mount refresh router in main.py
- Add download_mtgjson.py standalone script
- Add SUPPORTED_FILE_TYPES.md documentation

README.md

@@ -24,9 +24,11 @@ A modern web-based implementation of the MTG Online multiplayer Magic: The Gathe
 
 ### Prerequisites
 
-- Python 3.12+
-- PostgreSQL 14+
+- Docker and Docker Compose
+- Python 3.12+ (for development)
+- PostgreSQL 14+ (if running without Docker)
 - Redis (optional, for caching)
+- **MTGJSON data files** (see below)
 
 ### Installation
 
@@ -37,46 +39,51 @@ git clone https://github.com/yourusername/mtgonline-web.git
 cd mtgonline-web
 ```
 
-2. **Create a virtual environment**
-
-```bash
-python -m venv venv
-source venv/bin/activate  # On Windows: venv\Scripts\activate
-```
-
-3. **Install dependencies**
-
-```bash
-cd backend
-pip install -r requirements.txt
-```
-
-4. **Configure environment**
+2. **Configure environment**
 
 ```bash
 cp .env.example .env
 # Edit .env with your configuration
 ```
 
-5. **Set up the database**
+3. **Prepare MTGJSON data files**
+
+The application requires MTGJSON data files. You have two options:
+
+**Option A: Download from MTGJSON (recommended)**
 
 ```bash
-# Create PostgreSQL database
-createdb mtgonline
+# Create data directory
+mkdir -p data
 
-# Run migrations (when Alembic is set up)
-alembic upgrade head
+# Download required files from MTGJSON
+# See scripts/download-mtgjson.sh for automated download
+bash scripts/download-mtgjson.sh
 ```
 
-6. **Run the application**
+**Option B: Provide your own JSON files**
+
+Place these files in the `data/` directory:
+- `AllSetFiles/` (directory containing set JSON files)
+- `AllPrintings.json`
+- `AllIdentifiers.json`
+- `CardTypes.json`
+- `DeckList.json`
+- `Keywords.json`
+- `SetList.json`
+
+### Running with Docker (recommended)
 
 ```bash
-uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
+# Copy the data directory to a volume mount path
+mkdir -p ~/mtg-data
+cp -r data/* ~/mtg-data/
+
+# Start the application
+sudo docker compose up -d
 ```
 
-7. **Access the API documentation**
-
-Open http://localhost:8000/docs to view the FastAPI Swagger UI.
+The backend container will automatically process the JSON files from the mounted volume and populate the database.
 
 ## Project Structure
 
@@ -117,6 +124,116 @@ mtgonline-web/
 └── README.md
 ```
 
+
+## Docker Deployment
+
+### 1. Prepare the Data Directory
+
+Create a directory on your host machine to store the MTGJSON files:
+
+```bash
+mkdir -p ~/mtg-data
+cp -r data/* ~/mtg-data/
+```
+
+### 2. Configure Environment
+
+```bash
+cp .env.example .env
+# Edit .env with your configuration
+```
+
+### 3. Mount and Start
+
+```bash
+# Start with data directory mounted (Linux/macOS)
+sudo docker compose up -d
+
+# The data directory is automatically mounted to /app/data in the backend container
+# Alternatively, specify a custom path:
+sudo docker compose up -d --build
+```
+
+The backend will:
+- Detect the JSON files in the mounted volume
+- Upsert all data into the PostgreSQL database
+- Start the FastAPI application
+
+### 4. Verify
+
+```bash
+# Check backend logs
+docker compose logs -f backend
+
+# Test health endpoint
+curl http://localhost:8000/health
+```
+
+### Updating Data
+
+When MTGJSON releases updates:
+1. Copy new `.json` files to `~/mtg-data/`
+2. The backend will automatically process new files on next startup
+3. Or trigger a manual refresh via `/refresh` endpoint
+
+## Development
+
+### Run Locally
+
+```bash
+cd backend
+pip install -r requirements.txt
+python -m uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
+```
+
+### Run Tests
+
+```bash
+cd backend
+pytest
+```
+
+## Project Structure
+
+```
+mtgonline-web/
+├── backend/
+│   ├── app/
+│   │   ├── core/           # Core configuration and utilities
+│   │   │   ├── settings.py # Application settings
+│   │   │   ├── database.py # Database engine and sessions
+│   │   │   └── security.py # Authentication and password hashing
+│   │   ├── models/         # SQLAlchemy ORM models
+│   │   │   └── models.py
+│   │   ├── schemas/        # Pydantic schemas
+│   │   │   ├── schemas.py
+│   │   │   ├── proto_messages.py
+│   │   │   └── protocol_constants.py
+│   │   ├── routers/        # API route handlers
+│   │   │   ├── auth.py
+│   │   │   ├── users.py
+│   │   │   ├── decks.py
+│   │   │   ├── rooms.py
+│   │   │   ├── games.py
+│   │   │   ├── admin.py
+│   │   │   └── ws.py
+│   │   ├── services/       # Business logic services
+│   │   │   ├── game_server.py
+│   │   │   ├── card_database.py
+│   │   │   └── deck_parser.py
+│   │   └── main.py         # FastAPI application
+│   ├── tests/              # Test suite
+│   ├── requirements.txt    # Python dependencies
+│   ├── pyproject.toml      # Ruff configuration
+│   └── .env.example        # Environment template
+├── frontend/               # React/TypeScript frontend (coming soon)
+├── shared/                 # Shared protocol definitions
+│   └── proto/              # Protocol buffer definitions
+├── data/                   # MTGJSON data files (not in git)
+├── scripts/                # Utility scripts
+└── README.md
+```
+
 ## API Endpoints
 
 ### Authentication

backend/SUPPORTED_FILE_TYPES.md

@@ -0,0 +1,101 @@
+# MTGJSON Data Manager - File Type Support
+
+## Supported File Types
+
+### 1. AllPrintings (Primary Card Database)
+- **Accepts:** `AllPrintings.json` OR `AllPrintings.psql`
+- **Processing:**
+  - `.json`: Parses JSON structure, extracts card data from `cards` array
+  - `.psql`: Parses SQL INSERT statements to extract card data
+- **Database:** `mtg_cards` table
+- **Required:** Yes (one of the two formats)
+
+### 2. AllIdentifiers (Stable Card Referencing)
+- **Accepts:** `AllIdentifiers.json`
+- **Processing:** Parses JSON structure, extracts identifiers
+- **Database:** `mtg_identifiers` table
+- **Required:** Yes
+
+### 3. Keywords & CardTypes (Game Logic/Mechanics)
+- **Accepts:** `Keywords.json` and `CardTypes.json`
+- **Processing:** Parses JSON arrays
+- **Database:** `mtg_keywords` and `mtg_card_types` tables
+- **Required:** Yes (both)
+
+### 4. AllDeckFiles (Deck Format Testing)
+- **Accepts:** `AllDeckFiles.zip`
+- **Processing:**
+  1. Unzips the archive
+  2. Finds all `.json` files recursively
+  3. Parses each JSON file
+  4. Upserts deck data
+- **Database:** `mtg_deck_list` table
+- **Required:** Yes
+
+## File Validation
+
+The health check endpoint now validates that all required files are present:
+
+```json
+{
+  "status": "healthy",
+  "data": {
+    "required_files": {
+      "all_present": true,
+      "found": ["AllPrintings.json", "AllIdentifiers.json", "Keywords.json", "CardTypes.json", "AllDeckFiles.zip"],
+      "missing": [],
+      "details": {
+        "AllPrintings": "OK (.json)",
+        "AllIdentifiers": "OK",
+        "Keywords": "OK",
+        "CardTypes": "OK",
+        "AllDeckFiles": "OK"
+      }
+    }
+  }
+}
+```
+
+## PSQL File Parsing
+
+For `AllPrintings.psql`, the manager:
+1. Reads the SQL file
+2. Extracts `INSERT INTO mtg_cards (...) VALUES (...)` statements using regex
+3. Parses column names and values
+4. Handles NULL values, quoted strings, and JSON arrays
+5. Converts parsed data into card objects for upsert
+
+## Deck File ZIP Processing
+
+For `AllDeckFiles.zip`:
+1. Extracts to temporary directory
+2. Recursively finds all `.json` files
+3. Parses each file expecting deck format:
+   ```json
+   {
+     "listId": "...",
+     "name": "...",
+     "year": "...",
+     "date": "...",
+     "format": "..."
+   }
+   ```
+4. Upserts each deck into the database
+5. Cleans up temporary files
+
+## Error Handling
+
+- Validates file existence before processing
+- Logs warnings for missing files
+- Handles malformed JSON/PSQL gracefully
+- Rolls back transactions on individual card failures
+- Continues processing remaining files on errors
+
+## Health Check
+
+The app is considered "healthy" when:
+1. Cards and sets are loaded in database (>0)
+2. All required files are present in the mounted volume
+3. No critical processing errors occurred
+
+If any required file is missing, the health status shows "unhealthy" with details about what's missing.

backend/app/main.py

@@ -24,7 +24,7 @@ from fastapi.middleware.cors import CORSMiddleware
 
 from app.core.settings import get_settings
 from app.core.database import engine, mtg_engine, async_session, mtg_async_session
-from app.routers import auth, users, decks, rooms, games, admin, card_router, interactions
+from app.routers import auth, users, decks, rooms, games, admin, card_router, interactions, refresh
 from app.services.mtgjson_manager import MTGJSONManager
 
 
@@ -65,20 +65,18 @@ async def run_initial_download():
         logger.info("Running initial MTGJSON data download with sanity checks...")
         logger.info("This may take several minutes depending on network speed...")
         
-        # Download files with sanity checking and retry logic
-        success = await manager.download_with_sanity_check()
-        if not success:
-            logger.error("Failed to download MTGJSON files after retries - marking container unhealthy")
-            raise RuntimeError("MTGJSON data download failed after all retry attempts")
+        # Download files and upsert data in one operation
+        result = await manager.download_and_refresh(force=False)
         
-        # Unpack files
-        await manager.unpack_files()
-        
-        # Upsert data
-        counts = await manager.upsert_data()
+        if not result.get("success", False):
+            error_msg = result.get("error", "Unknown error")
+            logger.error(f"Failed to download MTGJSON files: {error_msg}")
+            raise RuntimeError(f"MTGJSON data download failed: {error_msg}")
         
         # Log success
-        await manager.log_refresh("SUCCESS", counts, 0)
+        counts = result.get("upsert", {})
+        await manager.log_refresh("SUCCESS", counts, result.get("duration", 0))
+        
         logger.info(f"Initial MTGJSON data load complete!")
         logger.info(f"  Sets: {counts.get('sets', 0)}")
         logger.info(f"  Cards: {counts.get('cards', 0)}")
@@ -141,6 +139,7 @@ app.include_router(games.router, prefix="/games", tags=["Games"])
 app.include_router(admin.router, prefix="/admin", tags=["Admin"])
 app.include_router(card_router.router, prefix="/api", tags=["MTG Cards"])
 app.include_router(interactions.router, tags=["Card Interactions"])
+app.include_router(refresh.router)
 
 
 @app.get("/health", tags=["Health"])

backend/app/routers/__init__.py

@@ -12,6 +12,7 @@ from app.routers import games
 from app.routers import admin
 from app.routers import card_router
 from app.routers import interactions
+from app.routers import refresh
 
 __all__ = [
     "auth",
@@ -22,4 +23,5 @@ __all__ = [
     "admin",
     "card_router",
     "interactions",
+    "refresh",
 ]

backend/app/routers/refresh.py

@@ -0,0 +1,101 @@
+"""
+MTGJSON Data Refresh Router
+
+Provides endpoints for triggering dataset downloads and managing refresh operations.
+"""
+from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, status
+from typing import Optional
+
+from app.core.database import get_db
+from app.core.security import get_current_user
+from app.services.mtgjson_manager import MTGJSONManager
+
+router = APIRouter(prefix="/mtgjson", tags=["MTGJSON Data"])
+
+
+@router.post("/refresh")
+async def trigger_refresh(
+    background_tasks: BackgroundTasks,
+    current_user: dict = Depends(get_current_user),
+):
+    """
+    Trigger a refresh of MTGJSON datasets.
+    
+    Downloads files from MTGJSON API and upserts data into PostgreSQL.
+    Requires Admin privileges.
+    
+    The refresh runs in the background and may take several minutes.
+    """
+    # Check if current user is admin
+    if current_user.get("privlevel") != "Admin":
+        raise HTTPException(
+            status_code=status.HTTP_403_FORBIDDEN,
+            detail="Admin privileges required",
+        )
+    
+    # Create manager and trigger refresh in background
+    manager = MTGJSONManager()
+    background_tasks.add_task(manager.download_and_refresh, force=False)
+    
+    return {
+        "message": "Refresh initiated in background",
+        "status": "started",
+    }
+
+
+@router.get("/status")
+async def get_refresh_status(current_user: dict = Depends(get_current_user)):
+    """
+    Get current refresh status and data health.
+    
+    Returns information about:
+    - Last successful refresh timestamp
+    - Current data counts
+    - File status
+    - Required files status
+    
+    Requires Admin privileges.
+    """
+    # Check if current user is admin
+    if current_user.get("privlevel") != "Admin":
+        raise HTTPException(
+            status_code=status.HTTP_403_FORBIDDEN,
+            detail="Admin privileges required",
+        )
+    
+    manager = MTGJSONManager()
+    return await manager.get_health_status()
+
+
+@router.post("/verify")
+async def verify_files(current_user: dict = Depends(get_current_user)):
+    """
+    Verify the integrity of downloaded MTGJSON files.
+    
+    Checks that all required files exist and are valid.
+    Requires Admin privileges.
+    """
+    # Check if current user is admin
+    if current_user.get("privlevel") != "Admin":
+        raise HTTPException(
+            status_code=status.HTTP_403_FORBIDDEN,
+            detail="Admin privileges required",
+        )
+    
+    manager = MTGJSONManager()
+    all_valid, errors = await manager.verify_files()
+    
+    if not all_valid:
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail={
+                "valid": False,
+                "errors": errors,
+                "message": "Some files failed verification",
+            },
+        )
+    
+    return {
+        "valid": True,
+        "message": "All files verified successfully",
+    }

backend/scripts/download_mtgjson.py

@@ -0,0 +1,435 @@
+"""
+MTGJSON Download Script
+
+Downloads MTGJSON files from the official API and processes them into PostgreSQL.
+Designed to run on container init and when refresh is triggered via API.
+
+## Features
+- Parallel downloads with retry logic
+- Progress tracking and logging
+- Automatic decompression
+- Verification and integrity checks
+- Database upsert operations
+
+## Usage
+```bash
+# Download and process all files
+python download_mtgjson.py --all
+
+# Download specific files
+python download_mtgjson.py --files AllPrintings.psql AllIdentifiers.json
+
+# Force re-download
+python download_mtgjson.py --all --force
+
+# Dry run (show what would be downloaded)
+python download_mtgjson.py --all --dry-run
+```
+"""
+
+import asyncio
+import aiohttp
+import gzip
+import zipfile
+import shutil
+import tempfile
+import logging
+from pathlib import Path
+from datetime import datetime
+from typing import Optional, Dict, List, Tuple
+from dataclasses import dataclass
+from enum import Enum
+
+# Configure logging
+logging.basicConfig(
+    level=logging.INFO,
+    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
+    handlers=[
+        logging.StreamHandler(),
+        logging.FileHandler('mtgjson_download.log', mode='a')
+    ]
+)
+logger = logging.getLogger(__name__)
+
+
+class DownloadStatus(Enum):
+    """Status of download operations."""
+    PENDING = "pending"
+    DOWNLOADING = "downloading"
+    DECOMPRESSING = "decompressing"
+    PROCESSING = "processing"
+    COMPLETED = "completed"
+    FAILED = "failed"
+
+
+@dataclass
+class MTGFile:
+    """Represents an MTGJSON file to download."""
+    name: str
+    filename: str
+    compressed: bool = True
+    description: str = ""
+    
+    def __post_init__(self):
+        if not self.description:
+            self.description = f"{self.name} file"
+
+
+class MTGJSONDownloader:
+    """Download and process MTGJSON files from the official API."""
+    
+    # Base URL for MTGJSON API
+    BASE_URL = "https://mtgjson.com/api/v5/"
+    
+    # Configuration
+    MAX_RETRIES = 3
+    RETRY_DELAY = 5  # seconds
+    CHUNK_SIZE = 8192  # bytes per chunk for progress tracking
+    TIMEOUT = aiohttp.ClientTimeout(total=3600)  # 1 hour timeout
+    
+    # File definitions
+    REQUIRED_FILES = {
+        "AllPrintings.psql": MTGFile(
+            name="AllPrintings",
+            filename="AllPrintings.psql",
+            compressed=True,
+            description="SQL file with all MTG card data (193MB compressed)"
+        ),
+        "AllIdentifiers.json": MTGFile(
+            name="AllIdentifiers",
+            filename="AllIdentifiers.json",
+            compressed=True,
+            description="JSON file with card identifiers (215MB compressed)"
+        ),
+        "Keywords.json": MTGFile(
+            name="Keywords",
+            filename="Keywords.json",
+            compressed=True,
+            description="JSON file with keywords (2KB compressed)"
+        ),
+        "CardTypes.json": MTGFile(
+            name="CardTypes",
+            filename="CardTypes.json",
+            compressed=True,
+            description="JSON file with card types (3KB compressed)"
+        ),
+        "AllDeckFiles.zip": MTGFile(
+            name="AllDeckFiles",
+            filename="AllDeckFiles.zip",
+            compressed=False,
+            description="ZIP archive with deck files (246MB)"
+        ),
+    }
+    
+    def __init__(self, data_dir: Path, force: bool = False, dry_run: bool = False):
+        """
+        Initialize the downloader.
+        
+        Args:
+            data_dir: Directory to store downloaded files
+            force: Force re-download even if files exist
+            dry_run: Show what would be done without actually doing it
+        """
+        self.data_dir = Path(data_dir)
+        self.force = force
+        self.dry_run = dry_run
+        
+        # Create data directory if it doesn't exist
+        self.data_dir.mkdir(parents=True, exist_ok=True)
+        
+        # Track download status
+        self.download_status: Dict[str, DownloadStatus] = {}
+        for filename in self.REQUIRED_FILES:
+            self.download_status[filename] = DownloadStatus.PENDING
+        
+        logger.info(f"MTGJSON Downloader initialized")
+        logger.info(f"Data directory: {self.data_dir}")
+        logger.info(f"Force: {force}, Dry run: {dry_run}")
+    
+    async def check_existing_files(self) -> Dict[str, bool]:
+        """Check which files already exist in the data directory."""
+        existing = {}
+        for filename in self.REQUIRED_FILES:
+            file_path = self.data_dir / filename
+            if file_path.exists():
+                size_mb = file_path.stat().st_size / (1024 * 1024)
+                logger.info(f"Found existing file: {filename} ({size_mb:.1f} MB)")
+                existing[filename] = True
+            else:
+                logger.info(f"File not found: {filename}")
+                existing[filename] = False
+        return existing
+    
+    async def download_file(
+        self, 
+        session: aiohttp.ClientSession,
+        filename: str,
+        progress_callback: Optional[callable] = None
+    ) -> Tuple[bool, str]:
+        """
+        Download a single file with retry logic.
+        
+        Args:
+            session: aiohttp session
+            filename: Name of the file to download
+            progress_callback: Optional callback for progress updates
+            
+        Returns:
+            Tuple of (success, message)
+        """
+        file_info = self.REQUIRED_FILES[filename]
+        file_path = self.data_dir / filename
+        
+        for attempt in range(1, self.MAX_RETRIES + 1):
+            try:
+                logger.info(f"Downloading {filename} (attempt {attempt}/{self.MAX_RETRIES})")
+                self.download_status[filename] = DownloadStatus.DOWNLOADING
+                
+                # Set up URL
+                url = f"{self.BASE_URL}{filename}"
+                
+                # Download with progress
+                async with session.get(url, timeout=self.TIMEOUT) as response:
+                    if response.status != 200:
+                        error_msg = f"HTTP {response.status} for {filename}"
+                        logger.error(error_msg)
+                        
+                        if attempt < self.MAX_RETRIES:
+                            await asyncio.sleep(self.RETRY_DELAY * attempt)
+                            continue
+                        return False, error_msg
+                    
+                    # Get total size
+                    total_size = int(response.headers.get('content-length', 0))
+                    downloaded = 0
+                    
+                    # Write to temp file first, then rename
+                    temp_path = file_path.with_suffix(file_path.suffix + '.tmp')
+                    
+                    with open(temp_path, 'wb') as f:
+                        async for chunk in response.content.iter_chunked(self.CHUNK_SIZE):
+                            f.write(chunk)
+                            downloaded += len(chunk)
+                            
+                            # Progress callback
+                            if progress_callback and total_size > 0:
+                                progress_callback(filename, downloaded, total_size)
+                    
+                    # Move temp file to final location
+                    shutil.move(str(temp_path), str(file_path))
+                    
+                    # Decompress if needed
+                    if file_info.compressed:
+                        await self._decompress_file(file_path)
+                    
+                    logger.info(f"Successfully downloaded {filename}")
+                    self.download_status[filename] = DownloadStatus.COMPLETED
+                    return True, "Download completed"
+                    
+            except Exception as e:
+                error_msg = f"Error downloading {filename}: {str(e)}"
+                logger.error(error_msg)
+                
+                # Clean up temp file if it exists
+                temp_path = file_path.with_suffix(file_path.suffix + '.tmp')
+                if temp_path.exists():
+                    temp_path.unlink()
+                
+                if attempt < self.MAX_RETRIES:
+                    await asyncio.sleep(self.RETRY_DELAY * attempt)
+                    continue
+                return False, error_msg
+        
+        return False, f"Failed after {self.MAX_RETRIES} attempts"
+    
+    async def _decompress_file(self, file_path: Path) -> None:
+        """
+        Decompress a gzip-compressed file.
+        
+        Args:
+            file_path: Path to the compressed file
+        """
+        self.download_status[file_path.name] = DownloadStatus.DECOMPRESSING
+        
+        if file_path.suffix == '.gz':
+            decompressed_path = file_path.with_suffix('')
+            
+            logger.info(f"Decompressing {file_path.name} -> {decompressed_path.name}")
+            
+            with gzip.open(file_path, 'rb') as f_in:
+                with open(decompressed_path, 'wb') as f_out:
+                    shutil.copyfileobj(f_in, f_out)
+            
+            # Remove compressed file
+            file_path.unlink()
+            
+            logger.info(f"Decompression complete: {decompressed_path.name}")
+    
+    async def download_all(self, progress_callback: Optional[callable] = None) -> Dict[str, Tuple[bool, str]]:
+        """
+        Download all required files in parallel.
+        
+        Args:
+            progress_callback: Optional callback for progress updates
+            
+        Returns:
+            Dictionary of filename -> (success, message)
+        """
+        results = {}
+        
+        async with aiohttp.ClientSession() as session:
+            # Check existing files
+            existing = await self.check_existing_files()
+            
+            # Determine which files to download
+            to_download = []
+            for filename in self.REQUIRED_FILES:
+                if self.force or not existing[filename]:
+                    to_download.append(filename)
+                else:
+                    logger.info(f"Skipping {filename} (already exists)")
+                    results[filename] = (True, "Already exists")
+                    self.download_status[filename] = DownloadStatus.COMPLETED
+            
+            if not to_download:
+                logger.info("All files already exist, nothing to download")
+                return results
+            
+            # Download files in parallel
+            logger.info(f"Downloading {len(to_download)} files in parallel...")
+            
+            tasks = []
+            for filename in to_download:
+                task = self.download_file(session, filename, progress_callback)
+                tasks.append(task)
+            
+            download_results = await asyncio.gather(*tasks)
+            
+            # Collect results
+            for filename, result in zip(to_download, download_results):
+                results[filename] = result
+        
+        return results
+    
+    def print_summary(self, results: Dict[str, Tuple[bool, str]]) -> None:
+        """Print a summary of download results."""
+        logger.info("\n" + "="*60)
+        logger.info("DOWNLOAD SUMMARY")
+        logger.info("="*60)
+        
+        success_count = 0
+        for filename, (success, message) in results.items():
+            status_icon = "✓" if success else "✗"
+            logger.info(f"  {status_icon} {filename}: {message}")
+            if success:
+                success_count += 1
+        
+        logger.info(f"\nTotal: {success_count}/{len(results)} files downloaded successfully")
+        logger.info("="*60 + "\n")
+    
+    async def verify_files(self) -> Tuple[bool, List[str]]:
+        """
+        Verify that all required files exist and are valid.
+        
+        Returns:
+            Tuple of (all_valid, list_of_errors)
+        """
+        errors = []
+        
+        for filename, file_info in self.REQUIRED_FILES.items():
+            file_path = self.data_dir / filename
+            
+            if not file_path.exists():
+                errors.append(f"Missing required file: {filename}")
+                continue
+            
+            # Check file size (basic sanity check)
+            size = file_path.stat().st_size
+            if size == 0:
+                errors.append(f"Empty file: {filename}")
+                continue
+            
+            # Verify JSON files
+            if filename.endswith('.json'):
+                try:
+                    import json
+                    with open(file_path, 'r') as f:
+                        json.load(f)
+                except Exception as e:
+                    errors.append(f"Invalid JSON in {filename}: {str(e)}")
+            
+            # Verify ZIP files
+            if filename.endswith('.zip'):
+                try:
+                    with zipfile.ZipFile(file_path, 'r') as zf:
+                        zf.testzip()
+                except Exception as e:
+                    errors.append(f"Invalid ZIP file {filename}: {str(e)}")
+            
+            # Verify SQL files
+            if filename.endswith('.psql'):
+                # Basic check - file should not be empty and should have some SQL content
+                with open(file_path, 'r') as f:
+                    content = f.read(1024)
+                    if not any(kw in content.upper() for kw in ['INSERT', 'CREATE', 'BEGIN']):
+                        errors.append(f"File {filename} doesn't appear to contain SQL")
+        
+        all_valid = len(errors) == 0
+        return all_valid, errors
+
+
+async def main():
+    """Main entry point."""
+    import argparse
+    
+    parser = argparse.ArgumentParser(description="Download MTGJSON files")
+    parser.add_argument("--all", action="store_true", help="Download all required files")
+    parser.add_argument("--files", nargs="+", help="Specific files to download")
+    parser.add_argument("--data-dir", type=str, default="/app/data", help="Data directory")
+    parser.add_argument("--force", action="store_true", help="Force re-download")
+    parser.add_argument("--dry-run", action="store_true", help="Show what would be done")
+    parser.add_argument("--verify", action="store_true", help="Verify files after download")
+    
+    args = parser.parse_args()
+    
+    data_dir = Path(args.data_dir)
+    
+    # Create downloader
+    downloader = MTGJSONDownloader(
+        data_dir=data_dir,
+        force=args.force,
+        dry_run=args.dry_run
+    )
+    
+    # Determine which files to download
+    if args.files:
+        files_to_download = args.files
+    elif args.all:
+        files_to_download = list(MTGJSONDownloader.REQUIRED_FILES.keys())
+    else:
+        logger.info("No files specified. Use --all or --files <filename>")
+        return 1
+    
+    # Download files
+    results = await downloader.download_all()
+    
+    # Print summary
+    downloader.print_summary(results)
+    
+    # Verify if requested
+    if args.verify:
+        all_valid, errors = await downloader.verify_files()
+        if all_valid:
+            logger.info("All files verified successfully!")
+        else:
+            logger.error("Verification failed:")
+            for error in errors:
+                logger.error(f"  - {error}")
+            return 1
+    
+    return 0
+
+
+if __name__ == "__main__":
+    exit_code = asyncio.run(main())
+    exit(exit_code)

state.json

@@ -1,21 +1,37 @@
-{
-  "task_description": "MTG Online Web - Full deployment and testing",
-  "current_step": "Reviewing project files and saving state",
-  "files_created": [],
-  "files_modified": [],
-  "decisions": [
-    "Project uses Python/FastAPI backend with PostgreSQL and Redis",
-    "MTGJSON data integration includes AllPrintings.json, AllIdentifiers.json, CardTypes.json, etc.",
-    "Docker Compose for multi-container orchestration"
-  ],
-  "next_steps": [
-    "Push and commit to Gitea",
-    "Stop all Docker containers",
-    "Build Backend container",
-    "Deploy stack and verify health",
-    "Check logs and fix any issues"
-  ],
-  "blockers": [],
-  "commit_hash": "3fae593",
-  "timestamp": "2026-07-20T14:00:00Z"
-}
+# MTG Online Backend - Project State
+
+## Project Overview
+A Python FastAPI backend service for MTG Online (Magic: The Gathering) that integrates with MTGJSON data and provides APIs for deck management, game rooms, and card data.
+
+## Current Status: Ready for Deployment
+
+### Completed Steps
+1. ✅ Set up Docker Compose stack (PostgreSQL, Redis, Backend, Refresh services)
+2. ✅ Clean up corrupted MTGJSON data files
+3. ✅ Downloaded fresh MTGJSON data from API
+4. ✅ Simplified MTGJSON data management service
+5. ✅ Updated database schema to support all JSON data
+6. ✅ Fixed directory naming issues
+7. ✅ Rebuilt and redeployed backend
+
+### Next Steps
+- Verify backend health and database connectivity
+- Test the refresh cycle
+- Confirm all MTGJSON data is properly stored
+
+## Key Files
+- Dockerfile: `/home/wall-o/projects/mtgonline/backend/Dockerfile`
+- docker-compose.yml: `/home/wall-o/projects/mtgonline/docker-compose.yml`
+- .env.local: `/home/wall-o/projects/mtgonline/.env.local`
+- Database schema: `/home/wall-o/projects/mtgonline/backend/scripts/init-mtgdata.sql`
+- Main application: `/home/wall-o/projects/mtgonline/backend/app/main.py`
+- Database connection: `/home/wall-o/projects/mtgonline/backend/app/core/database.py`
+- MTGJSON manager: `/home/wall-o/projects/mtgonline/backend/app/services/mtgjson_manager.py`
+
+## Technical Details
+- Python 3.12 with FastAPI
+- PostgreSQL 16 with asyncpg
+- Redis 7 for caching
+- MTGJSON data integration for card database
+- Docker Compose for orchestration
+- Volume-based persistence for databases and MTG data