Fix MTGJSON download: use .json files, remove gzip unpacking

- Changed REQUIRED_FILES to REQUIRED_JSON_FILES and REQUIRED_ZIP_FILES
- AllSetFiles.zip kept as zip (only available compressed on MTGJSON)
- All other files downloaded as .json (no compression)
- Removed gzip import and unpacking logic for .gz files
- Updated EXPECTED_MIN_SIZES to reflect actual .json file sizes
- Removed validation for non-existent files
This commit is contained in:
2026-07-20 13:59:45 +00:00
parent 2369c2cc8c
commit 3fae593a22
3 changed files with 127 additions and 109 deletions
+16 -32
View File
@@ -1,43 +1,27 @@
{
"task_description": "MTG Online Backend - MTGJSON Data Integration & Docker Deployment",
"current_step": "Testing full container lifecycle",
"task_description": "Update MTGJSON data manager to download .json files instead of .json.gz files, remove unnecessary unpacking logic",
"current_step": "Docker image rebuilt with updated code",
"files_created": [
"backend/app/services/mtgjson_manager.py",
"backend/scripts/sanity_check_mtgjson.py",
"backend/scripts/verify_mtgjson_data.py",
"backend/scripts/maintain_mtgjson.sh"
"backend/app/services/mtgjson_manager.py"
],
"files_modified": [
"backend/app/services/mtgjson_manager.py",
"backend/app/scripts/refresh_mtg.py",
"backend/app/main.py",
"docker-compose.yml",
"docker-compose.dev.yml",
"backend/Dockerfile"
"backend/app/services/mtgjson_manager.py"
],
"decisions": [
"Created comprehensive MTGJSONManager service in app/services/",
"Manager handles download, unpack, upsert with ON CONFLICT DO UPDATE",
"Startup triggers initial download on first container init",
"Health check verifies MTG data exists in database",
"Refresh interval set to 7 days (weekly)",
"Docker compose start_period set to 600s for download time",
"Added file size validation against expected minimums",
"Implemented retry logic with exponential backoff (up to 3 attempts)",
"Cleanup deletes corrupted data before retry",
"Container marked unhealthy if validation fails after all retries",
"Download timeout increased to 60 minutes (3600s) for large files"
"Changed REQUIRED_FILES to REQUIRED_JSON_FILES and REQUIRED_ZIP_FILES",
"AllSetFiles.zip kept as zip (only available compressed on MTGJSON API)",
"All other files downloaded as .json (no compression)",
"Removed gzip import and unpacking logic for .gz files",
"Updated EXPECTED_MIN_SIZES to reflect actual .json file sizes (500MB+ for large files)",
"Removed validation for non-existent files (MagicSets.json, MagicRoots.json, SetTranslations.json)"
],
"next_steps": [
"Save state to state.json",
"Commit and push to Gitea",
"Stop and destroy all Docker containers",
"Build backend Docker container",
"Deploy stack and monitor",
"Check container health status",
"Verify logs show successful initialization"
"Restart Docker container with new image",
"Run data refresh to verify download and upsert work correctly",
"Monitor logs for any issues",
"Verify database has correct data"
],
"blockers": [],
"commit_hash": "0643544",
"timestamp": 1784598400
"commit_hash": "",
"timestamp": "2026-07-20T13:52:00Z"
}