feat(vector): replace inline page-image payloads with chunk_bbox (Deck #76)
Per-chunk PDF page renders (~150–700 KB base64 PNG each) were the dominant disk consumer in production, repeatedly tripping `No space left on device: WAL buffer size exceeds available disk space` on welcomed-malamute Qdrant. Replace the inline highlighted_page_image / highlighted_page_number / highlight_count fields with a small `chunk_bbox` field: list[(x0, y0, x1, y1)] of normalized [0, 1] floats, ~32 bytes per chunk. Astrolabe (the only known consumer) renders the highlight client-side as a percentage-positioned overlay on top of the existing /api/v1/pdf-preview render-on-demand path (cbcoutinho/astrolabe#76). - pdf_highlighter: new compute_chunk_bboxes_batch() that reuses the existing _find_chunk_bbox text-search path, skipping all pixmap/PIL/PNG work. - processor: store chunk_bbox + chunk_bbox_page in the Qdrant payload, drop highlighted_page_image + friends, drop the base64 import. - visualization /api/v1/chunk-context and auth/viz_routes: read chunk_bbox instead of highlighted_page_image. - vector/__init__: stop eagerly re-exporting `processor`/`scanner` — fixes a pre-existing circular import (search.algorithms -> vector.placeholder -> vector/__init__ -> processor -> scanner -> server.semantic -> search.bm25_hybrid -> search.algorithms partial). Test suite that was broken on master (test_bm25_hybrid.py et al.) now collects and passes. - scripts/purge_page_images.py: ad-hoc, idempotent migration that delete_payload's the legacy keys from existing points. No reindex required; legacy chunks render the page with no overlay. Pairs with cbcoutinho/astrolabe#76. Frontend handles missing chunk_bbox gracefully, so this can land in either order. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
2e1ab99a88
commit
ee402ea00e
@@ -604,8 +604,10 @@ async def chunk_context_endpoint(request: Request) -> JSONResponse:
|
||||
f"after_len={len(chunk_context.after_context)}"
|
||||
)
|
||||
|
||||
# For PDF files, also fetch the highlighted page image from Qdrant
|
||||
highlighted_page_image = None
|
||||
# For PDF files, also fetch the chunk bbox from Qdrant so the client
|
||||
# can overlay a highlight on top of a render-on-demand page image
|
||||
# (Deck #76).
|
||||
chunk_bbox = None
|
||||
page_number = None
|
||||
if doc_type == "file":
|
||||
try:
|
||||
@@ -613,7 +615,6 @@ async def chunk_context_endpoint(request: Request) -> JSONResponse:
|
||||
qdrant_client = await get_qdrant_client()
|
||||
username = request.user.display_name
|
||||
|
||||
# Query for this specific chunk's highlighted image
|
||||
points_response = await qdrant_client.scroll(
|
||||
collection_name=settings.get_collection_name(),
|
||||
scroll_filter=Filter(
|
||||
@@ -635,22 +636,20 @@ async def chunk_context_endpoint(request: Request) -> JSONResponse:
|
||||
),
|
||||
limit=1,
|
||||
with_vectors=False,
|
||||
with_payload=["highlighted_page_image", "page_number"],
|
||||
with_payload=["chunk_bbox", "page_number"],
|
||||
)
|
||||
|
||||
points = points_response[0]
|
||||
if points and points[0].payload:
|
||||
highlighted_page_image = points[0].payload.get(
|
||||
"highlighted_page_image"
|
||||
)
|
||||
chunk_bbox = points[0].payload.get("chunk_bbox")
|
||||
page_number = points[0].payload.get("page_number")
|
||||
if highlighted_page_image:
|
||||
if chunk_bbox:
|
||||
logger.info(
|
||||
f"Found highlighted image for chunk: "
|
||||
f"page={page_number}, image_size={len(highlighted_page_image)}"
|
||||
f"Found chunk bbox: page={page_number}, "
|
||||
f"rects={len(chunk_bbox)}"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to fetch highlighted image: {e}")
|
||||
logger.warning(f"Failed to fetch chunk bbox: {e}")
|
||||
|
||||
# Return response compatible with frontend expectations
|
||||
response_data: dict = {
|
||||
@@ -662,9 +661,8 @@ async def chunk_context_endpoint(request: Request) -> JSONResponse:
|
||||
"has_more_after": chunk_context.has_after_truncation,
|
||||
}
|
||||
|
||||
# Add image data if available
|
||||
if highlighted_page_image:
|
||||
response_data["highlighted_page_image"] = highlighted_page_image
|
||||
if chunk_bbox:
|
||||
response_data["chunk_bbox"] = chunk_bbox
|
||||
response_data["page_number"] = page_number
|
||||
|
||||
return JSONResponse(response_data)
|
||||
|
||||
Reference in New Issue
Block a user