feat(vector): replace inline page-image payloads with chunk_bbox (Deck #76)

Per-chunk PDF page renders (~150–700 KB base64 PNG each) were the dominant
disk consumer in production, repeatedly tripping `No space left on device:
WAL buffer size exceeds available disk space` on welcomed-malamute Qdrant.

Replace the inline highlighted_page_image / highlighted_page_number /
highlight_count fields with a small `chunk_bbox` field:
list[(x0, y0, x1, y1)] of normalized [0, 1] floats, ~32 bytes per chunk.
Astrolabe (the only known consumer) renders the highlight client-side as
a percentage-positioned overlay on top of the existing /api/v1/pdf-preview
render-on-demand path (cbcoutinho/astrolabe#76).

- pdf_highlighter: new compute_chunk_bboxes_batch() that reuses the
  existing _find_chunk_bbox text-search path, skipping all pixmap/PIL/PNG
  work.
- processor: store chunk_bbox + chunk_bbox_page in the Qdrant payload,
  drop highlighted_page_image + friends, drop the base64 import.
- visualization /api/v1/chunk-context and auth/viz_routes: read
  chunk_bbox instead of highlighted_page_image.
- vector/__init__: stop eagerly re-exporting `processor`/`scanner` —
  fixes a pre-existing circular import (search.algorithms ->
  vector.placeholder -> vector/__init__ -> processor -> scanner ->
  server.semantic -> search.bm25_hybrid -> search.algorithms partial).
  Test suite that was broken on master (test_bm25_hybrid.py et al.) now
  collects and passes.
- scripts/purge_page_images.py: ad-hoc, idempotent migration that
  delete_payload's the legacy keys from existing points. No reindex
  required; legacy chunks render the page with no overlay.

Pairs with cbcoutinho/astrolabe#76. Frontend handles missing chunk_bbox
gracefully, so this can land in either order.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-05-08 21:16:00 +02:00
co-authored by Claude Opus 4.7
parent 2e1ab99a88
commit ee402ea00e
8 changed files with 428 additions and 72 deletions
+9 -9
View File
@@ -553,9 +553,10 @@ async def get_chunk_context(request: Request) -> JSONResponse:
status_code=404,
)
# For PDF files, also fetch the highlighted page image from Qdrant if available
# This is useful for clients that want to show a pre-rendered image
highlighted_page_image = None
# For PDF files, also fetch the chunk's bounding box from Qdrant if
# available so the client can overlay a highlight on top of a
# render-on-demand page image (Deck #76).
chunk_bbox = None
page_number = chunk_context.page_number
if doc_type == "file":
@@ -563,7 +564,6 @@ async def get_chunk_context(request: Request) -> JSONResponse:
settings = get_settings()
qdrant_client = await get_qdrant_client()
# Query for this specific chunk's highlighted image
points_response = await qdrant_client.scroll(
collection_name=settings.get_collection_name(),
scroll_filter=Filter(
@@ -585,19 +585,19 @@ async def get_chunk_context(request: Request) -> JSONResponse:
),
limit=1,
with_vectors=False,
with_payload=["highlighted_page_image", "page_number"],
with_payload=["chunk_bbox", "page_number"],
)
if points_response[0]:
payload = points_response[0][0].payload
if payload:
highlighted_page_image = payload.get("highlighted_page_image")
chunk_bbox = payload.get("chunk_bbox")
# Trust Qdrant page number if available (might be more accurate than context expansion logic)
if payload.get("page_number") is not None:
page_number = payload.get("page_number")
except Exception as e:
logger.warning(f"Failed to fetch highlighted image: {e}")
logger.warning(f"Failed to fetch chunk bbox: {e}")
# Build response
response_data = {
@@ -612,8 +612,8 @@ async def get_chunk_context(request: Request) -> JSONResponse:
"total_chunks": chunk_context.total_chunks,
}
if highlighted_page_image:
response_data["highlighted_page_image"] = highlighted_page_image
if chunk_bbox:
response_data["chunk_bbox"] = chunk_bbox
return JSONResponse(response_data)