fix(vector): dead-letter terminally-failed documents to stop multi-user re-queue loop
A pathological PDF (a 206-page ChronoScan scan with ~3400 JBIG2/JPX images)
jammed a tenant's structured ingest worker in an infinite reprocess loop,
re-burning a 120s pymupdf4llm parse (and occasionally OOM-racing the 2Gi pod)
every few minutes.
Root cause: the per-user placeholder "failed" mark could not stop the loop. The
placeholder point ID is user-agnostic (uuid5("file:<doc_id>:placeholder")) but
the scanner's freshness gate, query, and status update all filter by user_id.
For a file visible to several users the single shared placeholder's user_id is
overwritten by whoever scanned last, so every other user's scan sees "no record"
and re-queues -- an N-user ping-pong that never honours the failed status.
Fix: when a parse fails terminally (no higher escalation tier available, e.g.
structured with OCR off) record a durable, content-addressed, user-agnostic
dead-letter marker (mirrors vector/sharing_state.py). The scanner consults it
tenant-wide for every user and skips re-queuing until the content (etag) OR the
escalation-tier set (tiers_sig -- e.g. OCR enabled) changes, so the document is
attempted once per content-version instead of forever.
- new vector/dead_letter.py: mark/is/clear, content-addressed marker carrying
is_placeholder=True (inherits search exclusion) + dead_letter=True
- escalation.escalation_tiers_signature(settings): retry-on-tier-change key
- processor: dead-letter terminal failures, clear on successful (re-)index
- scanner: user-agnostic is_dead_lettered skip beside claim_existing_index
- placeholder: exempt dead_letter markers from the orphan sweep (durability)
- metrics: astrolabe_document_dead_lettered_total{reason}
Deck #349.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
d969526613
commit
8c9339501e
@@ -110,6 +110,26 @@ async def test_keeps_own_pod_placeholders(monkeypatch):
|
||||
client.delete.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
async def test_keeps_dead_letter_markers(monkeypatch):
|
||||
"""Dead-letter markers (vector/dead_letter.py) reuse is_placeholder=True for
|
||||
the search exclusion but are DURABLE terminal-state records, not in-flight
|
||||
placeholders. They carry a foreign/absent instance_id, so without the
|
||||
carve-out the sweep would delete them on every Pod restart and the
|
||||
dead-lettered document would loop again. They MUST be kept."""
|
||||
monkeypatch.setattr(placeholder_module, "_INSTANCE_ID", "pod-new")
|
||||
marker = SimpleNamespace(
|
||||
id="dl-1",
|
||||
payload={"is_placeholder": True, "dead_letter": True, "instance_id": "pod-old"},
|
||||
)
|
||||
client = _make_client([([marker], None)])
|
||||
|
||||
swept, kept = await sweep_orphan_placeholders(client, "nextcloud_content")
|
||||
|
||||
assert (swept, kept) == (0, 1)
|
||||
client.delete.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
async def test_noop_when_no_placeholders_exist(monkeypatch):
|
||||
"""Cold-boot tenant with an empty collection — sweep does NOT
|
||||
|
||||
Reference in New Issue
Block a user