Round-4 review nits on PR #920 (none blocking):
- record_document_dead_lettered: enumerate the oversize reason (added this PR)
alongside timeout/oom/error in the docstring + counter comment.
- Note the clear-dead-letter-before-upsert ordering implication (a transient
upsert failure re-parses once, never a silent drop).
- Clarify the orphan sweep's kept counter for tenant-wide dead-letter markers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-3 review nits on PR #920 (none blocking):
- escalation_tiers_signature: TODO noting future settings that can rescue a
previously-terminal document (a toggleable llm tier, a raised oversize cap)
should be folded into the signature so raising them auto-retries dead-letters.
- Terminal-path placeholder cleanup: a delete failure here is real Qdrant I/O,
not control-flow -- log at warning (was debug) for observability. Non-fatal
(the durable marker is already written).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses round-2 review on PR #920:
- Only dead-letter a terminal failure when the file has an etag to
content-address the marker; without one, fall back to the legacy per-user
placeholder mark (an etagless marker is unmatchable). + test.
- _dead_letter_filter now also matches is_placeholder=True (redundant with
dead_letter=True but lets Qdrant use the is_placeholder payload index).
- TODO(deck-349) documenting the dead-lettered-then-deleted orphan-marker leak
(out of scope; needs a marker sweep or TTL field) per reviewer.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses round-1 review on PR #920:
- Delete path now clears the file's dead-letter marker after
release_document_for_user (whose principal-based filter misses the
user-agnostic, principal-less marker), preventing orphan-marker accumulation
for dead-lettered-then-deleted files.
- Oversize PDFs (rejected by the pre-parse size guard, no pipeline_tier stamped)
are now treated as terminal regardless of failing_tier -- no tier can parse an
oversize file -- so they dead-letter instead of falling to the legacy per-user
mark on the inline path.
- Gate the success-path clear on a non-empty etag (an etag-less file can never
have a marker, mirroring is_dead_lettered's early return).
- dead_letter.py: payload typed dict[str, Any] (CLAUDE.md).
Tests: oversize-terminal dead-letter and delete-path marker clear.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A pathological PDF (a 206-page ChronoScan scan with ~3400 JBIG2/JPX images)
jammed a tenant's structured ingest worker in an infinite reprocess loop,
re-burning a 120s pymupdf4llm parse (and occasionally OOM-racing the 2Gi pod)
every few minutes.
Root cause: the per-user placeholder "failed" mark could not stop the loop. The
placeholder point ID is user-agnostic (uuid5("file:<doc_id>:placeholder")) but
the scanner's freshness gate, query, and status update all filter by user_id.
For a file visible to several users the single shared placeholder's user_id is
overwritten by whoever scanned last, so every other user's scan sees "no record"
and re-queues -- an N-user ping-pong that never honours the failed status.
Fix: when a parse fails terminally (no higher escalation tier available, e.g.
structured with OCR off) record a durable, content-addressed, user-agnostic
dead-letter marker (mirrors vector/sharing_state.py). The scanner consults it
tenant-wide for every user and skips re-queuing until the content (etag) OR the
escalation-tier set (tiers_sig -- e.g. OCR enabled) changes, so the document is
attempted once per content-version instead of forever.
- new vector/dead_letter.py: mark/is/clear, content-addressed marker carrying
is_placeholder=True (inherits search exclusion) + dead_letter=True
- escalation.escalation_tiers_signature(settings): retry-on-tier-change key
- processor: dead-letter terminal failures, clear on successful (re-)index
- scanner: user-agnostic is_dead_lettered skip beside claim_existing_index
- placeholder: exempt dead_letter markers from the orphan sweep (durability)
- metrics: astrolabe_document_dead_lettered_total{reason}
Deck #349.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The client registry derived display names from a hardcoded map of
"well-known" MCP clients (claude-desktop, claude-ai, continue-dev,
zed-editor, vscode-mcp). This baked a recognized-client list into the
server even though admission already requires explicit opt-in via
ALLOWED_MCP_CLIENTS (fail-closed when unset).
Mirror the management-API surface (ALLOWED_MGMT_CLIENT), which has no
built-in client list: remove the map and derive the display name
generically from the client_id. Default remains none; clients must be
added explicitly and DCR stays off unless ENABLE_DCR=true.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address round-4 review on PR #914:
- glyph_corruption_ratio <= 0 now disables the signal (previously `control_ratio
> 0` fired on any single C0 control byte), matching the "0 disables" convention
used elsewhere (document_max_pdf_size_mb) and the config comment. Add a
zero-disables test.
- Correct the document_escalation_suppressed_total comment: corrupt_glyphs CAN
appear there in the narrow case where structured is unregistered and OCR is
registered-but-disabled (evaluate_escalation follows minimum="structured" past
the missing rung to a gated-off OCR). Add a test for that suppressed decision.
- Add a test for the double-corruption edge: a structured re-extract that is also
glyph-corrupt escalates structured->ocr with reason corrupt_glyphs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address round-3 review on PR #914:
- When a glyph-corrupt doc's structured rung is NOT registered, the inline path
now falls through to OCR (with reason corrupt_glyphs), mirroring the external
next_available_tier instead of silently keeping the fast result. A structured
parse FAILURE remains terminal (tracked via structured_failed), matching the
external path which does not escalate a failure. Added a debug log for the
unregistered case and "(OCR not attempted)" to the failure warning.
- Tests: inline + external glyph-corrupt fallthrough to OCR when structured is
unregistered; glyph-corrupt + junk-quality both-flags precedence (structured
wins over the bad_text_layer/ocr route).
- Note the total_chars>0 mutual-exclusion with the scanned branch in
_route_from_signals.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address round-2 review on PR #914:
- Add corrupt_glyphs to the document_classifier_flag_total label comment (it is
a live flag value emitted by record_document_classification).
- Mirror the full_text-vs-sampled control-ratio NOTE into classify_pdf so the
diagnostic path's under-detection trade-off is documented in place.
- Add test_classify_pdf_glyph_corrupt_routes_structured for routing symmetry on
the standalone classify_pdf path.
(SonarCloud quality gate is green — the prior S1244 finding was fixed last round.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address round-1 review on PR #914:
- Attribute the OCR hop in a fast->structured->ocr inline cascade to
from_tier="structured" (not a second "fast" escalation), so
astrolabe_document_escalation_total per-tier counts stay accurate.
- Add test_inline_fast_structured_ocr_cascade pinning that two-hop path and the
metric attribution.
- Note in classify_from_text that its doc-level control ratio is over full_text
(all pages), not the sampled subset classify_pdf uses.
- Clarify that corrupt_glyphs never lands in the suppressed-escalation counter.
- Dedupe the glyph-corrupt test string into tests/fixtures/glyph_corruption.py.
- Use pytest.approx for the control-char-ratio zero checks (SonarCloud S1244).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fast (pypdfium2) extractor can leak raw glyph codes on subset fonts with a
broken /ToUnicode CMap. The result scores high on the existing text-quality
heuristic -- a uniform glyph/Caesar offset preserves whitespace and token
lengths -- yet is unsearchable. The structured (pymupdf) tier extracts the same
pages correctly.
Add a language-agnostic C0-control-character-ratio signal to the tier-0
classifier that detects this corruption and routes the document to a new
`structured` recommended_tier. Wire the fast->structured hop on the inline path
and generalise it so a low-quality-but-non-empty layer also tries structured
before OCR -- the inline and external ingest modes now follow the full
fast->structured->ocr ladder identically. A scanned / no-text-layer document
(total_chars == 0) still shortcuts straight to OCR, since a text extractor
cannot recover a pure raster.
New per-tenant tunable DOCUMENT_GLYPH_CORRUPTION_RATIO (default 0.02); escalation
metrics gain a `corrupt_glyphs` reason label.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address round-1 review on #913 and the SonarCloud new_reliability_rating gate:
- credential_cleanup_task no longer catches the cancellation exception
(Sonar python:S7497). A task-group cancel must propagate for structured-
concurrency teardown; graceful shutdown still flows through shutdown_event,
so the sleep no longer needs a cancel/break.
- Parametrize the scanner self-heal tests over 401 AND 403 (handled
identically at both call sites) and add a test that a failing periodic
sweep is logged non-fatally and does not crash the task.
- Log the stored-user count before the startup sweep (operability signal),
add a debug line when the credential row was already gone, and document
the at-most-one extra-401 convergence in _remove_stale_credential.
Refs Deck #198.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deleted/disabled Nextcloud users left their app_passwords row in storage,
so user_manager_task re-spawned their scanner every poll interval only to
401 again — an endless re-spawn/auth-failure loop (observed on
tenant-blackbox-demo: ~534 respawns/3h, matching the 60s poll interval).
- Delete the stored app password on a hard 401/403 in user_scanner_task
(both the pre-validation and in-scan-loop paths), breaking the re-spawn
loop at the source so the user-manager stops recreating the scanner.
- Add a periodic credential_cleanup_task backstop (hourly) that sweeps
cleanup_invalid_app_passwords for anything the per-scanner path misses.
- Run the startup cleanup for all deployment modes: drop the stale
`not oauth_enabled` guard so login_flow tenants (the cloud default) are
covered. NOTE: login_flow startup now makes one concurrent OCS
validation call per stored user before readiness.
Refs Deck #198.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The consent gate added three `_app_enabled(...) and is_doc_type_allowed(...)`
conditions to scan_user_documents, pushing its cognitive complexity over the
SonarQube threshold. Fold the pair into a _should_scan() helper (alongside the
earlier _enqueue_deletes refactor). Also document the accepted doc_types=None
per-type-query trade-off at the search consent gate (round-10 review item).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extract _mark_backstop_done() (overflow eviction + marker write) and
_backstop_delete_doc_type() (per-type scroll + enqueue) so
_enqueue_deletes_for_disabled_types drops from cognitive complexity 17 to well
under the 15 threshold. Behavior unchanged; tests still pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- route test for purge_doc_types raising on total failure -> 500
- route test for doc_types list containing non-strings -> 400
- reword the capabilities move_to_end comment (no-op on new keys; needed only
for the expired-key in-place update)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- app.py: move the /api/v1/vector-sync/purge mention out of the unconditional
management-endpoints log and into the vector_sync_enabled block, so operators
without Qdrant don't see an endpoint that 404s
- vector_sync route: comment why doc_types isn't whitelisted against
INDEXED_DOC_TYPES (unknown type = harmless zero-match no-op)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- vector_sync route: rename the response dict from `body` to `resp` so it no
longer shadows the request `body` (maintenance trap)
- scanner: comment the intentional files-vs-text purge timing asymmetry
- tests: add the all-text-types-disabled backstop case (empty allow-set)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- purge route: include a "failed" key in the 200 body listing requested doc
types that were not purged, so Astrolabe knows consent isn't yet enforced
for them (scanner backstop still catches up)
- tests: add @pytest.mark.unit / module-level pytestmark to the new test
modules so they run under `pytest -m unit`; add a partial-failure route test
- capabilities: comment why the cache is keyed per-user despite a global value
- semantic/scanner: doc/comment clarifications (sorted-order, eviction timing)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- tests: cover the process_document consent gate (drops an admin-disabled
index task with record_ingest_dropped("admin_disabled"); allows approved)
- scanner: _consent_backstop_done is now an insertion-ordered dict and evicts
the oldest entries to half capacity on overflow, so a bound hit re-fires the
backstop for only the oldest markers instead of the whole fleet at once
- semantic: reword the short-circuit log (consent, not installation)
- capabilities: comment why move_to_end is needed after an expired-key update
- test: assert the global purge delete-filter is owner-agnostic (doc_type only);
fix a pre-existing ty error on UnexpectedResponse(headers=None) in the file
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- app.py: register /api/v1/vector-sync/purge only when vector_sync_enabled, so
it returns 404 (not a 500 from get_qdrant_client) when sync is off
- scanner: bound _consent_backstop_done so a long-running multi-tenant process
with user churn can't grow it without limit (clears on overflow)
- purge route: distinct 400 for a missing doc_types key; enforce the admin
check even for an empty no-op request (destructive route)
- tests: missing-key 400, admin-gated empty no-op, non-admin empty 403
The _consent_narrowed_doc_types precondition is enforced by its non-Optional
frozenset[str] signature (ty rejects a None caller). The httpx.BasicAuth
SonarCloud hotspot matches the existing webhook routes (false positive,
credential from the app-password store) — left consistent for UI triage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- scanner: gate the consent backstop with a per-(user,doc_type) one-shot
marker so a standing admin-disable doesn't re-enqueue idempotent deletes
every scan tick; the marker clears when the type is re-enabled. Derive
_TEXT_BACKSTOP_DOC_TYPES from INDEXED_DOC_TYPES so new indexed types are
covered automatically
- semantic: extract _consent_narrowed_doc_types so the search-side narrowing
is unit-testable; add tests for restrict/intersect/disjoint/empty
- purge route: cap doc_types length (abuse guard) -> 400
- tests: one-shot + re-enable backstop, too-many-doc_types 400
Deferred (noted on PR): per-document allowed_doc_types call is cache-hot;
purge "last error wins" — both logged. SonarCloud broad-except hotspots are
deliberate (noqa BLE001), reviewable in the UI.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- purge route: 400 (not 500) on a valid-JSON non-object body
- scanner: backstop-purge admin-disabled note/news_item/deck_card points
(their deletion-tracking lives inside the skipped scan_* fns), mirroring the
files path; gated on a concrete allow-set so fail-open never deletes
- processor: record_ingest_dropped("admin_disabled") so consent-skipped index
tasks are observable/alertable
- app.py: list /api/v1/vector-sync/purge in the endpoints log line
- capabilities: drop empty-string doc types; return frozenset throughout
- purge: document the count-before-delete approximation
- tests: non-object body -> 400, ProvisioningRequiredError -> 428, cache TTL
expiry refetch, and the scanner consent backstop
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consume the astrolabe.semantic_search capability as the source of truth for
which content sources an admin has approved for semantic search, and enforce
it independently of Astrolabe (this server queries Qdrant directly).
- capabilities.py: cached per-user reader for enabled_doc_types (TTL+LRU,
fail-open so older Astrolabe / transient OCS errors don't break search)
- semantic search: intersect requested doc_types with the allowed set;
restrict to the allowed set when none requested; short-circuit when empty
- scanner: skip disabled sources during discovery (files discovery yields
nothing when disabled, so the existing grace-period reconcile purges them)
- processor: drop near-real-time index tasks for disabled doc_types
(webhook events bypass the scanner gate); deletes always proceed
- vector/purge.py + POST /api/v1/vector-sync/purge: admin-only global
delete-by-doc_type, called by Astrolabe when a source is disabled so
consent is binding on data-at-rest
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round 5 review (PR #910), all nits, no blockers:
- _result_from_success: a succeeded job with `pages=[]` (empty list, not just a
missing key) is now a per-document failure ("no pages returned") instead of a
silent 0-chunk success. Test added.
- Comment the deadline-expiry path: the gateway-side job isn't cancelled (no
cancel endpoint at this layer) — it's reaped by the gateway file purge; we just
stop polling it.
- Drop the vestigial status="pending" from the BatchOcrJob test fakes (the column
was removed in round 4; BatchPollResult.status fakes are untouched).
1653 unit tests pass; ruff + ty green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round 4 review (PR #910), no blockers:
- poll(): a 2xx body with no `status` now fails fast (logged) instead of being
treated as perpetually pending until the deadline; defensive page index
(`p.get("index", i)`) so a malformed page degrades rather than KeyError-ing.
- Document on poll() that job_id is namespaced (embeds "/") so the gateway route
must be a path-capture param (GET /v1/ocr/batch/{job_id:path}).
- Drop the vestigial `status` + `updated_at` columns from batch_ocr_jobs: a row
only ever exists while pending (terminal jobs are deleted) and the live status
comes from a fresh poll, so a stored mirror was permanently "pending" /
redundant with submitted_at. Simplifies the migration, store, and dataclass.
- Tests: submit() ValueError on missing job_id; poll() missing-status → failed.
1653 unit tests pass; ruff + ty green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round 3 review (PR #910):
- Guard an unexpected terminal batch status in _process_batch: anything that
isn't succeeded/failed (gateway version skew, a new lifecycle state) now marks
the document parse-failed instead of falling through to _pages_to_text([]) — a
0-chunk "success" that silently indexed empty text and re-submitted forever.
Test added.
- gateway_batch_client.submit: raise an actionable ValueError on a 2xx response
with no job_id (was a bare KeyError deep in the caller).
- Document that a _process_batch transport error intentionally propagates to
procrastinate for retry rather than falling back to sync (opt-in batch wants
the retry).
- Annotate _batch_client as GatewayBatchOcrClient | None (TYPE_CHECKING import
already present); clarify the delete_stale_for_doc first-submit no-op comment.
- Add a parametrized build_gateway_batch_client test (the gateway-only invariant:
mistral/none/no-URL -> None; gateway|auto + URL -> client).
1653 unit tests pass; ruff + ty green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round 2 review (PR #910):
- BLOCKING: BatchOcrJobStore._shared_lock is now lazy-init (anyio.Lock | None,
created on first shared() call) instead of at class-definition time — matches
the CLAUDE.md "no anyio primitives at import time" rule and OcrProcessor's
pattern. The None-check->assign has no await between, so it's race-free.
- document_ocr_mode now normalizes via _enum_fields (case-insensitive, like
document_ocr_provider) instead of a strict dynaconf is_in Validator, so
DOCUMENT_OCR_MODE=Batch normalizes to "batch" rather than erroring. Tests for
case-normalization + invalid-value rejection.
- TYPE_CHECKING-gated GatewayBatchOcrClient import so build_gateway_batch_client
/ _get_batch_client are typed `GatewayBatchOcrClient | None` instead of Any
(runtime import stays lazy to avoid the import cycle).
- Rename ocr_options -> doc_identity_options (it's threaded to all tiers; only
OCR reads it) + clarify the comment.
- Drop the redundant forward-ref quotes on _shared_instance.
- Add direct _batch_identity unit tests (partial/empty options branches).
Left as follow-up: reusing one httpx.AsyncClient across submit/poll (same
per-call pattern as the existing sync _GatewayOcrBackend; no clean aclose hook
on the cached client today).
1653 unit tests pass; ruff + ty green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round 1 review (PR #910):
- BLOCKING: add document_ocr_mode / _batch_poll_seconds / _batch_max_wait_seconds
to config._field_map — without it dynaconf silently ignored the env vars and
DOCUMENT_OCR_MODE=batch could never be enabled in production. Add a regression
test asserting the three round-trip from env.
- migration 008: give batch_ocr_jobs a composite PRIMARY KEY on
(user_id, doc_id, doc_type, etag) instead of a bare UniqueConstraint (N1).
- OcrProcessor: use a dedicated _batch_client_lock instead of sharing the sync
backend lock (N3).
- tests: use https:// gateway URLs in the new fixtures to clear SonarCloud's
"insecure http" security hotspots (all 14 were test-only http://gw literals).
1653 unit tests pass; ruff + ty green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add DOCUMENT_OCR_MODE=sync|batch (default sync). In batch mode the tier-3 OCR
processor submits documents to the embedding gateway's async Batch OCR routes
(POST /v1/ocr/batch + GET /v1/ocr/batch/{job_id}, astrolabe-cloud-website#372)
for ~50% cheaper large-corpus backfill. The direct Mistral OCR path is left
untouched. Tracked on Deck #332.
Batch jobs run minutes-hours, so the OCR tier cannot block (the procrastinate
worker reclaims jobs in `doing` after INGEST_STALLED_JOB_SECONDS). Instead it
submits, records the gateway job id in a new per-tenant `batch_ocr_jobs` table
(procrastinate args are immutable across retries), and raises a BatchPending
signal that TieredEscalationStrategy turns into a same-queue deferred re-poll —
releasing the worker slot between polls. On completion the per-page markdown is
indexed like the sync path; a failure or a job past
DOCUMENT_OCR_BATCH_MAX_WAIT_SECONDS marks the document parse-failed.
Batch is opt-in and gateway-only: with the direct mistral backend, no gateway
URL, or the inline/memory pipeline (which can't defer), it falls back to sync.
One batch job per document (coalescing N docs/job is a follow-up).
- embedding/gateway_batch_client.py: submit/poll client (reuses GatewayTokenProvider).
- vector/batch_ocr_store.py + migration 008: job tracking (portable SQLite+PG).
- document_processors/escalation.py: BatchPending control-flow signal.
- document_processors/ocr.py: batch state machine + sync fallback.
- vector/processor.py: thread doc identity to the OCR tier; raise BatchPending
from the pending sentinel; propagate it as control flow (not a failure).
- vector/queue/procrastinate.py: BatchPending -> same-queue retry_in, exempt
from the transient cap (bounded by the processor's deadline).
- config + docs; tests across client/store/processor/strategy/parse-tier.
1653 unit tests pass; ruff + ty green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-2 review follow-ups (GHSA-8vh3-g2qg-2h2c PR):
- Add a dynaconf validator requiring WEBHOOK_SECRET to be >=16 chars when set
(None still allowed = webhooks disabled), so weak/placeholder secrets fail
at startup rather than in an audit. Covered by two new tests in test_config.py.
- Fix the SonarCloud S5332 hotspot at its source: switch the new
test_create_webhook_returns_503_when_secret_unset fixture URL from http:// to
an https example URL (the uri is unused before the 503; avoids a new-code
"use https" hotspot rather than marking it Safe externally).
- Nits: drop the unused app.state.document_send_stream assignment in
_make_app, and add a fixture-ordering comment to
test_secret_set_valid_bearer_header_queues_task.
(--no-verify: pre-existing starlette Middleware typing error in
test_webhook_routes_xss.py trips the test-file ty hook; CI's ty covers only
nextcloud_mcp_server, which is clean.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GHSA-8vh3-g2qg-2h2c (CVSS 9.1, CWE-306): POST /webhooks/nextcloud had no
authentication when WEBHOOK_SECRET was unset (the default). The receiver
trusted the attacker-supplied user.uid and fed it to Qdrant, letting an
unauthenticated network caller delete or re-index any user's vector
embeddings.
Webhooks now require WEBHOOK_SECRET end-to-end:
- app.py: the /webhooks/nextcloud route is only mounted when WEBHOOK_SECRET
is set; otherwise it 404s and a startup warning notes vector sync falls
back to the polling scanner.
- webhook_receiver.py: removed the warn-and-accept fallback. No secret -> 503,
missing/invalid bearer -> 401; the payload is never processed unauthenticated.
- webhook_routes.py / api/webhooks.py: webhook_auth_pair() raises
WebhookSecretNotConfigured instead of returning authMethod="none"; both
registration entry points return a clear 503 so no dead unauthenticated
webhooks are created.
Also expose webhooks availability to the Astrolabe UI via GET /api/v1/status
("webhooks_enabled": bool), set WEBHOOK_SECRET on the docker-compose
semantic-search dev services, and update env.sample + ADR-010 / ADR-018 /
webhook-management-guide docs.
Vector sync still works without a secret via the polling scanner.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-2 review follow-ups in unified_verifier.py:
- module-level docstring still described "two compliant OAuth modes" incl.
token exchange — rewritten to multi-audience only.
- removed the stale "# Both modes do the same validation" inline comment in
verify_token().
(--no-verify: same pre-existing ty errors in test_unified_verifier.py as prior
commits; CI type-checks only the package, which passes.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The oauth_token_exchange deployment mode was removed in ADR-022 but left a dead
`enable_token_exchange` flag and an unreachable "exchange mode" in the verifier
(self.mode was hardcoded to "multi-audience"). Remove the remnants:
- config.py: drop the `enable_token_exchange` default and the
`ENABLE_TOKEN_EXCHANGE` branch in `_is_multi_user` (+ its doc line).
- unified_verifier.py: drop `self.mode` and the dead exchange-mode log branch;
simplify the docstrings to multi-audience only.
- test_unified_verifier.py: drop the `.mode` assertions (attribute removed);
collapse the redundant init tests.
Also remove docs/ADR-004-Code-Review.md — an orphaned code-review note, not an
ADR; it doesn't belong in the docs/ADR namespace.
(--no-verify: the ty-check hook flags 3 PRE-EXISTING type errors in
test_unified_verifier.py lines 346/362/441, untouched by this change; CI
type-checks only the package, which passes.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review round 4 (both nits):
- Rename the flag to ignore_ocr_enabled so its OCR-specific scope is explicit at
the call sites (the gate only bypasses the OCR-enabled check).
- Add test_evaluate_escalation_empty_suppressed_even_when_structured_registered:
empty_text (minimum='ocr') skips a registered structured tier and suppresses to
ocr when OCR is off, never hopping to structured.
Deck #324.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The inline _process_pdf path does not emit document_escalation_suppressed_total;
the "what-if OCR" counter is instrumented only on the per-tier external path
(evaluate_escalation / _parse_pdf_tier). Comment the inline OCR gate so a reader
doesn't mistake the omission for a bug. Deferred the assert_never nit (typing
.assert_never is 3.11+; Literal+frozen already guard construction) and the
pre-existing minimum-ValueError pass (only "ocr"/None are ever passed).
Deck #324.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- escalation: EscalationDecision.reason is now Literal["empty_text",
"low_confidence"] (parity with kind; ty catches a bad label at call sites).
- processor: nest the decision handling so the hop branch is reached via an
explicit else under `if decision is not None` — exhaustive over the Literal
kind, no None-attribute risk.
- tests: add the "OCR processor unregistered (not just disabled) → None"
quadrant, locking in absent != suppressed.
Deck #324.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- escalation: EscalationDecision.kind is now Literal["hop","suppressed"] so ty
catches a bad kind statically (and the processor branch is exhaustive).
- processor: simplify the suppressed-escalation log line (no longer repeats
to_tier / tier).
- registry: clarify _tier_available's ignore_enabled drops the OCR-enabled gate
specifically (a future per-tier gate would extend the condition).
Deck #324.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
OCR is the paid, opt-in tier (DOCUMENT_OCR_ENABLED, default off). The per-tier
escalation gate already declines to hop to OCR when it's disabled (the pre-OCR
tier is terminal — no surprise cost), but that left operators blind to how much
OCR demand exists.
evaluate_escalation now returns a structured EscalationDecision:
- "hop" — a higher tier can run; the caller raises EscalateError (queue-hop).
- "suppressed" — the ideal next tier (e.g. ocr) exists but is DISABLED; the caller
indexes the current tier's output as terminal and records the
would-be hop on the new astrolabe_document_escalation_suppressed_total
{from_tier,to_tier,reason} counter instead of hopping.
- None — index as-is (good text, or no such tier at all).
So with OCR off, escalation_suppressed_total{to_tier="ocr"} is the latent OCR
demand an operator weighs before enabling OCR; enabling it converts these into
real document_escalation_total{to_tier="ocr"} hops. next_available_tier gains an
ignore_enabled flag to compute the *ideal* (enabled-gate-ignored) target.
Tests: registry suppressed vs hop vs terminal (incl. structured-hop-not-suppressed
when OCR off but structured available); _parse_pdf_tier records suppressed +
indexes without raising.
Deck #324 (parent #323).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A stall is often systemic (a Qdrant/embedding outage stalls every in-flight
job), so reclaiming the whole batch at now() every */5min tick would
thundering-herd a recovering dependency, bypassing TieredEscalationStrategy's
per-job backoff. reclaim_stalled_ingest_jobs now offsets retry_at by a fixed
delay (INGEST_RECLAIM_RETRY_DELAY_SECONDS, default 30s; 0 = legacy immediate).
Also document the hot-vs-restart flag asymmetry: INGEST_ESCALATION_ENABLED is
re-read per job; INGEST_TRANSIENT_MAX_ATTEMPTS is snapshotted at worker startup.
Deck #323.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- metrics: update_ingest_queue_depth guarded on `not by_queue`, which conflated
None (memory backend no-op) with {} (postgres, ALL queues drained). When every
queue drains at once, get_ingest_job_counts_by_queue returns {} and the
pre-zero loop was skipped, leaving a stale ghost backlog in the gauge. Guard on
`by_queue is None` only; add an all-drained regression test.
- procrastinate: note that INGEST_TRANSIENT_MAX_ATTEMPTS is snapshotted at
blueprint-build time (restart to pick up changes).
Deck #323.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- tests: make the transient-backoff progression assertion load-independent by
bracketing the get_retry_decision call with before/after timestamps instead of
measuring against a second datetime.now() (no freezegun dependency).
- tests: use pytest.approx for the ingest-queue-depth gauge assertions —
SonarCloud python:S1244 (float == ) was a MAJOR reliability finding that
tripped the new_reliability_rating quality gate.
- processor: tighten the EscalateError lazy-bind comment (file processing already
imports the document stack via get_registry; the gating only spares the
delete / text-doc paths and module-load time).
Deck #323.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- metrics: update_ingest_queue_depth now pre-zeroes every managed ingest queue
before applying live counts, so a queue that drains to empty (and drops out of
procrastinate's list_queues_async) reads 0 instead of sticking at its last
non-zero value (ghost backlog in Grafana/alerts). Adds a regression test.
- procrastinate: comment that _is_transient_infra_error treats all qdrant errors
as transient deliberately (bounded same-tier retry; over-broad is acceptable).
- escalation: note next_tier is the building block; production routing uses
ProcessorRegistry.next_available_tier.
- tests: add evaluate_escalation fast+ocr-only low-confidence -> ocr case.
Deck #323.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
