- escalation: EscalationDecision.reason is now Literal["empty_text",
"low_confidence"] (parity with kind; ty catches a bad label at call sites).
- processor: nest the decision handling so the hop branch is reached via an
explicit else under `if decision is not None` — exhaustive over the Literal
kind, no None-attribute risk.
- tests: add the "OCR processor unregistered (not just disabled) → None"
quadrant, locking in absent != suppressed.
Deck #324.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- escalation: EscalationDecision.kind is now Literal["hop","suppressed"] so ty
catches a bad kind statically (and the processor branch is exhaustive).
- processor: simplify the suppressed-escalation log line (no longer repeats
to_tier / tier).
- registry: clarify _tier_available's ignore_enabled drops the OCR-enabled gate
specifically (a future per-tier gate would extend the condition).
Deck #324.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
OCR is the paid, opt-in tier (DOCUMENT_OCR_ENABLED, default off). The per-tier
escalation gate already declines to hop to OCR when it's disabled (the pre-OCR
tier is terminal — no surprise cost), but that left operators blind to how much
OCR demand exists.
evaluate_escalation now returns a structured EscalationDecision:
- "hop" — a higher tier can run; the caller raises EscalateError (queue-hop).
- "suppressed" — the ideal next tier (e.g. ocr) exists but is DISABLED; the caller
indexes the current tier's output as terminal and records the
would-be hop on the new astrolabe_document_escalation_suppressed_total
{from_tier,to_tier,reason} counter instead of hopping.
- None — index as-is (good text, or no such tier at all).
So with OCR off, escalation_suppressed_total{to_tier="ocr"} is the latent OCR
demand an operator weighs before enabling OCR; enabling it converts these into
real document_escalation_total{to_tier="ocr"} hops. next_available_tier gains an
ignore_enabled flag to compute the *ideal* (enabled-gate-ignored) target.
Tests: registry suppressed vs hop vs terminal (incl. structured-hop-not-suppressed
when OCR off but structured available); _parse_pdf_tier records suppressed +
indexes without raising.
Deck #324 (parent #323).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A stall is often systemic (a Qdrant/embedding outage stalls every in-flight
job), so reclaiming the whole batch at now() every */5min tick would
thundering-herd a recovering dependency, bypassing TieredEscalationStrategy's
per-job backoff. reclaim_stalled_ingest_jobs now offsets retry_at by a fixed
delay (INGEST_RECLAIM_RETRY_DELAY_SECONDS, default 30s; 0 = legacy immediate).
Also document the hot-vs-restart flag asymmetry: INGEST_ESCALATION_ENABLED is
re-read per job; INGEST_TRANSIENT_MAX_ATTEMPTS is snapshotted at worker startup.
Deck #323.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- metrics: update_ingest_queue_depth guarded on `not by_queue`, which conflated
None (memory backend no-op) with {} (postgres, ALL queues drained). When every
queue drains at once, get_ingest_job_counts_by_queue returns {} and the
pre-zero loop was skipped, leaving a stale ghost backlog in the gauge. Guard on
`by_queue is None` only; add an all-drained regression test.
- procrastinate: note that INGEST_TRANSIENT_MAX_ATTEMPTS is snapshotted at
blueprint-build time (restart to pick up changes).
Deck #323.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- tests: make the transient-backoff progression assertion load-independent by
bracketing the get_retry_decision call with before/after timestamps instead of
measuring against a second datetime.now() (no freezegun dependency).
- tests: use pytest.approx for the ingest-queue-depth gauge assertions —
SonarCloud python:S1244 (float == ) was a MAJOR reliability finding that
tripped the new_reliability_rating quality gate.
- processor: tighten the EscalateError lazy-bind comment (file processing already
imports the document stack via get_registry; the gating only spares the
delete / text-doc paths and module-load time).
Deck #323.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- metrics: update_ingest_queue_depth now pre-zeroes every managed ingest queue
before applying live counts, so a queue that drains to empty (and drops out of
procrastinate's list_queues_async) reads 0 instead of sticking at its last
non-zero value (ghost backlog in Grafana/alerts). Adds a regression test.
- procrastinate: comment that _is_transient_infra_error treats all qdrant errors
as transient deliberately (bounded same-tier retry; over-broad is acceptable).
- escalation: note next_tier is the building block; production routing uses
ProcessorRegistry.next_available_tier.
- tests: add evaluate_escalation fast+ocr-only low-confidence -> ocr case.
Deck #323.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Register the periodic stalled-job reclaim on a dedicated ingest-maintenance
queue that every worker drains (any --tier), so reclaim still fires when the
fast fleet is scaled to zero and only ocr workers run. procrastinate's
periodic-defer dedup keeps it single-run across drainers.
- escalation: mark `unsupported`/`forced` reason labels as reserved (not raised).
- processor: note that options/progress_callback are intentionally not threaded
through _parse_pdf_tier yet (symmetric with the inline path).
- tests: assert TieredEscalationStrategy backoff progression (4/8/16/…/300s);
cover get_ingest_pending per-queue aggregation + the legacy job_counts
fallback; add an external-path zero-page no-escalation case; use the canonical
INGEST_QUEUE_FAST instead of the back-compat alias.
Deck #323.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Split external (procrastinate) document processing into per-tier queues so a
document is attempted at most once per tier and requeued to the next tier's
queue on a low-quality parse, using procrastinate's native retry.
- escalation.py: TIER_LADDER (fast->structured->ocr) + EscalateError signal
- registry: process_tier (one tier) + evaluate_escalation post-parse gate
(reuses classify_from_text) + next_available_tier; shared _classify_result
and _oversize_result with the inline pipeline
- processor: process_document(tier=...) runs one tier and raises EscalateError
before embed (junk text never indexed); inline memory path unchanged
- queue/procrastinate: ingest-fast|structured|ocr queues; TieredEscalationStrategy
(queue-hop on EscalateError, bounded same-tier transient retry); queue-aware
task; producer defers to ingest-fast; per-queue counts + all-queue reclaim
- cli: worker --tier {fast,structured,ocr}
- billing: pages_ocr usage event + pipeline_tier metadata (paid OCR billed apart)
- observability: astrolabe_ingest_queue_depth{queue,status} gauge + per-queue
counts in nc_get_vector_sync_status / management status endpoint
- config: INGEST_ESCALATION_ENABLED (default true), INGEST_TRANSIENT_MAX_ATTEMPTS
INGEST_ESCALATION_ENABLED=false and INGEST_QUEUE=memory preserve prior behaviour.
Deck #323.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-1 review nits. The constant now only gates the diagnostic `image_heavy`
flag (not routing), so the old name was misleading. Rename + reword its comment
to state the diagnostic-only intent. Also add a classify_pdf symmetry test
(`test_classify_pdf_image_heavy_clean_text_stays_fast`) pinning that a full-page
raster image with a clean text layer routes fast on the classify_pdf path too.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The tier-0 classifier escalated any page with raster-image coverage >=0.80 to
the OCR tier regardless of its text layer. On OHR-Bench this drove ~45% of all
OCR escalations: clean born-digital pages dominated by a figure, and scanned
pages that already carry a usable OCR text layer -- re-OCR adds nothing for
either, but each one was routed to the paid tier-3 OCR.
Route on the text signals only (near-empty or junk-quality layer). Image
coverage is still computed and still raises the `image_heavy` diagnostic flag,
but no longer routes. True scans with no/garbage text continue to escalate via
the empty-text and quality signals, so genuine OCR needs are unaffected.
Trade-off: image-only content on an otherwise-clean page (handwriting, stamps,
text inside figures) is no longer force-routed to OCR. This was previously
intentional; the escalation cost outweighed the benefit for RAG indexing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add test_check_status_completion_wakes_user_manager: register the auth tools
against a stub MCP, drive nc_auth_check_status through a completed Login Flow
(mocked storage + poll), and assert it stores the app password and rings the
background-sync doorbell. The MCP-tool wake path was previously only verified
by inspection (round-2 review nit); all three notify_user_provisioned() call
sites now have dedicated coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Quality-gate fixes (new-code conditions on PR #902):
- new_security_hotspots_reviewed: drop the fake "http://nextcloud" host in the
manager tests to https:// (python:S5332 ×2).
- new_security_rating: generate the integration test's fake app password with
secrets.token_urlsafe instead of a hardcoded literal (python:S2068).
- new_reliability_rating: restructure the user_manager sleep so an explicit
await checkpoint lives inside the cancellation scope — await one waiter
directly while watching shutdown via start_soon (python:S7490). Behaviour is
unchanged: timeout, shutdown, or a provisioning ring all end the sleep.
Review nits:
- Move the shutdown test's fail_after(2) to wrap the whole task group so it
actually bounds the task-group exit (was guarding a no-op sleep); drop the
sleep(0) stub (python:S7491).
- Type _wake_on's wait_fn as Callable[[], Awaitable[object]].
- Note in _wire_vector_sync_state why provision_signal is set on the singleton
only, not fanned out to app.state.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- ProvisionSignal.wait() re-arms in a finally so a cancelled wait (shutdown
racing the doorbell) leaves a fresh unset event, not a stale set-but-consumed
one; preserves the no-await-before-swap lost-wakeup guarantee.
- Hoist user_manager_task's _wake_on helper out of the while loop (one object,
not one per iteration).
- Test: assert ProvisionSignal via its public wait() contract instead of the
private _event attribute.
- Add test_provision_app_password_wakes_user_manager covering the
api/passwords.py wake path (previously only LFv2 web + MCP tool were tested).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Background vector sync discovered newly provisioned users only on the
periodic user-manager poll (VECTOR_SYNC_USER_POLL_INTERVAL, default 60s),
delaying first indexing by up to a minute. Add a ProvisionSignal doorbell
that provisioning paths ring after storing a user's app password, waking
user_manager_task to re-poll and spawn the user's scanner at once. The
periodic poll remains the backstop (covers cross-replica provisioning).
- ProvisionSignal (stable reference, wait-and-re-arm) held on
VectorSyncState; closes the lost-wakeup window (no await between observing
the ring and re-arming; anyio.Event stickiness covers a mid-poll ring)
- user_manager_task races its poll timeout against the doorbell + shutdown
- notify_user_provisioned() rung from the three app-password provisioning
sites: Login Flow v2 web, MCP provisioning tool, management/BasicAuth API
Note: the pre-existing scanner_wake_event was never .set() and only wakes
existing scanners; a brand-new user has none, so the manager is what must
be nudged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address round-1 claude-review nits on PR #894:
- Document why offline_access is advertised unconditionally (independent of
settings.enable_offline_access): per RFC 8414, scopes_supported lists what
the AS *can* support, with actual issuance still gated upstream by Nextcloud.
- Add a regression test proving the offline_access invariant holds on an empty
FastMCP instance with no registered tools.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
discover_all_scopes() builds the scopes_supported lists exposed at
/.well-known/oauth-protected-resource and /.well-known/oauth-authorization-server.
It previously emitted only the base OIDC scopes plus tool-derived
@require_scopes, so offline_access was never advertised and
discovery-driven MCP clients had no way to know they could request a
refresh token.
Add offline_access unconditionally. The AS proxy already forwards
client-requested scopes upstream to Nextcloud, which issues a refresh
token when the MCP server's OIDC client is permitted the scope. This
only changes what is advertised; it is not added to ALL_SUPPORTED_SCOPES
(the app-level permission set), since offline_access is an OIDC behavior
rather than a Nextcloud resource permission.
Add a regression test asserting offline_access is always present in
discover_all_scopes() output.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-4 review on PR #893 (no blockers, minor items):
- Document why Mistral's _is_transient is SDK-level only (429/5xx): a bare
connection drop the SDK surfaces as httpx/ConnectionError isn't an SDKError
and isn't retried here by design — the pod-rollover target is the gateway
(OpenAI-compatible) path, which does cover connection errors.
- Include the last error (%r) in the retry helper's "not resolved after N
attempts" error log.
- Add test_mistral_embed_batch_retries_on_5xx (batch path parity with embed()).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-3 review on PR #891 (no blockers):
- Add test_encode_dav_path_encodes_exactly_once pinning the documented
decoded-input precondition ("already%20encoded.pdf" -> "already%2520...").
- format_exception_group: proper singular/plural ("1 sub-exception" vs
"N sub-exceptions") instead of "(s)".
- oauth_sync: use `if doc_task is not None:` to match processor_task's guard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-3 review on PR #892 found a real bug: the gateway backend's httpx.Timeout
raises httpx.ReadTimeout (a httpx.TimeoutException, NOT a builtin TimeoutError),
so the `except TimeoutError` added in r2 only covered the Mistral
(anyio.fail_after) path — gateway timeouts still fell through to
reason="error". Catch both (TimeoutError, httpx.TimeoutException) so either
backend's timeout lands in the dedicated parse_failed_reason="timeout" bucket.
Add an end-to-end test driving a gateway httpx.ReadTimeout through the
processor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-3 review on PR #893:
- record_qdrant_operation("upsert","error") now fires only when the exhausted
retry was actually a Qdrant failure (reason=="qdrant"); an embed/connection
failure exhausts retries before Qdrant is called, so attributing it to
mcp_qdrant_operations_total{error} inflated that signal. The cause is still
captured by record_ingest_dropped.
- Add test_mistral_embed_retries_on_5xx: exercises the full Mistral retry path
(5xx SDKError then success), not just the predicate.
- Add test_generate_does_not_retry_on_bad_request: generate() fast-fails on a
permanent 4xx.
- Move astrolabe_vector_ingest_dropped_total's definition into the astrolabe_
pipeline-metrics block (was in the mcp_ section).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-2 review on PR #891 (non-blocking):
- Add a parametrised test_webdav_path_encoding covering empty path,
leading-slash stripping, '#'/comma/space, and a non-ASCII name — the single
source of truth for every caller-path builder's encoding, so write_file /
delete_resource / create_directory / attachments are covered transitively.
- Document the decoded-input precondition on _webdav_path (encode-exactly-once;
passing an already-encoded path would double-encode).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-2 review on PR #892:
- OcrProcessor.process now catches TimeoutError separately and returns
parse_failed_reason="timeout" with a populated message ("OCR timed out after
Ns"), instead of conflating timeouts with API errors under "error" and logging
an empty suffix. Lets dashboards tell a too-low timeout from a failing
provider. Test added.
- Add validator-rejection tests for DOCUMENT_OCR_TIMEOUT_SECONDS=0 (gte=1) and
DOCUMENT_MAX_PDF_SIZE_MB=-1 (gte=0), matching the existing validator-test
pattern.
- Comment the _Settings test fixture's max_pdf_size_mb=0.0 default.
SonarCloud: quality gate was failing on new_security_hotspots_reviewed (S5332
"use https") from an http:// URL in the new gateway-timeout test — switched to
https:// (mirrors commit 98c9d58e).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-2 review on PR #893:
- Add test_generate_retries_on_connection_error (generate() shares the transient
retry; guards the decorator against accidental removal).
- Add test_process_document_records_drop_on_exhausted_retries: drives
process_document to retry-exhaustion and asserts record_ingest_dropped is
called once with the classified reason (processor-level coverage, not just the
_drop_reason unit).
- Note in _drop_reason that a multi-failure group is labelled by its first leaf
(best-effort, no "mixed" bucket).
SonarCloud: the quality gate was failing on new_security_hotspots_reviewed
(S5332 "use https") from http:// URLs in the test _req() helpers — switched to
https:// (mirrors commit 98c9d58e).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Collapse _init_worker_observability's docstring to one line; the WHY moves
to a concise inline comment (per review).
- Note that _fake_settings.ingest_queue is unused by the helper (test realism).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-1 review on PR #893:
- _drop_reason now descends through nested ExceptionGroups to the first leaf
(was single-level), so a doubly-wrapped cause isn't mislabelled "other";
added a nested-group test. Commented why both the httpx and openai isinstance
branches exist (raw Nextcloud-API errors vs SDK-wrapped variants).
- Documented that generate() intentionally shares the broadened transient retry
(RAG sampling path), with the worst-case latency note.
- Added a docstring note to process_document on how the provider-level retry
(5x) layers over the outer loop (3x in-process / 1x procrastinate).
- Added test_embed_batch_retries_on_connection_error for the batch path.
- Renamed test_retry_reraises_non_rate_limit_immediately ->
test_retry_reraises_when_predicate_returns_false (it tests the predicate, not
a specific status).
SonarCloud:
- S5708 (BLOCKER) on the helper's dynamic `except exception_type`: the type is
constrained to BaseException/tuple by the signature; suppressed with a
justified NOSONAR.
- S7503 (async without await) in the embed-retry test: use AsyncMock side_effect
instead of a hand-rolled async function.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- tests: use https in the OTLP endpoint fixture to clear the S5332
"http protocol is insecure" security hotspot (quality gate:
new_security_hotspots_reviewed).
- cli: add the "tracing disabled" else branch in
_init_worker_observability so the worker logs parity with app.py when no
OTLP endpoint is set.
- cli: trim the verbose inline comment in worker() (the WHY lives in the
helper docstring), per review.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-1 review on PR #892:
- Wire DOCUMENT_OCR_TIMEOUT_SECONDS into _MistralOcrBackend too (was
gateway-only): wrap process_async in anyio.fail_after so the SDK-managed
client honours the setting; on expiry it fails fast as a clean parse error.
Test added.
- Tighten the misleading "honoured without a restart" comment — per-call
get_settings() is for test monkeypatching; a live change still needs a
restart since the backend is cached for the pod lifetime.
- Comment the size guard's two intentional gaps: an explicit processor_name
override bypasses it, and the early return skips the parse-duration histogram.
SonarCloud (new-code smells in the added tests):
- S1244 float-equality asserts → pytest.approx (test_config.py, test_ocr_processor.py).
- S1186/S7503: rewrite the gateway-timeout test with mocker AsyncMock/MagicMock
instead of a hand-rolled fake client (no empty method, no async-without-await).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-1 review on PR #891:
- Guard processor_task's broad except handler against an unbound doc_task
(mirrors multi_user_processor_task): initialise doc_task=None before the loop
and branch the error log. Fixes a latent NameError if receive() raises a
non-TimeoutError/EndOfStream before the first document binds. Regression test
added.
- Drop the unnecessary `from __future__ import annotations` in vector/_errors.py
and express format_exception_group's non-group fast path as an explicit
isinstance check.
- Add a copy_resource Destination-header encoding test (analogue to MOVE);
strengthen the ExceptionGroup test to assert the full leaf repr survives.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The external split-worker ingest pods (MCP_ROLE=worker / procrastinate) had
no observability: the worker CLI entrypoint never started a Prometheus
metrics server and never configured structured logging, so the pods that do
the real parse/embed/upsert work were invisible to Prometheus and emitted
plain-text logs the platform pipeline couldn't parse.
The always-on API pod bootstraps observability in its lifespan (app.py), but
the worker has its own entrypoint and never went through that path (or
uvicorn's JSON log_config). Add `_init_worker_observability()` mirroring the
API pod: setup_logging (JSON), setup_metrics on METRICS_PORT when
METRICS_ENABLED, and setup_tracing when an OTLP endpoint is configured.
Runs after the INGEST_QUEUE=postgres check so a misconfigured worker fails
fast without binding a metrics port.
This also unblocks the document-pipeline observability shipped in #831
(Deck #175): the astrolabe_* parse/embed/chunk metrics and the
document_processor.parse span are recorded in the shared registry/processor
code the worker executes — they were simply never exposed in external mode
because the worker served no /metrics and set up no tracer.
Deck #310, unblocks #175.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
