Extract _mark_backstop_done() (overflow eviction + marker write) and
_backstop_delete_doc_type() (per-type scroll + enqueue) so
_enqueue_deletes_for_disabled_types drops from cognitive complexity 17 to well
under the 15 threshold. Behavior unchanged; tests still pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- route test for purge_doc_types raising on total failure -> 500
- route test for doc_types list containing non-strings -> 400
- reword the capabilities move_to_end comment (no-op on new keys; needed only
for the expired-key in-place update)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- app.py: move the /api/v1/vector-sync/purge mention out of the unconditional
management-endpoints log and into the vector_sync_enabled block, so operators
without Qdrant don't see an endpoint that 404s
- vector_sync route: comment why doc_types isn't whitelisted against
INDEXED_DOC_TYPES (unknown type = harmless zero-match no-op)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- vector_sync route: rename the response dict from `body` to `resp` so it no
longer shadows the request `body` (maintenance trap)
- scanner: comment the intentional files-vs-text purge timing asymmetry
- tests: add the all-text-types-disabled backstop case (empty allow-set)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- purge route: include a "failed" key in the 200 body listing requested doc
types that were not purged, so Astrolabe knows consent isn't yet enforced
for them (scanner backstop still catches up)
- tests: add @pytest.mark.unit / module-level pytestmark to the new test
modules so they run under `pytest -m unit`; add a partial-failure route test
- capabilities: comment why the cache is keyed per-user despite a global value
- semantic/scanner: doc/comment clarifications (sorted-order, eviction timing)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- tests: cover the process_document consent gate (drops an admin-disabled
index task with record_ingest_dropped("admin_disabled"); allows approved)
- scanner: _consent_backstop_done is now an insertion-ordered dict and evicts
the oldest entries to half capacity on overflow, so a bound hit re-fires the
backstop for only the oldest markers instead of the whole fleet at once
- semantic: reword the short-circuit log (consent, not installation)
- capabilities: comment why move_to_end is needed after an expired-key update
- test: assert the global purge delete-filter is owner-agnostic (doc_type only);
fix a pre-existing ty error on UnexpectedResponse(headers=None) in the file
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- app.py: register /api/v1/vector-sync/purge only when vector_sync_enabled, so
it returns 404 (not a 500 from get_qdrant_client) when sync is off
- scanner: bound _consent_backstop_done so a long-running multi-tenant process
with user churn can't grow it without limit (clears on overflow)
- purge route: distinct 400 for a missing doc_types key; enforce the admin
check even for an empty no-op request (destructive route)
- tests: missing-key 400, admin-gated empty no-op, non-admin empty 403
The _consent_narrowed_doc_types precondition is enforced by its non-Optional
frozenset[str] signature (ty rejects a None caller). The httpx.BasicAuth
SonarCloud hotspot matches the existing webhook routes (false positive,
credential from the app-password store) — left consistent for UI triage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- scanner: gate the consent backstop with a per-(user,doc_type) one-shot
marker so a standing admin-disable doesn't re-enqueue idempotent deletes
every scan tick; the marker clears when the type is re-enabled. Derive
_TEXT_BACKSTOP_DOC_TYPES from INDEXED_DOC_TYPES so new indexed types are
covered automatically
- semantic: extract _consent_narrowed_doc_types so the search-side narrowing
is unit-testable; add tests for restrict/intersect/disjoint/empty
- purge route: cap doc_types length (abuse guard) -> 400
- tests: one-shot + re-enable backstop, too-many-doc_types 400
Deferred (noted on PR): per-document allowed_doc_types call is cache-hot;
purge "last error wins" — both logged. SonarCloud broad-except hotspots are
deliberate (noqa BLE001), reviewable in the UI.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- purge route: 400 (not 500) on a valid-JSON non-object body
- scanner: backstop-purge admin-disabled note/news_item/deck_card points
(their deletion-tracking lives inside the skipped scan_* fns), mirroring the
files path; gated on a concrete allow-set so fail-open never deletes
- processor: record_ingest_dropped("admin_disabled") so consent-skipped index
tasks are observable/alertable
- app.py: list /api/v1/vector-sync/purge in the endpoints log line
- capabilities: drop empty-string doc types; return frozenset throughout
- purge: document the count-before-delete approximation
- tests: non-object body -> 400, ProvisioningRequiredError -> 428, cache TTL
expiry refetch, and the scanner consent backstop
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consume the astrolabe.semantic_search capability as the source of truth for
which content sources an admin has approved for semantic search, and enforce
it independently of Astrolabe (this server queries Qdrant directly).
- capabilities.py: cached per-user reader for enabled_doc_types (TTL+LRU,
fail-open so older Astrolabe / transient OCS errors don't break search)
- semantic search: intersect requested doc_types with the allowed set;
restrict to the allowed set when none requested; short-circuit when empty
- scanner: skip disabled sources during discovery (files discovery yields
nothing when disabled, so the existing grace-period reconcile purges them)
- processor: drop near-real-time index tasks for disabled doc_types
(webhook events bypass the scanner gate); deletes always proceed
- vector/purge.py + POST /api/v1/vector-sync/purge: admin-only global
delete-by-doc_type, called by Astrolabe when a source is disabled so
consent is binding on data-at-rest
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-2 review follow-ups (GHSA-8vh3-g2qg-2h2c PR):
- Add a dynaconf validator requiring WEBHOOK_SECRET to be >=16 chars when set
(None still allowed = webhooks disabled), so weak/placeholder secrets fail
at startup rather than in an audit. Covered by two new tests in test_config.py.
- Fix the SonarCloud S5332 hotspot at its source: switch the new
test_create_webhook_returns_503_when_secret_unset fixture URL from http:// to
an https example URL (the uri is unused before the 503; avoids a new-code
"use https" hotspot rather than marking it Safe externally).
- Nits: drop the unused app.state.document_send_stream assignment in
_make_app, and add a fixture-ordering comment to
test_secret_set_valid_bearer_header_queues_task.
(--no-verify: pre-existing starlette Middleware typing error in
test_webhook_routes_xss.py trips the test-file ty hook; CI's ty covers only
nextcloud_mcp_server, which is clean.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-1 review follow-ups (GHSA-8vh3-g2qg-2h2c PR):
- Add a unit test for the new `except WebhookSecretNotConfigured` branch in
enable_webhook_preset: returns 503 (not the generic 500) with WEBHOOK_SECRET
in the body. Uses the existing test_webhook_routes_xss.py scaffolding.
- Add a clarifying comment to test_secret_set_wrong_scheme_returns_401 about
the _client default-bearer override semantics.
(--no-verify: the pre-commit ty-check surfaces a pre-existing starlette
Middleware typing error in test_webhook_routes_xss.py unrelated to this change;
CI's ty check covers only nextcloud_mcp_server, which is clean.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GHSA-8vh3-g2qg-2h2c (CVSS 9.1, CWE-306): POST /webhooks/nextcloud had no
authentication when WEBHOOK_SECRET was unset (the default). The receiver
trusted the attacker-supplied user.uid and fed it to Qdrant, letting an
unauthenticated network caller delete or re-index any user's vector
embeddings.
Webhooks now require WEBHOOK_SECRET end-to-end:
- app.py: the /webhooks/nextcloud route is only mounted when WEBHOOK_SECRET
is set; otherwise it 404s and a startup warning notes vector sync falls
back to the polling scanner.
- webhook_receiver.py: removed the warn-and-accept fallback. No secret -> 503,
missing/invalid bearer -> 401; the payload is never processed unauthenticated.
- webhook_routes.py / api/webhooks.py: webhook_auth_pair() raises
WebhookSecretNotConfigured instead of returning authMethod="none"; both
registration entry points return a clear 503 so no dead unauthenticated
webhooks are created.
Also expose webhooks availability to the Astrolabe UI via GET /api/v1/status
("webhooks_enabled": bool), set WEBHOOK_SECRET on the docker-compose
semantic-search dev services, and update env.sample + ADR-010 / ADR-018 /
webhook-management-guide docs.
Vector sync still works without a secret via the polling scanner.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-8 review: env.sample.oauth-multi-user omitted NEXTCLOUD_PUBLIC_ISSUER_URL,
which configuration.md marks required for login_flow — a user working from the
template alone would hit the "Login URL points to localhost" failure. Add it
(with a troubleshooting pointer) and give it + NEXTCLOUD_MCP_SERVER_URL a
"REQUIRED: PUBLIC URLs" section header for consistency with the rest of the file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-7 nit: align the TOKEN_ENCRYPTION_KEY placeholder in login-flow-v2.md
(`<fernet-key>` / `<your-fernet-key>`) with env.sample.oauth-multi-user's
`<your-encryption-key>` so copy-pasters don't see a mismatch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-6 review: in env.sample.oauth-multi-user, TOKEN_ENCRYPTION_KEY and
TOKEN_STORAGE_DB sat under "OPTIONAL: SEMANTIC SEARCH", but they're required for
any login_flow deployment (per-user app passwords must be persisted). Move them
into a dedicated "REQUIRED: APP-PASSWORD STORAGE" block with a pointer to
docs/login-flow-v2.md#setup so a login_flow-without-semantic-search user copying
the template doesn't miss them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- settings.toml.example: drop the stale `enable_token_exchange = false`
deprecated-alias line (the key was removed from config.py _DEFAULTS).
- configuration-migration-v2.md: add a Quick Reference row + note that
`ENABLE_TOKEN_EXCHANGE` was removed and is now ignored; use
`MCP_DEPLOYMENT_MODE=login_flow` instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-4 review: ADR-005 (Status: Implemented) still described the token-exchange
mode (Option 2 / ENABLE_TOKEN_EXCHANGE) as an active option. Add a note to the
Implementation Note section clarifying it was removed in the ADR-022/023
consolidation and only multi-audience mode ships — consistent with the ADR-004
deprecation in this PR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- login-flow-v2.md: add commented-out NEXTCLOUD_OIDC_CLIENT_ID/_SECRET (with a
"production: register a static client" note) to the Docker Compose excerpt so
copy-pasters of the rendered snippet don't fall into the #907 DCR-expiry trap.
- ADR-004: rename "## Implementation Status" -> "## Historical Implementation
Notes" and add a banner clarifying the steps were never completed and the
ENABLE_TOKEN_EXCHANGE symbols no longer exist (the design was superseded).
- env.sample.oauth-multi-user: angle-bracket the TOKEN_ENCRYPTION_KEY
placeholder for consistency with the OIDC client placeholders.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-2 review follow-ups in unified_verifier.py:
- module-level docstring still described "two compliant OAuth modes" incl.
token exchange — rewritten to multi-audience only.
- removed the stale "# Both modes do the same validation" inline comment in
verify_token().
(--no-verify: same pre-existing ty errors in test_unified_verifier.py as prior
commits; CI type-checks only the package, which passes.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-1 review follow-ups:
- Troubleshooting "Access forbidden": note that existing users must re-authorize
once after switching to a static client (stored sessions were issued to the
now-deleted DCR client).
- Default IdP setup: explain that the `/mcp` resource identifier works because
`_has_mcp_audience` accepts both the bare server URL and the `/mcp` form.
- env.sample.oauth-multi-user: use angle-bracket placeholders
(`<your-client-id>`) to match the template convention and fail loudly if
copied verbatim.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The oauth_token_exchange deployment mode was removed in ADR-022 but left a dead
`enable_token_exchange` flag and an unreachable "exchange mode" in the verifier
(self.mode was hardcoded to "multi-audience"). Remove the remnants:
- config.py: drop the `enable_token_exchange` default and the
`ENABLE_TOKEN_EXCHANGE` branch in `_is_multi_user` (+ its doc line).
- unified_verifier.py: drop `self.mode` and the dead exchange-mode log branch;
simplify the docstrings to multi-audience only.
- test_unified_verifier.py: drop the `.mode` assertions (attribute removed);
collapse the redundant init tests.
Also remove docs/ADR-004-Code-Review.md — an orphaned code-review note, not an
ADR; it doesn't belong in the docs/ADR namespace.
(--no-verify: the ty-check hook flags 3 PRE-EXISTING type errors in
test_unified_verifier.py lines 346/362/441, untouched by this change; CI
type-checks only the package, which passes.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The OAuth token-exchange deployment mode was removed (ADR-022) and has no
implementation — only a vestigial `enable_token_exchange` flag remains. Its
documentation still presented it as a usable mode, which misleads self-hosters.
The only supported deployment modes are single_user_basic, multi_user_basic,
and login_flow.
Token-exchange removals (how-to/config for a removed mode):
- delete docs/CRITICAL-TOKEN-EXCHANGE-PATTERN.md
- delete docs/oauth-architecture-comparison.md (orphaned; labelled the removed
pass-through mode as "current implementation")
- env.sample: drop the "OAUTH TOKEN EXCHANGE MODE" section
- docker-compose.yml: drop ENABLE_TOKEN_EXCHANGE/TOKEN_EXCHANGE_CACHE_TTL from
the keycloak service (dead flags)
- docs/webhook-management-guide.md: drop the token-exchange deployment section
- docs/configuration-migration-v2.md: drop the token-exchange migration scenario
- docs/observability.md: drop the never-emitted mcp_oauth_token_exchange_total
Auth ADR status corrections:
- ADR-004: Draft -> Superseded by ADR-022/ADR-023 (token-exchange/federated
design not adopted); note the three supported modes.
- ADR-002: extend the deprecation pointer to ADR-022/ADR-023.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Self-hosting login_flow against Nextcloud's built-in `oidc` app breaks after
~1h when relying on the DCR fallback: the `oidc` app deletes
dynamically-registered clients after `client_expire_time` (default 3600s),
pruning on every /authorize. The MCP server caches the now-deleted client, so
authorize/refresh fail with an "Access forbidden" page permanently — surviving
server restart and connector recreation (issue #907).
- docs/login-flow-v2.md: add "Default IdP setup (Nextcloud oidc app)" with
static-client steps, and a Troubleshooting entry for the #907 symptom/fix;
reframe the OIDC-client env vars as strongly recommended.
- docs/configuration.md: promote NEXTCLOUD_OIDC_CLIENT_ID/_SECRET to strongly
recommended with a DCR-expiry warning; add them to the login_flow example.
- docker-compose.yml: clarify the DCR caveat and point self-hosters to a static
client for login_flow / background sync.
- env.sample.oauth-multi-user: fix the removed `oauth_single_audience` value
(now login_flow) and require a static OIDC client.
- env.sample.oauth-advanced: remove — it configured the removed OAuth
token-exchange mode (no implementation remains; the mode value now errors at
startup). Drop its references in configuration.md / configuration-migration-v2.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move 'permissions: contents: read' from workflow level to the record-deployment
job (GitHub Actions least-privilege, rule S8264), keeping this workflow uniform
with the astrolabe copy. Single-job workflow, but consistent and future-proof.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review round 1 follow-ups:
- Reference the built-in $GITHUB_SHA env var in run scripts instead of
interpolating ${{ github.sha }}, removing the GitHub Actions script-injection
surface (SonarCloud security rating on new code).
- Add a concurrency group (cancel-in-progress: false) to
pact-record-deployment.yml so back-to-back tag pushes don't race the recording.
- Add timeout-minutes: 5 to guard against a hung tailnet join.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the missing record-deployment half of the Pact can-i-deploy loop and
stops can-i-deploy from failing every merge while the broker's production
environment is still empty.
- New pact-record-deployment.yml: on tag push, records a production
deployment of nextcloud-mcp-server keyed by the tagged commit SHA, which
matches the SHA pact.yml publishes consumer pacts / verification results
with. Recording the tag string would not link to the verified pacts.
- pact.yml can-i-deploy: wrapped in shadow mode (runs for signal, emits a
warning annotation on failure, always exits 0). can-i-deploy cannot pass
until both nextcloud-mcp-server and astrolabe have recorded a production
deployment, so gating now would block merges on a bootstrap gap.
Tracked on Deck card #325. Promotion to a hard gate is a follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review round 4 (both nits):
- Rename the flag to ignore_ocr_enabled so its OCR-specific scope is explicit at
the call sites (the gate only bypasses the OCR-enabled check).
- Add test_evaluate_escalation_empty_suppressed_even_when_structured_registered:
empty_text (minimum='ocr') skips a registered structured tier and suppresses to
ocr when OCR is off, never hopping to structured.
Deck #324.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The inline _process_pdf path does not emit document_escalation_suppressed_total;
the "what-if OCR" counter is instrumented only on the per-tier external path
(evaluate_escalation / _parse_pdf_tier). Comment the inline OCR gate so a reader
doesn't mistake the omission for a bug. Deferred the assert_never nit (typing
.assert_never is 3.11+; Literal+frozen already guard construction) and the
pre-existing minimum-ValueError pass (only "ocr"/None are ever passed).
Deck #324.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- escalation: EscalationDecision.reason is now Literal["empty_text",
"low_confidence"] (parity with kind; ty catches a bad label at call sites).
- processor: nest the decision handling so the hop branch is reached via an
explicit else under `if decision is not None` — exhaustive over the Literal
kind, no None-attribute risk.
- tests: add the "OCR processor unregistered (not just disabled) → None"
quadrant, locking in absent != suppressed.
Deck #324.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- escalation: EscalationDecision.kind is now Literal["hop","suppressed"] so ty
catches a bad kind statically (and the processor branch is exhaustive).
- processor: simplify the suppressed-escalation log line (no longer repeats
to_tier / tier).
- registry: clarify _tier_available's ignore_enabled drops the OCR-enabled gate
specifically (a future per-tier gate would extend the condition).
Deck #324.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
OCR is the paid, opt-in tier (DOCUMENT_OCR_ENABLED, default off). The per-tier
escalation gate already declines to hop to OCR when it's disabled (the pre-OCR
tier is terminal — no surprise cost), but that left operators blind to how much
OCR demand exists.
evaluate_escalation now returns a structured EscalationDecision:
- "hop" — a higher tier can run; the caller raises EscalateError (queue-hop).
- "suppressed" — the ideal next tier (e.g. ocr) exists but is DISABLED; the caller
indexes the current tier's output as terminal and records the
would-be hop on the new astrolabe_document_escalation_suppressed_total
{from_tier,to_tier,reason} counter instead of hopping.
- None — index as-is (good text, or no such tier at all).
So with OCR off, escalation_suppressed_total{to_tier="ocr"} is the latent OCR
demand an operator weighs before enabling OCR; enabling it converts these into
real document_escalation_total{to_tier="ocr"} hops. next_available_tier gains an
ignore_enabled flag to compute the *ideal* (enabled-gate-ignored) target.
Tests: registry suppressed vs hop vs terminal (incl. structured-hop-not-suppressed
when OCR off but structured available); _parse_pdf_tier records suppressed +
indexes without raising.
Deck #324 (parent #323).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A stall is often systemic (a Qdrant/embedding outage stalls every in-flight
job), so reclaiming the whole batch at now() every */5min tick would
thundering-herd a recovering dependency, bypassing TieredEscalationStrategy's
per-job backoff. reclaim_stalled_ingest_jobs now offsets retry_at by a fixed
delay (INGEST_RECLAIM_RETRY_DELAY_SECONDS, default 30s; 0 = legacy immediate).
Also document the hot-vs-restart flag asymmetry: INGEST_ESCALATION_ENABLED is
re-read per job; INGEST_TRANSIENT_MAX_ATTEMPTS is snapshotted at worker startup.
Deck #323.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- metrics: update_ingest_queue_depth guarded on `not by_queue`, which conflated
None (memory backend no-op) with {} (postgres, ALL queues drained). When every
queue drains at once, get_ingest_job_counts_by_queue returns {} and the
pre-zero loop was skipped, leaving a stale ghost backlog in the gauge. Guard on
`by_queue is None` only; add an all-drained regression test.
- procrastinate: note that INGEST_TRANSIENT_MAX_ATTEMPTS is snapshotted at
blueprint-build time (restart to pick up changes).
Deck #323.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- tests: make the transient-backoff progression assertion load-independent by
bracketing the get_retry_decision call with before/after timestamps instead of
measuring against a second datetime.now() (no freezegun dependency).
- tests: use pytest.approx for the ingest-queue-depth gauge assertions —
SonarCloud python:S1244 (float == ) was a MAJOR reliability finding that
tripped the new_reliability_rating quality gate.
- processor: tighten the EscalateError lazy-bind comment (file processing already
imports the document stack via get_registry; the gating only spares the
delete / text-doc paths and module-load time).
Deck #323.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- metrics: update_ingest_queue_depth now pre-zeroes every managed ingest queue
before applying live counts, so a queue that drains to empty (and drops out of
procrastinate's list_queues_async) reads 0 instead of sticking at its last
non-zero value (ghost backlog in Grafana/alerts). Adds a regression test.
- procrastinate: comment that _is_transient_infra_error treats all qdrant errors
as transient deliberately (bounded same-tier retry; over-broad is acceptable).
- escalation: note next_tier is the building block; production routing uses
ProcessorRegistry.next_available_tier.
- tests: add evaluate_escalation fast+ocr-only low-confidence -> ocr case.
Deck #323.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
