d421bf6953ca2cc3d60a215a7c288b8d6c5cd3dd
97
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris CoutinhoandClaude Opus 4.8
|
bc957b5bc7 |
fix(review): _enum_fields validation, gate classify_from_text flags, OCR warnings
Address PR #858 review round 3: - document_tier1_engine / document_ocr_provider now validate + normalize via Settings.__post_init__ _enum_fields (the repo's canonical opt-in-enum pattern; case-insensitive) instead of dynaconf Validators. A typo now raises ValueError at load and "Gateway" normalizes to "gateway". - classify_from_text gates no_text_layer/bad_text_layer on ocr_frac >= OCR_PAGE_FRACTION, matching classify_pdf -- a "fast"-routed doc with a few junk pages no longer emits a misleading flag (keeps the shadow vs hot-path classification metrics consistent). - build_ocr_backend warns when an EXPLICIT provider is misconfigured (gateway without EMBEDDING_GATEWAY_URL, mistral without MISTRAL_API_KEY) instead of silently returning None. - Pypdfium2FastProcessor.health_check probes the import; documented why OcrProcessor.health_check is unconditionally True (lazy per-tenant backends). - Removed the leftover per-boundary / per-chunk debug logging loops. Tests: enum normalization + rejection for the two new settings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.8
|
3bd1b46d9c |
feat: tier-3 OCR processor (gateway or direct Mistral)
Adds the OCR escalation target the tiered registry already routes to. Scanned /
no-text-layer PDFs (the tier-0 "ocr" verdict) escalate here when
document_ocr_enabled (default off).
Two interchangeable backends, selected by document_ocr_provider
(auto | gateway | mistral | none):
- gateway: POST to the Astrolabe Cloud model gateway's /v1/ocr -- the same
M2M-authenticated gateway as embeddings, so NO provider keys live in the pod
(the platform default; reuses EMBEDDING_GATEWAY_URL + the M2M creds).
- mistral: call the Mistral OCR API directly from the pod (MISTRAL_API_KEY), for
self-hosters / deployments without the gateway.
"auto" prefers the gateway, then direct Mistral.
Both return per-page markdown joined into text + exact page_boundaries (the
pdf_highlighter contract; bbox re-derived from the PDF bytes as for other tiers).
Validated end-to-end via direct Mistral on the scanned Student 147.pdf:
success, 15 pages, 22k chars, offsets exact, ~4s.
Settings: document_ocr_provider (enum-validated), document_ocr_model
("mistral/mistral-ocr-latest" -- gateway routes on the prefix, the direct mistral
backend strips it). OcrProcessor registered at lowest priority so it is never the
non-tiered default.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.8
|
c48a797896 |
feat: tiered PDF processor with pypdfium2 fast path (deprecate pymupdf4llm)
Replaces single-engine pymupdf4llm extraction with a tiered pipeline (Deck #205, follows the tier-0 classifier #855). pypdfium2 becomes the default and only hot-path PDF extractor; pymupdf4llm is deprecated to a rollback toggle. Why: pymupdf4llm's O(n^2) find_tables drove the OOM (#852) and the form-PDF parse timeouts (#856), carries AGPL/commercial licensing liability, and -- per the benchmarks -- recovers near-zero usable tables on the real corpus. pypdfium2 (Apache/BSD) extracts the same text far faster (Student 1a.pdf: 120s timeout -> 0.2s) with no table-detection bomb. - document_processors/pypdfium2_fast.py: tier-1 "fast" processor emitting text + exact page_boundaries (the pdf_highlighter contract). pymupdf processor is now tier "structured" (the rollback engine), registered but not default. - registry: tiered routing in ProcessorRegistry. tier-1 fast extracts, then classification is DERIVED from that text (classifier.classify_from_text -- no PDF re-open), records the classification metrics, and escalates scanned / no-text-layer docs to the "ocr" tier when document_ocr_enabled (default off; no provider yet, so fast is terminal). Wires record_document_escalation + the real "escalated" span attribute (was hardcoded False). - Removes the separate _shadow_classify pass from vector/processor.py -- it re-opened every PDF and re-extracted text (~0.5-1.3s/doc of pure duplicated CPU that lowered throughput); classification now rides the tier-1 extraction. - Settings: document_tier1_engine ("pypdfium2" default | "pymupdf" rollback, enum-validated), document_ocr_enabled (default false). Tests: pypdfium2 extractor, registry tiering (fast routing, rollback, classify recording, OCR escalation on/off), classify_from_text. Full unit suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.8
|
044c1da750 |
feat: tier-0 document classifier in shadow mode
First step of the tiered document-processor effort (Deck #203): a cheap, local pre-pass that recommends which extraction tier a PDF should start in, emitting metrics WITHOUT changing routing yet -- so we gather per-tenant doc-mix data before turning escalation on. document_processors/classifier.py: classify_pdf(content) -> DocClassification. Page-sampled (bounded on large docs), <~1s. Cheap signals only -- text-layer chars, a text-quality score (catches the "Student 147" failure where a text layer exists but is mashed/space-less junk), and image coverage. A page that is mostly a raster image routes to OCR: its content (handwriting, stamps) isn't in any text layer. Deliberately no get_drawings/graphics-density signal -- it's slow on the exact pages it'd flag, the hotfix's graphics_limit already makes the parse safe, and the (future) tier-1 quality gate catches lost tables. Validated on the sample corpus: born-digital 2-col arxiv and a digital student record -> fast (tier 1); a scanned+handwritten form -> ocr (tier 3). Wiring (vector/processor.py): _shadow_classify runs the classifier on PDFs in a worker thread, best-effort (never blocks/fails indexing), gated by the new DOCUMENT_CLASSIFY_ENABLED setting. Metrics: astrolabe_document_classified_total {recommended_tier}, astrolabe_document_classifier_flag_total{flag}, astrolabe_document_text_quality histogram. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.8
|
77a36fa821 |
fix: lower DOCUMENT_PDF_GRAPHICS_LIMIT default 5000 -> 1000
The OOM hotfix (#852) set graphics_limit=5000, which caught the 955k-drawing bomb but let a second pathology through: form/table PDFs (e.g. student records) have ~1.5k grid-line vector drawings per page -- under 5000, so uncapped. With those pages uncapped, pymupdf4llm's O(n^2) find_tables grinds ~17s/page, so a 7-page form hits the 120s timeout. All 6 current backfill parse failures in tenant-blackbox-demo are this exact timeout (zero OOM, zero error). Measured on a 7-page sample: graphics_limit=2000 -> 119s (timeout), 1000 -> 2.9s -- with identical extracted text and ZERO recovered tables either way (the expensive analysis produces nothing useful on these dense forms). Lowering the default to 1000 makes them index in ~3s; the bomb file (955k >> 1000) stays capped, and pages with genuine simple tables (<1000 drawings) still get table detection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.8
|
93f0f4f881 |
fix(review): type timeout as float; document worker reuse + identity check
Address PR #852 round 3 (all 🟡, no blockers): - config: DOCUMENT_PARSE_TIMEOUT_SECONDS is now float (default 120.0) so a fractional value is honoured rather than silently stored in an int field; matches anyio.move_on_after's float seconds. - _isolation: comment that a clean rlimit MemoryError leaves the worker alive in anyio's pool (vs the SIGKILL/BrokenWorkerProcess path that respawns) -- acceptable since RLIMIT_AS caps virtual address space, not RSS. - processor: note the `if indexed is False` is a deliberate identity check -- a successful index (incl. dedup hit) returns None and must not be mistaken for a parse failure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.8
|
6589e8e7fc |
fix(review): require graphics_limit>=1, type _index_document, cover rlimit branch
Address PR #852 round 2: - config: DOCUMENT_PDF_GRAPHICS_LIMIT validator is now gte=1 (pymupdf4llm treats 0 as "no cap", which would re-expose the OOM); documented the zero semantics and that the per-worker mem rlimit needs a pod restart to change. - processor: annotate `_index_document -> bool | None` and document the contract so the `if indexed is False` check is explicit/type-checkable. - tests: add the RLIM_INFINITY-hard branch assertion for _apply_mem_limit (soft==target, hard stays unbounded). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.8
|
7db8d3e301 |
fix: isolate PDF parse in a subprocess so a bad file can't OOM the pod
The document processor crash-looped on one pathological PDF: pymupdf4llm's
table/graphics detection over a page with ~1M vector path items ballooned past
the 2 GiB pod limit. The parse ran in a thread, so nothing could interrupt or
memory-bound it -- a single bad file OOM-killed the whole pod.
Run the parse in an isolated worker subprocess (anyio.to_process, cancellable)
with an RLIMIT_AS memory cap and a wall-clock timeout, so a pathological file
fails THAT document instead of the pod (new document_processors/_isolation.py).
Also pass graphics_limit (default 5000) to to_markdown -- validated to cut the
known trigger page from 112 s to 23 s with bounded memory.
On a permanent parse failure the processor returns success=False (instead of
raising, which would retry 3x); vector/processor.py marks the placeholder
"failed" and skips indexing, and the scanner stops re-queuing failed placeholders
until the file changes -- so a doomed file no longer churns.
New per-tenant (per-pod env) settings: DOCUMENT_PDF_GRAPHICS_LIMIT,
DOCUMENT_PARSE_TIMEOUT_SECONDS, DOCUMENT_PARSE_MEM_LIMIT_MB. New metric
astrolabe_document_parse_failed_total{reason=timeout|oom|error} surfaces hard
failures that previously killed the process before any except ran.
First PR of the tiered document-processor effort (Deck #199); tier 0/1/3
pipeline tracked separately.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.8
|
fc3e0f28f6 |
fix: add metrics-interval validator + type/test gaps (review #850)
- Add Validator("VECTOR_SYNC_METRICS_REFRESH_INTERVAL", gte=1) so a 0/negative
value can't turn the publish loop into a busy-spin.
- Annotate count_indexed's qdrant_client param as AsyncQdrantClient.
- Add tests: exact kwarg is forwarded to qdrant count, and the placeholder
filter matches False (excludes placeholders) with chunk_index pinned to 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.8
|
fbe70ecd9c |
feat: backend-agnostic vector-sync gauges (pending/documents/chunks)
The only queue metric, mcp_vector_sync_queue_size, was updated inline by the single-user consumer (processor_task) but never by the multi-user consumer (oauth_processor_task). On multi-user tenants (e.g. blackbox-demo, 5 users) the gauge read 0 for 24h while the live anyio buffer held ~2214 pending documents (shown by /api/v1/vector-sync/status). The "indexed" figure was also a chunk count (16039 points ≈ 480 docs) mislabelled as documents. Publish a consumer-independent snapshot from a periodic task (vector/metrics_publisher.vector_sync_metrics_task), spawned in BOTH lifespan task groups (single-user and multi-user) and every queue backend: - mcp_vector_sync_pending_documents — outstanding work via ingest_status.get_ingest_pending() (anyio buffer depth or procrastinate todo+doing); also keeps the legacy queue_size gauge meaningful on all paths. - mcp_vector_sync_indexed_documents — distinct documents, counted exactly and cheaply via the one chunk_index=0 point per document (no facet). - mcp_vector_sync_indexed_chunks — total non-placeholder points. The /api/v1/vector-sync/status endpoint now returns indexed_documents (distinct docs) AND indexed_chunks separately, so documents and chunks are no longer conflated. The publisher uses approximate Qdrant counts (every-N-seconds gauge); the on-demand endpoint counts exactly. New knob: VECTOR_SYNC_METRICS_REFRESH_INTERVAL (default 20s). Fail-safe: a metrics refresh never disturbs ingest. BREAKING CHANGE: /api/v1/vector-sync/status field `indexed_documents` now holds the distinct-document count (was the chunk count); the chunk count moved to the new `indexed_chunks` field. The Astrolabe UI + the nc_get_vector_sync_status MCP tool / userinfo page are harmonized in a follow-up (Deck #195). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.8
|
ad211ee2da |
fix: make procrastinate ingest queue opt-in (default to in-process anyio)
An unset INGEST_QUEUE auto-derived "postgres" whenever DATABASE_URL was PostgreSQL, silently starting the procrastinate ingest worker (schema migration, reclaim cron, deferred jobs) on every Postgres-backed tenant — even though none had opted into the api/worker split. Observed on tenant-blackbox-demo (:0.98.0): ~600 "Deferred 1 job" log lines / 24h. Resolve an unset INGEST_QUEUE to "memory" (the in-process anyio queue) regardless of the database backend. procrastinate is now strictly opt-in via an explicit INGEST_QUEUE=postgres; the existing guard still rejects postgres against a SQLite DATABASE_URL. Docs + unit test updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.8
|
704a537847 |
chore: round-4 polish + standardize on module-level loggers
Round-4 review (non-blocking) items: - get_procrastinate_conninfo: warn on an empty connect_timeout= value (it falls back to the 10s default); preserve an explicit connect_timeout=0. - Document the _doc_queueing_lock user_id invariant (NC rejects ':' in usernames). - docs/configuration.md: note that `db downgrade` leaves procrastinate's tables in place and how to drop them on a full teardown. - reclaim_stalled_ingest_jobs: debug heartbeat log when nothing is stalled. - Drop the redundant list() wrap in the integration stalled-jobs assertion. Logging pattern: define a module-level `logger = logging.getLogger(__name__)` and use it instead of function-local or inline getLogger(__name__) calls (config.py, config_validators.py, tests/.../test_scope_authorization.py). The test file's dev-only `scripts.*` import gets a ty: ignore since it resolves via sys.path at runtime, not as an installed package. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.8
|
820b98dac1 |
fix: address PR #836 round-2 review (connect/timeout/observability)
🟡 Document why ProcrastinateTaskProducer.connect() uses `await app.open_async()` (AwaitableContext: await opens a long-lived pool, closed by drain()) and add a connect()/drain() lifecycle unit test (InMemoryConnector) asserting the pool is opened by connect and closed by drain — previously untested. 🟡 get_procrastinate_conninfo: forward connect_timeout from DATABASE_URL or default 10s so an unreachable DB can't hang worker/API startup indefinitely; warn only on other dropped query params. + tests. 🟢 INGEST_DELETE_SUCCEEDED_JOBS (default true) makes the worker's succeeded-job deletion configurable for audit retention. 🟢 Worker startup logs via logger.info (structured/OTel) instead of click.echo. 🟢 INGEST_STALLED_JOB_SECONDS (default 300) makes the crash-reclaim threshold tunable for slow embedding backends; reclaim reads it per-run. The broad `except` in _apply_ingest_queue_schema_open is kept deliberately: procrastinate wraps psycopg errors, so narrowing to psycopg.errors.* would miss the wrapped DDL-conflict and turn a benign concurrent-apply race into a failure; the presence re-check re-raises genuine errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.8
|
cfdef3c2c5 |
fix: address PR #836 review — forward task_producer to MCP contexts + cleanups
🔴 nc_get_vector_sync_status reported pending=0 for INGEST_QUEUE=postgres: the AppContext/OAuthAppContext per-session yields snapshotted the stream fields but never forwarded task_producer, so lifespan_ctx.task_producer was always None. Convert task_producer to a @property that reads _vector_sync_state live (like eviction_task_group), removing the snapshot field so the yields can't drop it. Add a regression test pinning the contract on both contexts. 🟡 Remove the unused _RECLAIM_TASK_NAME constant. 🟡 get_procrastinate_conninfo: warn + document that DATABASE_URL query params (application_name, connect_timeout, …) are dropped. 🟡 worker: open the procrastinate App once — apply_ingest_queue_schema gains manage_connection=False so the worker reuses its own open connector instead of a redundant open/close before run_worker_async. 🟢 Clarify the apply-schema broad-except comment (non-race errors re-raise) and document the deliberate Any typing in ingest_status.get_ingest_pending. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.8
|
21b7922bac |
feat: replace NATS ingest with procrastinate Postgres queue (#183)
Re-architect document ingest from the shared NATS-glued document-processor to a per-tenant, in-process model owned by nextcloud-mcp-server (Deck #183). The MCP server now owns both sides of ingest: - Producer (api role): the scanner defers one job per changed document into the app's Postgres via procrastinate (queueing_lock dedup; no execution lock, so a crashed worker can't deadlock a doc — Qdrant upserts are idempotent). - Consumer (worker role): `nextcloud-mcp-server worker` drains the queue and runs the existing process_document pipeline; a periodic task reclaims jobs orphaned in `doing` by a crash. INGEST_QUEUE selects the transport (auto: postgres when DATABASE_URL is Postgres, else the in-process anyio queue for SQLite/dev). procrastinate manages its own tables (applied on a fresh DB at startup and by `db upgrade`). The vector-sync status surface reads job counts from Postgres in postgres mode. procrastinate + psycopg3 ship in the [postgres] extra; the app's own engine still uses asyncpg (driver unification is a follow-up handled in the rendered Helm chart). NATS JetStream, the Postgres-queue stub, the bus status subscriber, and nats-py are removed. BREAKING CHANGE: the external-NATS-ingest env vars are removed (INGEST_MODE, STATUS_BACKEND, INGEST_BUS_URL, INGEST_BUS_NUM_REPLICAS, FACT_EVENT_EMITTER). Use INGEST_QUEUE (memory|postgres) and the `worker` command instead. TENANT_ID is retained (no longer NATS-subject-charset-validated). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandGitHub
|
d561b350d1 |
Merge pull request #831 from cbcoutinho/feat/document-pipeline-observability
feat(observability): astrolabe_* metrics + traces for the document pipeline |
||
|
Chris CoutinhoandClaude Opus 4.8
|
b779627fa6 |
fix(observability): address third review round
Remaining items from the PR #831 Claude review: - processor span symmetry: add "vector_sync.total_chars" to the sparse embedding span (already on the dense span) and drop the redundant "embedding.batch_size" attribute from both spans — it always equalled vector_sync.chunk_count and would mislead once batching is split. - metrics: document the deliberate "throughput counts only on full success" contract in record_document_parse (partial extractions flagged success=False are counted as a parse-error but never inflate pages/chars/bytes throughput). - config: extract _detect_base_provider() -> (family, model) as the single source of truth for the provider-detection priority chain, shared by get_embedding_model_name() and get_embedding_provider_family(). Preserves the intentional gateway asymmetry (only the family method short-circuits). - base.py: Optional[...] -> PEP 604 `... | None`; drop now-unused import. Behavior unchanged (get_embedding_* outputs covered by test_config.py). Refs Deck #175, PR #831. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.8
|
8cae7d1708 |
fix(search): address PR #834 review findings
Resolve the latest PR-review comment on the verify-on-read tag-gate work:
- tests: stringify note IDs in test_verify_on_read.py so SearchResult.id
matches production (scanner stringifies all IDs on write) — helper and the
keeps/deleted/mixed/dedupe assertions (blocking).
- tests: make the unshared-file negative control a PDF so the drop is
unambiguously "unshared", not a mime_type_filter mismatch.
- config: add Validator("VECTOR_SYNC_PDF_TAG", len_min=1) — an empty tag name
would make find_files_by_tag("") misbehave in the verifier and scanner.
- verification: correct the _verify_files comment — two batch fetches (tag
REPORT + EXCLUDED_TAGS lookup) are held under one semaphore slot; the
pure-Python intersection runs outside it.
- tests: de-duplicate the minimal-PDF constant into a shared PDF_BYTES in
tests/integration/conftest.py, imported by both integration modules.
Verified: ruff/format/ty/unit all green; the two integration modules
(10 tests) pass against a local Nextcloud (app-only, no MCP profile needed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.8
|
d4dbf01b0a |
fix(search): gate verify-on-read file results on vector-index tag membership
Verify-on-read only checked file *accessibility* (file_accessible_by_id), never tag membership, so a file removed from the `vector-index` tag (but still readable) kept surfacing in semantic search, and stale points only got evicted when they happened to rank in a search's top-K. Rework `_verify_files` to gate on current `vector-index` tag membership via a single batch `find_files_by_tag(tag, mime_type_filter="application/pdf")` REPORT per search (plus a one-shot EXCLUDED_TAGS lookup for exclusion-wins parity) — exactly what the scanner indexes. A file is kept iff it is in that set, so untagged / deleted / excluded files drop out immediately and the existing eviction wiring reclaims their Qdrant points. The gate is strict for all file results, own and shared. Mirrors the batch-fetch-and-intersect shape of `_verify_news_items` (one semaphore slot, fail-open on fetch error, malformed-id keep). - Promote the tag name to a `vector_sync_pdf_tag` Settings field (dynaconf env mapping VECTOR_SYNC_PDF_TAG) used by both scanner and verifier; drop the scanner's direct os.getenv. - Expose `find_files_by_tag` on NextcloudClientProtocol. - Rewrite the file-verifier unit tests (tagged/untagged/deleted/excluded/ fail-open/non-numeric); update the ACL + verify-on-read integration tests to seed tagged PDFs. - Amend ADR-019 and the configuration.md verify-on-read latency budget. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.8
|
68c9e20636 |
fix(observability): address second review round
- Failed deletes no longer bump astrolabe_documents_indexed_total: the outer except in process_document now gates doc_type on operation != "delete", so a delete error is counted as processed-error but not as an indexing event. Added test_failed_delete_is_processed_but_not_indexed. - registry parse span: pass record_exception=True explicitly (matches instrument_tool) and add a structured logger.warning on the parse-error path (processor/tier/byte_size/duration_ms) for a Loki-aggregatable failed-parse signal. - test_error_does_not_increment_throughput: snapshot-before/delta pattern instead of absolute 0.0 (counters are global singletons). - config: document the deliberate gateway asymmetry between get_embedding_model_name() (no gateway branch) and get_embedding_provider_family() (short-circuits on gateway). - Cleanup in touched scope: narrow `except (HTTPStatusError, Exception)` to `except Exception` (drop now-unused import); convert registry signatures from Optional[...] to `... | None`. Refs Deck #175, PR #831. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.8
|
5d205fcaab |
feat(observability): astrolabe_* metrics + traces for the document pipeline
Make per-tier bottlenecks in the document-processing pipeline
(scan -> fetch -> parse -> chunk -> embed -> Qdrant upsert) visible via
metrics, traces, and structured logs. Today the document_processors layer
emits only a logger.info line: no metric, no span, and page counts live only
inside a log string. The single processing-duration histogram is unlabeled and
whole-document, so it cannot isolate parse vs embed vs upsert.
New astrolabe_* metric family (distinct from the mcp_* protocol metrics):
- astrolabe_document_parse_{duration_seconds,total} + pages/chars/bytes counters
recorded at the ProcessorRegistry.process() boundary (covers all current and
future processors uniformly)
- astrolabe_document_escalation_total (dormant; tiered-pipeline readiness)
- astrolabe_embedding_{duration_seconds,requests_total,chunks_total,chars_total}
- astrolabe_document_chunks_total, astrolabe_documents_indexed_total{source,status}
Tracing: new document_processor.parse child span + enriched embed/chunk span
attributes (provider/model/batch_size/chunk_count). Structured logs gain a
consistent field vocabulary (doc_id, doc_type, processor, tier, pages, chars,
byte_size, chunks, duration_ms, status) so Loki can aggregate without regex.
Tier-readiness: processor/tier are labels from day one and a tier property is
added to DocumentProcessor, so adding docling/OCR/LLM tiers later is additive
(new label values, never new metrics). Tenant comes from the kube namespace
label; mime_type/model are span attributes only (cardinality). Existing
mcp_vector_sync_*/mcp_qdrant_* are left untouched.
Refs Deck #175 (superset of #173 Phase 2). Dashboard/recording-rules follow-up
tracked on #175 for homelab-argocd.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.8
|
4f170d32cb |
fix(embedding): normalize gateway base_url to the /v1 base path
EMBEDDING_GATEWAY_URL is configured as a bare origin (scheme://host:port) —
the deployment's Service URL. GatewayProvider now appends the gateway's /v1
base path before handing the URL to the OpenAI SDK, so both embed posts
({base}/embeddings) and dimension discovery ({base}/models) land under /v1.
Idempotent: a URL already ending in /v1 is left unchanged.
This lets EMBEDDING_GATEWAY_URL stay a bare domain (matching the gitops
Service URLs) instead of requiring a hand-appended /v1.
Also align the `embedding_gateway_model` field default with _DEFAULTS
("mistral/mistral-embed"). The gateway catalog is provider-namespaced, and
_detect_dimension matches `entry.id == embedding_model`; the stale
un-namespaced default would silently miss the catalog entry and leave the
dimension unresolved (re-triggering the external-mode startup crash).
Tests: bare / trailing-slash / idempotent normalization + a bare-origin
discovery test asserting /v1/models. 16 gateway-provider tests pass;
providers + vector suites green (129 total); ruff clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Chris Coutinho
|
d84a95842f |
feat(embedding): gateway provider discovers dimension via GET /v1/models
External-mode tenant pods CrashLoop at startup: Qdrant collection init calls get_dimension() before any embed(), but GatewayProvider only learns its dimension lazily after the first embed, and the gateway model isn't an OpenAI model so the base class can't know it statically. - Add GatewayProvider._detect_dimension() — the async startup hook the vector-sync bootstrap already invokes (vector/qdrant_client.py: hasattr(provider, "_detect_dimension")) for Ollama. It GETs the gateway's GET /v1/models and sets _dimension from the entry whose id matches the configured model. Best-effort: any failure (old gateway, model absent, network) leaves _dimension unset so the inherited lazy detect-on-first-embed still applies — never fatal. Presents the M2M bearer when configured. - Switch the default embedding_gateway_model to the gateway's provider- namespaced id "mistral/mistral-embed" (the gateway routes on the "/"-prefix and sends "mistral-embed" upstream); collapse a duplicated config field. Pairs with astrolabe-cloud-website#229 (gateway /v1/models, namespaced ids). Tests: discovery sets dim w/o embed, sends bearer, non-fatal on 404/absent/error, skips when already known. |
||
|
Chris CoutinhoandClaude Opus 4.8
|
d883052fb8 |
feat: add opt-in MCP decomposition hook points (design §10)
Adds the seven §10.2 hook-point modules + five env vars so Astrolabe Cloud can offload document processing to the external document-processor / embedding gateway. Purely additive: with every setting unset the server behaves exactly as today, so self-hosters are unaffected (Deck #92). Hook points (all default to current monolith behavior): - config: EMBEDDING_PROVIDER, INGEST_MODE, STATUS_BACKEND, COLLECTION_METADATA_SOURCE, FACT_EVENT_EMITTER (+ supporting settings), validated in Settings.__post_init__ (fail-fast STATUS_BACKEND=local with INGEST_MODE=external); shared canonical.py. - vector/payload_keys.py + acl_hash.py: cross-impl NAMESPACE/point_id (§2.2) and BLAKE2b-128 ACL hash (§11), pinned by fixtures shared with the document-processor repo. - embedding/gateway_client.py: OpenAI-compatible GatewayProvider authenticating via M2M OIDC client-credentials (separate realm); manual-only registry entry. - vector/collection_metadata.py: sentinel-point / API metadata source with env fallback. - vector/queue/: hexagonal ingest producer ports + memory/NATS adapters (Postgres seam); INGEST_MODE=external publishes mcp.ingest.requested.{tenant} instead of the in-memory stream and skips the in-process processor pool. The lifespan becomes a composition root across both deployment branches. - vector/queue/status.py: STATUS_BACKEND=bus subscriber feeding a StatusStore the vector-sync status endpoint reads. - admin/payload_backfill.py: POST /api/v1/admin/payload-backfill (admin scope); processor writes the new payload keys; query-side ACL pre-filter gated behind ACL_PREFILTER_ENABLED (default off). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
a5cbe91b29 |
fix(vector-sync): sweep placeholder orphans at Pod startup (#101)
When the per-tenant nextcloud-mcp-server Pod OOMKills mid-batch, the in-memory anyio processor queue is lost but the placeholder Qdrant points (is_placeholder=true, status=pending) survive. The next Pod's scanner re-runs, sees the existing placeholders, applies the 5 × VECTOR_SYNC_SCAN_INTERVAL staleness gate (~5h with the deployed 1h scan interval), and skips them. Result: 0 documents indexed for the duration of the gate after every restart. Stamps a process-level instance_id (UUID per Pod-process) onto every placeholder write. A new sweep_orphan_placeholders helper, called once from starlette_lifespan after the Qdrant client is initialised and before the scanner / user-manager spawns, scrolls the collection and deletes any placeholder whose instance_id doesn't match the current Pod's (including placeholders with no instance_id field — back-compat for pre-fix Pod versions). The scanner's next cycle naturally re-creates fresh placeholders and queues work normally; no DocumentTask reconstruction needed. Sweep is one-shot at startup, not periodic — the existing staleness gate still covers same-Pod recovery, and the cross-Pod-restart gap was the only failure mode. Failure is non-fatal (logged via vector_sync.orphan_sweep_failed) so a transient Qdrant hiccup at boot doesn't prevent the scanner from running. Both lifespan branches (single-user BasicAuth, OAuth / multi-user BasicAuth) call the sweep via a module-local helper. A new VECTOR_SYNC_ORPHAN_SWEEP_ENABLED setting (default True) provides an escape hatch. Closes Deck #101. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
e98903c502 |
fix(storage): address review on PR #799 (stale comments, docs deprecation, unit test)
claude-review on #799 flagged: 1. Stale inline comment in ``initialize()`` (line 466) still said "Postgres uses a small bounded pool". Updated to reflect both backends now use NullPool. 2. Stale ``close()`` docstring referenced pool-size starving max_connections — irrelevant with NullPool. Replaced with the NullPool-aware rationale (dispose still tears down in-flight asyncpg connections cleanly). 3. ``docs/configuration.md`` actively directed operators to tune DATABASE_POOL_SIZE / DATABASE_MAX_OVERFLOW, with worked examples and pool math. Both are now deprecated no-ops; the table entries explain the deprecation and link to PR #799. Operators reading the docs will no longer be confused into tuning settings that don't do anything. 4. ``config.py`` comment for the deprecated fields updated to record the deprecation. Validators are intentionally kept (still reject < 1 / < 0) so misconfigured deploys fail loudly rather than silently — the reviewer flagged this as a minor UX wart but explicitly "not a blocker"; the docs change in (3) keeps operators away from the config altogether. 5. New ``tests/unit/test_storage_engine.py`` with three tests: - ``test_postgres_engine_uses_nullpool`` — pins ``isinstance( engine.pool, NullPool)`` so a refactor back to QueuePool / SingletonThreadPool can't silently re-introduce the cross- event-loop crashes. - ``test_postgres_engine_ignores_pool_sizing_settings`` — setting DATABASE_POOL_SIZE / DATABASE_MAX_OVERFLOW to huge values must not change pool type (proves the deprecated fields are wired-up no-ops). - ``test_postgres_engine_missing_asyncpg_driver_message`` — guards the existing actionable-error branch when the optional ``[postgres]`` extra isn't installed. Verified: - ``uv run pytest tests/unit/`` — 1028 passed - ``uv run ruff check`` clean on the touched python files Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
d717c64750 |
fix(storage): address PR #798 round-4 review (NOSONAR syntax + pg_advisory_lock + engine dispose + nits)
Addresses all 8 items in the round-4 bot review plus 4 remaining SonarQube OPEN issues that were silently broken by round 3's malformed NOSONAR markers. NOSONAR syntax fix (clears the remaining 4 OPEN SQ issues) ---------------------------------------------------------- Round 3 used ``# NOSONAR S<rule_key>`` form. SonarQube Python doesn't recognize the rule-key suffix — it treats the whole thing as a malformed suppression directive (S7632) AND lets the underlying rule keep firing (S7503 on ``_Cursor.__aenter__/__aexit__``). Switch every marker to bare ``# NOSONAR``, with the rationale moved into a preceding comment block. Affected sites: - storage.py: ``_Cursor.__aenter__``, ``_Cursor.__aexit__`` - config.py: ``get_database_ssl()`` ``return False`` + ``ssl.create_default_context()`` - test_storage_logging.py: ``SENTINEL_PASSWORD_FRAGMENT`` constant - test_storage_postgres.py: three ``bob_pw_v1`` / ``bob_pw_v2`` / ``carol_pw`` literals Bot 🔴#1 — defensive NOSONAR on get_database_ssl `return False` -------------------------------------------------------------- Bot predicted S4830 fires on the operator-opt-out path. SQ output shows it doesn't currently fire, but bare NOSONAR added defensively with rationale comment. Bot 🔴#2 — defensive NOSONAR on f-string SQL -------------------------------------------- ``update_oauth_session`` builds its SET clause via ``f"{', '.join(update_fields)}"``; ``get_audit_logs`` builds its WHERE clause via string concatenation. Both are safe (the fragments only come from this function's own branches, no user input), but the patterns trip taint analysers. Annotated both with bare NOSONAR + safety comment explaining the hardcoded-fragments invariant. Note: S2077 doesn't currently fire on these; defensive. Bot 🟡#3 — pg_advisory_lock for concurrent migrations ----------------------------------------------------- Without coordination, two pods rolling-updating simultaneously can both observe ``has_alembic=False`` and both try to apply migrations from scratch — the second crashes with "relation already exists". New ``_migration_lock()`` async context manager: - On Postgres: ``SELECT pg_advisory_lock(:lock_id)`` on a fresh connection (separate from the engine pool so it survives the ``to_thread.run_sync`` worker), held across BOTH the schema-inspect AND the migration call. Without that span, two pods could each observe "no alembic_version" before either started migrating, defeating the lock. - On SQLite: yields immediately (file-level locking serializes writes natively). Lock ID derived from ``sha256(b"nextcloud-mcp-server:migrations")[:8]`` as a stable signed int64 so we can't collide with other apps sharing the same Postgres. Bot 🟡#4 — RefreshTokenStorage.close() + lifespan wiring -------------------------------------------------------- New idempotent ``close()`` method calls ``await engine.dispose()``, nulls the engine, resets ``_initialized``. Wired into both ``app_lifespan_basic`` (BasicAuth) and the OAuth lifespan teardown, each wrapped in ``try/except Exception`` with ``logger.warning`` so a buggy dispose can't block SIGTERM. Without this, pooled asyncpg connections leak server-side slots until ``idle_in_transaction_session_timeout`` reaps them — with small pool defaults and frequent k8s rolling restarts this can starve ``max_connections``. Bot 🟢#5 — is_sqlite_url docstring on :memory: ---------------------------------------------- Updated docstring to note both file-backed and in-memory forms are recognized; caller is responsible for ``:memory:`` magic. Bot 🟢#6 — db_path via make_url(...).database --------------------------------------------- Replaced ``database_url.split("///", 1)[1]`` hack with SQLAlchemy's own URL parsing. Naturally handles in-memory (``.database is None`` → falls back to ``""``). Same lazy-import pattern as the existing ``mask_db_password`` to avoid module-import-time cost. Bot 🟢#7 — _to_sync_url unrecognized-driver guard ------------------------------------------------- Pulled ``_KNOWN_ASYNC_DRIVERS = ("aiosqlite", "asyncpg")`` into a module constant. When an unrecognized ``+<driver>`` token survives the strip, emits ``logger.warning`` with the known-supported list. Behavior unchanged for valid URLs. Bot 🟢#8 — get_audit_logs SELECT * → explicit columns ----------------------------------------------------- Replaced ``SELECT *`` with explicit column list. Future schema additions stay out of the dict return. New tests --------- - ``test_close_disposes_engine``: pins the public contract — engine nulled, state reset, second call is a no-op. - ``test_concurrent_initialize_serialized_by_advisory_lock``: spawns 3 concurrent inits against a fresh schema; asserts no "relation already exists" and exactly one ``alembic_version`` row at the end. Without the lock, this reliably fails on the second concurrent task. Docs ---- - ADR-026: new "Concurrent migrations across pods" subsection documents the advisory-lock approach + lock-ID derivation. Verification ------------ - ``uv run pytest tests/unit/`` — 1025 passed. - ``TEST_DATABASE_URL=… uv run pytest tests/integration/test_storage_postgres.py -m postgres`` — 9 passed (was 7). - ``ruff check && ruff format --check && ty check`` — clean. Expected post-push: SQ scan reports 0 OPEN issues (was 4). Tracked on Astrolabe Cloud POC board, card #99. --- _This PR was generated with the help of AI, and reviewed by a Human_ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
51419329b0 |
fix(storage): address PR #798 round-3 review (SonarQube + pool sizing + RETURNING test)
Round-3 fixes. Two threads: - 9 OPEN SonarQube issues caused the "E Security Rating on New Code" gate failure. The bot's diagnosis (sa.text(text_sql) → SQL injection) was a wrong guess; the actual SQ rules firing were different. - Bot's substantive concerns: pool defaults too aggressive, delete_browser_session RETURNING path untested on Postgres, schema_version legacy table created on Postgres, stale module docstring. - User's underlying question on the pool: "isn't 1 connection enough?" Right-sized to 2+5 and documented the concurrency model in ADR-026 so the rationale is durable. SonarQube quality-gate fixes (clears all 9 OPEN issues) ------------------------------------------------------- - BLOCKER S6418: rename `SECRET` constant in test_storage_logging.py to `SENTINEL_PASSWORD_FRAGMENT` + NOSONAR with rationale. - CRITICAL S3776: extract `_build_postgres_engine()` from `initialize()` (was complexity 26 > 15); incidentally creates a clean unit-test seam for engine args. - CRITICAL S4423: `ssl.create_default_context(cafile=...)` is flagged as "weak protocol" — Python 3.10+ already negotiates the strongest available protocol. Explicitly pass `purpose=ssl.Purpose.SERVER_AUTH` and NOSONAR with the Python-version rationale. - MAJOR S3358: split the TLS-mode nested ternary in the engine factory into a `_describe_ssl_arg()` helper. - MAJOR S2068 ×3: bind test app-password literals to local vars and put `# NOSONAR S2068` on the same line as the literal (anchoring requirement) instead of on the closing paren. - MINOR S7503 ×2: `# NOSONAR S7503` on `_Cursor.__aenter__/__aexit__` — they MUST be `async` per the context-manager protocol. Pool sizing right-sized (answers "why so many connections?") ------------------------------------------------------------ - `DATABASE_POOL_SIZE` default 10 → **2**. - `DATABASE_MAX_OVERFLOW` default 20 → **5**. - Per-pod max drops from 30 to 7. With 3 replicas, total = 21 connections (was 90) — well under managed-Postgres `max_connections=100`. - New INFO log at startup: `Postgres engine ready: pool_size=N max_overflow=M (per-pod max K connections)`. Surfaces the active sizing without grepping config. - New ADR-026 § "Concurrency model and pool sizing" explains asyncpg's single-flight connection semantics, the MCP workload shape (read-mostly point lookups), why-not-1 (multi-user serialization), and the tune-up/tune-down recipe. - `docs/configuration.md` table updated with new defaults + homelab-vs-prod tuning guidance, linking the ADR. RETURNING path covered on Postgres ---------------------------------- - New `test_browser_session_delete_returning` exercises the `DELETE … RETURNING user_id` path — the only RETURNING clause in the storage layer and the most dialect-sensitive SQL in this PR. Asserts both present-row (returns True, row gone) and absent-row (returns False) branches. Schema portability polish ------------------------- - `alembic 001`: gate `schema_version` table creation on `op.get_bind().dialect.name == "sqlite"`. The table exists purely to match the fingerprint of pre-Alembic SQLite databases; fresh Postgres installs no longer carry the dead legacy table. Misc polish ----------- - Module docstring: "SQLite-based" → "SQL-backed", with a sentence on the DATABASE_URL opt-in and an ADR-026 link. - Comment on `_wrap_row` noting `row._mapping` is the documented RowMapping accessor in SQLAlchemy 2.x despite the underscore. Skipped (rationale in PR reply) ------------------------------- - `_qmark_to_named` SQL-comment handling: docstring already notes the limitation; no `?` in storage SQL comments today. - Module-level `anyio.Lock()`: established precedent confirmed by the bot itself. - `get_audit_logs` `SELECT *`: pre-existing pattern, out of scope. Verification ------------ - `uv run pytest tests/unit/` — 1025 passed. - `TEST_DATABASE_URL=… uv run pytest tests/integration/test_storage_postgres.py -m postgres` — 7 passed. - `ruff check && ruff format --check && ty check` — clean. - Confirmed `schema_version` absent on fresh Postgres, still present on fresh SQLite. Tracked on Astrolabe Cloud POC board, card #99. --- _This PR was generated with the help of AI, and reviewed by a Human_ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
f2b7bf132f |
fix(storage): address PR #798 review feedback (credentials, asyncpg extra, TLS, pool)
Round-2 fixes after the bot review on PR #798 plus two user follow-ups (self-signed Postgres support; asyncpg should be a PyPI extra). Folded into the same PR rather than a follow-up since the work is still unmerged. Security -------- - Mask database credentials in all 5 log call sites (storage.py × 4, migrations.py × 1) via a new `mask_db_password()` helper in config.py. Uses SQLAlchemy's `make_url(...).render_as_string(hide_password=True)` with a regex fallback so the masking path never raises. - New `tests/unit/test_storage_logging.py` asserts a sentinel password never appears in `caplog` during `RefreshTokenStorage.initialize()`. Distribution ------------ - `asyncpg` moved to `[project.optional-dependencies] postgres` so a vanilla `pip install nextcloud-mcp-server` no longer pulls in the ~5 MB C extension. The Docker image runs `uv sync --extra postgres`, so containerized deployments are unchanged. - When `DATABASE_URL=postgresql+asyncpg://...` is set on a venv missing the extra, `RefreshTokenStorage.initialize()` raises a friendly RuntimeError pointing at `[postgres]` rather than the generic ModuleNotFoundError. TLS for the Postgres backend ---------------------------- - New `DATABASE_VERIFY_SSL` + `DATABASE_CA_BUNDLE` env vars mirror the existing `NEXTCLOUD_VERIFY_SSL` / `NEXTCLOUD_CA_BUNDLE` pattern (validators in Settings.__post_init__, `get_database_ssl()` helper alongside `get_nextcloud_ssl_verify()`). `DATABASE_VERIFY_SSL=false` wins over `DATABASE_CA_BUNDLE` for incident-response convenience. - Default is **None** rather than True — keeps PR #798's behavior intact for cluster-internal Postgres that runs without TLS. Operators opt into verify-full or supply a private CA. ADR-026 records the reasoning vs the Nextcloud HTTPS default. - Engine factory in `storage.py` passes `ssl` via `connect_args` only when `get_database_ssl()` returns non-None; otherwise asyncpg's default (`prefer`) applies. - Storage logs which TLS mode is active at INFO (no secret material). Configurable connection pool ---------------------------- - `DATABASE_POOL_SIZE` (default 10) and `DATABASE_MAX_OVERFLOW` (default 20) replace the hardcoded engine values. With many replicas this can blow past managed-Postgres `max_connections=100`; tune down for large fleets. - gte-1 / gte-0 validators in __post_init__ reject 0/negative pool sizes at startup with the offending value in the error. Consistency polish ------------------ - Migration 006: convert raw `op.execute("ALTER TABLE ... ADD COLUMN")` to `op.batch_alter_table(...).add_column(sa.Column("nonce", sa.Text))` for stylistic consistency with the rewritten 001-005. Downgrade now drops the column instead of being a no-op. - `registered_webhooks.created_at` standardized from `sa.Float` to `sa.BigInteger` (all other `*_at` columns); `store_webhook()` casts `time.time()` → `int`. - `is_sqlite_url()` made case-insensitive. Testing ------- - New `tests/integration/test_storage_postgres.py::test_cleanup_expired_roundtrip` exercises `cleanup_expired_tokens`, `cleanup_expired_sessions`, and `cleanup_expired_browser_sessions` — relies on DELETE rowcount, historically dialect-tricky. - `tests/unit/test_ssl_config.py` extended with `TestDatabaseSSLSettings` + `TestGetDatabaseSSL` classes (9 new tests) mirroring the existing Nextcloud SSL tests one-for-one. Docs ---- - `docs/configuration.md` Centralized-Storage section grew the four new env vars + a homelab example with a private CA. - `docs/ADR-026` grew Distribution, TLS, and `alembic/env.py` async-pattern subsections explaining the non-obvious design choices. Helm chart counterpart in cbcoutinho/helm-charts PR #34 (separate commit on `feat/nextcloud-mcp-server-database-url`). Verification ------------ - `uv run pytest tests/unit/` — 1025 passed. - `TEST_DATABASE_URL=... uv run pytest tests/integration/test_storage_postgres.py -m postgres` — 6 passed (including new cleanup test). - `uv run ruff check && uv run ruff format --check && uv run ty check -- nextcloud_mcp_server` — clean. Tracked on Astrolabe Cloud POC board, card #99. --- _This PR was generated with the help of AI, and reviewed by a Human_ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
292cbb3292 |
feat(storage): pluggable database backend via DATABASE_URL (ADR-026)
Adds a `DATABASE_URL` setting that lets `RefreshTokenStorage` run against
any SQLAlchemy async backend, primarily `postgresql+asyncpg://...` for
HA k8s deployments. Default behavior is unchanged: when `DATABASE_URL` is
unset the server falls back to the existing `TOKEN_STORAGE_DB` path /
ephemeral SQLite tempfile.
Why
---
Today every MCP pod needs its own PVC to hold the SQLite file, which
pins the Deployment to one replica and blocks horizontal scaling. With
this change, operators can point all replicas at a shared Postgres
(CNPG, RDS, etc.) and the pods become stateless. Encryption stays in
Python (Fernet); the database only sees ciphertext.
What changed
------------
- `config.get_database_url()` resolves DATABASE_URL → TOKEN_STORAGE_DB →
ephemeral tempfile in that priority order.
- `RefreshTokenStorage` builds a process-shared `AsyncEngine` in
`initialize()`. SQLite gets NullPool; Postgres gets pool_size=10,
max_overflow=20, pool_pre_ping=True. 30 aiosqlite call sites adapted
via a thin `_DBConn` / `_Cursor` / `_Row` / `_ExecuteCtx` shim so
existing method bodies need no churn beyond the connection
context-manager swap.
- 7 `INSERT OR REPLACE` statements rewritten as portable
`INSERT ... ON CONFLICT (...) DO UPDATE` (SQLite ≥ 3.24, Postgres ≥ 9.5).
- `sqlite_master` legacy-detection lookup replaced with SQLAlchemy
inspector so the path works against either backend.
- File-permission hardening + parent-dir creation gated on
`is_sqlite_url(...)` — centralized backends manage their own filesystem.
- Alembic migrations 001/002/003/005 converted from raw `op.execute(SQL)`
to portable `op.create_table()` / `op.create_index()` with SQLAlchemy
types. All timestamp columns are `sa.BigInteger` so Postgres allocates
BIGINT (unix epochs don't fit in INT4). SQLite treats BIGINT as
INTEGER, so existing deployments at revision 006 see no schema drift.
- `migrations.py` + CLI take URLs; `db {upgrade,downgrade,current,history}`
gain `--database-url / -u` alongside the legacy `--database-path / -d`.
`get_current_revision()` uses SQLAlchemy inspector instead of raw
sqlite3, so the CLI works against Postgres too.
- `docker-compose.yml` adds a `postgres-test` service under the
`postgres` profile (pinned `postgres:16-alpine` digest) for
integration testing.
- Unit storage tests parametrized over backends via shared
`tests/fixtures/storage_backend.py` — every test in
`test_app_password_storage.py` and `test_webhook_storage.py` runs
once per backend that is available. Postgres is opted in by
`TEST_DATABASE_URL`.
- New `tests/integration/test_storage_postgres.py` (5 tests, marked
`postgres` + `integration`) covers refresh-token, app-password,
OAuth-session, webhook, and audit-log paths end-to-end on Postgres.
- New `docs/ADR-026-pluggable-database-backend.md` records the decision;
`docs/configuration.md` documents `DATABASE_URL` with examples.
Out of scope
------------
- No SQLite → Postgres data migration tool (clean cutover; tokens reissue
on next login, webhooks re-register on next sync tick).
- This repo does not provision Postgres. The matching helm chart change
lives in cbcoutinho/helm-charts (database.url / existingSecret values).
Verification
------------
- `uv run pytest tests/unit/` — 1012 passed, SQLite path unchanged.
- `docker compose --profile postgres up -d postgres-test`
- `TEST_DATABASE_URL=... uv run pytest tests/integration/test_storage_postgres.py -m postgres -v`
— 5 passed.
- `TEST_DATABASE_URL=... uv run pytest tests/unit/test_app_password_storage.py
tests/unit/test_webhook_storage.py` — 50 passed (25 per backend).
- `uv run ruff check && uv run ruff format --check && uv run ty check -- nextcloud_mcp_server` — clean.
Tracked on Astrolabe Cloud POC board, card #99.
---
_This PR was generated with the help of AI, and reviewed by a Human_
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
79ea4e9e21 |
fix(config): emit background-ops advisory logs once per process
`_get_background_operations_enabled()` was emitting three advisory log lines (1 INFO + 2 deprecation WARNINGs) on every call. Because `get_settings()` is intentionally non-cached and runs on every MCP tool invocation via `get_client()`, the "Automatically enabled background operations for semantic search in multi-user mode" INFO line was firing per-request — 569 entries/hour in one production tenant. Gate the three log emissions behind a module-level `_bg_ops_advisories_logged` flag, mirroring the existing `_warn_missing_secret_once` precedent in `vector/webhook_receiver.py`. The boolean-derivation path stays unchanged, so the `Settings` value remains fresh per call. Extends the autouse `_reload_dynaconf_after_test` fixture to reset the new flag between tests, and adds two regression tests that call `get_settings()` five times and assert each advisory fires exactly once. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris Coutinho
|
55ca44c9ec |
Merge remote-tracking branch 'origin/master' into chore/lazy-logging-g004-sweep
# Conflicts: # nextcloud_mcp_server/vector/oauth_sync.py |
||
|
Chris CoutinhoandClaude Opus 4.7
|
665cb9b1eb |
refactor: convert f-string logging to lazy %-style format (G004)
Sweep all 1676 G004 violations across 112 files, converting
`logger.<level>(f"…{x}…")` to `logger.<level>("…%s…", x)`.
Why: ruff rule G004 was added to pyproject.toml to enforce lazy
%-style logging — defers formatting until the log level is enabled
and lets structured log tooling match the unformatted template.
Conversion preserves rendered output byte-for-byte:
- `{x}` → `%s` + `x`
- `{x!r}` / `{x!s}` / `{x!a}` → `%r` / `%s` / `%a`
- Format specs (`{x:.2f}`, `{x:>10}`) → `%s` + `format(x, 'spec')`
(printf-style specs aren't 1:1 with Python format specs, so we
delegate to `format()` to keep identical output)
- Literal `%` → `%%`
- Concatenated f-strings (`f"a {x} " "b"`) flattened
- Trailing kwargs (`exc_info=True`) preserved
Verified:
- `uv run ruff check --select G004` → 0 violations
- `uv run ty check -- nextcloud_mcp_server` → passes
- `uv run pytest tests/unit/` → 1010 passed
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
18db9d6cb3 |
refactor: prune dead pre-LOGIN_FLOW config/runtime branches
Two small post-merge cleanups deferred from PR #787 (ADR-022 follow-up). Both were explicitly noted in the reviewer's "acknowledged deferred items" list. 1. config.py: drop `enable_multi_user_basic_auth` and `enable_login_flow` from the dynaconf `_DEFAULTS` dict. They were removed from `_field_map` in PR #787, so `get_settings()` never read them anyway, but their presence in `_DEFAULTS` was visually misleading — readers might think they could be set via TOML when in fact `Settings.__post_init__` derives them from `MCP_DEPLOYMENT_MODE`. Replaced with a NOTE comment pointing at the canonical derivation site. 2. app.py: the lifespan code had `use_basic_auth = not oauth_enabled or settings.enable_login_flow`, which became always-True once PR #787 enforced `oauth_enabled ↔ enable_login_flow` via __post_init__. Hard-coded to `True` with a comment explaining the invariant and pointing at the separate follow-up that will prune the now-unreachable `use_basic_auth=False` code paths in `vector/oauth_sync.py` (which includes deleting the `use_basic_auth` parameter from `user_manager_task` / `oauth_processor_task` and the OAuth-token-refresh branch in `get_user_client`). Kept the variable name and the call-site conditionals as-is for now so that follow-up is a clean mechanical diff. No runtime behaviour change: `use_basic_auth` already evaluated to True in every supported mode after PR #787, and the `_DEFAULTS` entries were already shadowed by `__post_init__`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
1fa4c82fd2 |
chore: address review-round-4 nits — stale delenv, upgrade hint, in-sync notes
Four small follow-ups from the reviewer's latest pass:
- tests/unit/test_stdio.py:18: the single_user_env fixture used
monkeypatch.delenv("ENABLE_MULTI_USER_BASIC_AUTH", ...). That env var
is no longer read after the ADR-022 follow-up; switched to delenv of
MCP_DEPLOYMENT_MODE which is the canonical mode-selection input today.
Comment updated to match.
- config_validators.py: when detect_auth_mode rejects an invalid
MCP_DEPLOYMENT_MODE, surface a one-line ADR-022 migration hint if the
rejected value is exactly "oauth_single_audience" (the most common
upgrade pain — users carrying that value over from ADR-021 .env files).
Other invalid values get the regular "Valid values: …" message
unchanged.
- config.py + config_validators.py: added cross-reference comments on
both mode-resolution sites (Settings.__post_init__ and
detect_auth_mode) noting that they each compute the canonical mode
independently and must be kept in sync when a new mode is added.
Surfaces the parallel-duplication intentionally so the next maintainer
doesn't have to discover it.
- docs/ADR-021-configuration-consolidation.md:92: appended a trailing
comment to the historical "valid values" example, marking
oauth_single_audience and oauth_token_exchange as removed in ADR-022.
ADR-021 stays as the historical record; the trailer points future
readers at the current state.
No functional changes; 1009 unit tests still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
ade42b55dc |
docs: clear review-round-3 nits — stale login_flow_v2, duplicates, field comments
Five small findings from the reviewer's third round, plus a SonarCloud
quality-gate failure on a test fixture.
- docs/troubleshooting.md, docs/configuration.md: six pre-PR references
to a non-existent `login_flow_v2` mode value (the actual enum value is
`login_flow`). They predated this PR but became actively misleading
once `detect_auth_mode` started raising ValueError for anything not in
the mode_map. Replaced with `login_flow` via sed.
- docs/configuration-migration-v2.md: removed a duplicate
`MCP_DEPLOYMENT_MODE=multi_user_basic` line in the troubleshooting
section (around line 447) — same shape as the round-2 duplicate
caught earlier in the migration-steps section. Also dropped the
`oauth_token_exchange` row from the mode-value table around line 364
(that enum value was removed in
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
6e7c821761 |
fix(config): derive mode flags in Settings.__post_init__; address review round 2
The integration jobs for `mcp-multi-user-basic` and `mcp-login-flow`
were failing with HTTP 500s. Root cause: `get_settings()` builds a
fresh Settings on every call (not cached). Commits 3 and 4 set the
derived `enable_login_flow` / `enable_multi_user_basic_auth` flags as
a side effect of `detect_auth_mode`. detect_auth_mode runs once at
startup, against the Settings instance owned by `validate_configuration`.
Every per-request call site that does `settings = get_settings()` got
a fresh Settings with both flags at their default `False` (since the
env-var aliases were dropped), causing the multi-user dispatcher in
`context.py` to take the wrong branch and crash.
Fix: move the derivation into `Settings.__post_init__`. Every Settings
instance now carries correct flags from the moment it's constructed —
no caching needed, no mutation-after-construction race. detect_auth_mode
becomes a pure reader of the already-derived state.
The legacy env-var deprecation check moves with it. It also picks up
the reviewer's truthy-string fix: previously `os.getenv(legacy)` fired
for the literal string "false" (a non-empty Python string is truthy),
which would have errored on any user with a leftover
`ENABLE_LOGIN_FLOW=false` in their `.env`. The check now only fires
when the value lowercases to one of {"1", "true", "yes", "on"}.
- nextcloud_mcp_server/config.py: extend Settings.__post_init__ with
the legacy-deprecation block and the derived-flag derivation
(resolve mode from deployment_mode + username/password, set flags).
- nextcloud_mcp_server/config_validators.py: drop the
`_sync_derived_flags` helper (superseded by __post_init__). Drop the
legacy-env-var deprecation block (moved). `detect_auth_mode` is now
pure — no mutation. Drop the now-unused `import os`.
- tests/unit/test_config_validators.py: legacy-env-var tests now
expect `ValueError` at `Settings(...)` construction (via `get_settings()`),
not at `detect_auth_mode` call. Added two new tests:
* `test_legacy_env_var_check_ignores_falsy_strings` — pins the
truthy-string fix (reviewer round 2 finding).
* `test_derived_flags_stable_across_get_settings_calls` — regression
test pinning the integration-test fix (two consecutive
`get_settings()` calls return Settings instances with the same
derived flags).
Also reworked `test_login_flow_mode_auto_derives_enable_login_flow_flag`
to assert at-construction derivation (not the old mutation pattern).
- docs/configuration-migration-v2.md: dropped the duplicate
`MCP_DEPLOYMENT_MODE=multi_user_basic` line (review round 2 nit — a
sed artifact from commit 4).
- docs/ADR-021-configuration-consolidation.md: sed-replaced the in-body
`MCP_DEPLOYMENT_MODE=oauth_single_audience` examples with `login_flow`
(review round 2 nit — only the status header was updated in commit 4).
- tests/conftest.py: docstring comment for the multi-user-basic fixture
switched from `ENABLE_MULTI_USER_BASIC_AUTH=true` to
`MCP_DEPLOYMENT_MODE=multi_user_basic` (review round 2 nit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
282c245da1 |
refactor(config)!: drop ENABLE_MULTI_USER_BASIC_AUTH env var, fail loud on legacy aliases
Same pattern as the ENABLE_LOGIN_FLOW removal in the previous commit:
the deployment mode (MCP_DEPLOYMENT_MODE) is the single source of truth
for selecting an auth flow. The ENABLE_MULTI_USER_BASIC_AUTH env-var
alias is redundant with `MCP_DEPLOYMENT_MODE=multi_user_basic`.
Unlike the ENABLE_LOGIN_FLOW removal — where silent removal was safe
because Login Flow v2 is the auto-detection default — silent removal
here would be a surprise: a user with only ENABLE_MULTI_USER_BASIC_AUTH=true
in their .env would auto-detect into LOGIN_FLOW after upgrade (wrong
runtime mode). Mitigation: detect_auth_mode now reads os.environ
directly for both legacy aliases and raises ValueError with a one-line
migration message if either is set. Applied retroactively to
ENABLE_LOGIN_FLOW as well — loud is better than silent.
- nextcloud_mcp_server/config.py:
- Drop the dynaconf env-var alias entry for ENABLE_MULTI_USER_BASIC_AUTH.
- Update the `enable_multi_user_basic_auth` field docstring to mark it
as derived / not user-settable.
- `_is_multi_user_mode()` (early-config helper, runs before Settings
is built) switched to checking MCP_DEPLOYMENT_MODE directly. Now
consistent with the canonical detection in detect_auth_mode.
- nextcloud_mcp_server/config_validators.py:
- Drop the auto-detection branch (`if settings.enable_multi_user_basic_auth`).
Selection of MULTI_USER_BASIC is now exclusively via the explicit
MCP_DEPLOYMENT_MODE branch.
- Add `enable_multi_user_basic_auth` to `_sync_derived_flags` alongside
`enable_login_flow` — both flags are now derived from the resolved mode.
- Drop `enable_multi_user_basic_auth` from
`MODE_REQUIREMENTS[MULTI_USER_BASIC].required` and from the
`forbidden` lists of SINGLE_USER_BASIC and LOGIN_FLOW (no longer
user input → no meaningful forbidden check).
- Add loud-deprecation `ValueError` block at the top of detect_auth_mode
that errors with a clear migration message when ENABLE_MULTI_USER_BASIC_AUTH
or ENABLE_LOGIN_FLOW is found in os.environ.
- tests/unit/test_config_validators.py:
- Switch ~10 fixtures from `enable_multi_user_basic_auth=True` to
`deployment_mode="multi_user_basic"` (mirrors `enable_login_flow`
treatment from the previous commit).
- Switch two `patch.dict(os.environ, {"ENABLE_MULTI_USER_BASIC_AUTH": "true"})`
blocks to use MCP_DEPLOYMENT_MODE.
- Rename `test_forbidden_multi_user_basic_auth` to
`test_forbidden_multi_user_basic_when_credentials_present` — the
scenario is now an explicit-mode + credentials conflict, not an
env-var-flag conflict.
- Add `test_legacy_enable_multi_user_basic_auth_env_var_errors` and
`test_legacy_enable_login_flow_env_var_errors` to exercise the new
loud-deprecation ValueError path.
- docker-compose.yml: mcp-multi-user-basic profile switched to
`MCP_DEPLOYMENT_MODE=multi_user_basic`.
- env.sample: replaced `#ENABLE_MULTI_USER_BASIC_AUTH=true` example with
`#MCP_DEPLOYMENT_MODE=multi_user_basic`.
- docs/authentication.md, configuration.md, troubleshooting.md,
auth-flows.md, webhook-management-guide.md,
configuration-migration-v2.md, ADR-025: replaced env-var examples
with the canonical MCP_DEPLOYMENT_MODE form.
- docs/ADR-020: marked partly superseded by ADR-022.
- CLAUDE.md: Multi-User BasicAuth section updated to set
MCP_DEPLOYMENT_MODE.
- nextcloud_mcp_server/vector/oauth_sync.py: module docstring updated.
BREAKING CHANGE: ENABLE_MULTI_USER_BASIC_AUTH is no longer read from
the environment, and setting it now raises a startup ValueError with
a migration message. Replace `ENABLE_MULTI_USER_BASIC_AUTH=true` with
`MCP_DEPLOYMENT_MODE=multi_user_basic`. The same loud-deprecation
check is also applied to the recently-removed ENABLE_LOGIN_FLOW —
replace with `MCP_DEPLOYMENT_MODE=login_flow` (or drop both;
`login_flow` is the auto-detect default when no other auth env vars
are set).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
df4994e860 |
refactor(config)!: derive enable_login_flow from mode, remove ENABLE_LOGIN_FLOW env var
Once OAUTH_SINGLE_AUDIENCE was renamed to LOGIN_FLOW and the validation
gate ensured the only meaningful configuration was
`MCP_DEPLOYMENT_MODE=login_flow + ENABLE_LOGIN_FLOW=true`, the two
controls became redundant. Setting the mode is sufficient; the
ENABLE_LOGIN_FLOW env var doesn't add information.
This commit makes the deployment mode the single source of truth for
the Login Flow v2 toggle:
- `nextcloud_mcp_server/config.py`: drop the `ENABLE_LOGIN_FLOW`
dynaconf env-var alias. The `enable_login_flow` field stays as an
internal attribute so the 6 runtime call sites (app.py x4,
context.py, auth/scope_authorization.py) keep working unchanged.
Updated field docstring to flag it as derived.
- `nextcloud_mcp_server/config_validators.py`:
- Drop `enable_login_flow` from `MODE_REQUIREMENTS[LOGIN_FLOW].required`.
- Drop the validation gate that required ENABLE_LOGIN_FLOW=true for
LOGIN_FLOW mode (no longer possible to misconfigure — the flag is
derived, not user input).
- Add `_sync_derived_flags()` helper called at every return path of
`detect_auth_mode` to set `settings.enable_login_flow` from the
resolved mode.
- `tests/unit/test_config_validators.py`: drop `enable_login_flow=True`
from happy-path fixtures (no longer needed — detection sets it).
Repurpose `test_login_flow_requires_enable_login_flow_flag` into
`test_login_flow_mode_auto_derives_enable_login_flow_flag` which
asserts the new auto-derivation behaviour for both LOGIN_FLOW and a
non-LOGIN_FLOW mode.
- `docker-compose.yml`: remove `ENABLE_LOGIN_FLOW=true` from the
`mcp-login-flow` and `mcp-keycloak` profiles.
- `env.sample`: remove the ENABLE_LOGIN_FLOW reference; the comment
on `MCP_DEPLOYMENT_MODE` now notes the derived flag.
- `docs/configuration.md`, `docs/authentication.md`,
`docs/login-flow-v2.md`, `docs/auth-flows.md`,
`docs/troubleshooting.md`, `docs/ADR-025-*.md`: replace
ENABLE_LOGIN_FLOW=true examples and references with
MCP_DEPLOYMENT_MODE=login_flow.
BREAKING CHANGE: `ENABLE_LOGIN_FLOW` is no longer read from the
environment. Anyone who relied on `ENABLE_LOGIN_FLOW=true` to activate
Login Flow v2 should set `MCP_DEPLOYMENT_MODE=login_flow` instead (or
rely on it being the default when no other auth env vars are set).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
cafd318f36 |
refactor(config)!: rename OAUTH_SINGLE_AUDIENCE to LOGIN_FLOW, gate on ENABLE_LOGIN_FLOW
The AuthMode.OAUTH_SINGLE_AUDIENCE enum was a vestige of ADR-021's
original design where it co-existed with OAUTH_TOKEN_EXCHANGE. The
un-augmented OAuth bearer pass-through it represented relied on
Nextcloud-side patches to user_oidc (Bearer token validation on
non-OCS endpoints) that were never merged upstream (see
docs/authentication.md, docs/login-flow-v2.md). The working path —
mcp-login-flow profile — sets ENABLE_LOGIN_FLOW=true on top of this
mode so Login Flow v2 acquires per-user Nextcloud app passwords via
a browser flow. With OAUTH_TOKEN_EXCHANGE removed in
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
e360a7782b |
refactor(providers): address PR #772 review — shared retry, cleaner imports, no-op close
Addresses the Claude Code review on PR #772 plus the SonarCloud S1192 finding: - Extract `retry_on_rate_limit` into `nextcloud_mcp_server/providers/_retry.py` as a parametric decorator. OpenAI and Mistral now share the same backoff loop; future providers can reuse it without copy-paste. - New `tests/unit/providers/test_retry.py` covers the decorator: 429 retry + success, non-429 immediate re-raise, MAX_RETRIES exhaustion, default predicate, and unrelated exception passthrough. - Tighten Mistral SDK import to `from mistralai.client.errors import SDKError` (the canonical sub-path; the reviewer's `from mistralai.models import SDKError` does not exist in mistralai 2.4.5). - Replace `MistralProvider.close()`'s direct `__aexit__` call with a no-op + comment — the Speakeasy-generated client has no public close hook and the underlying httpx client is closed by GC. - Extract the duplicated "Embedding not supported" message to a module-level constant (SonarCloud S1192). - Align `Settings.get_embedding_model_name()` Bedrock check with the registry by also considering `bedrock_generation_model`. - Add the `mock_mistral_client` fixture to `test_mistral_no_embeddings_disabled` for parity with the rest of the file. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
3268a13d11 |
feat(providers): add Mistral embedding provider, route registry through dynaconf
Adds a hosted Mistral embedding option (mistral-embed, 1024-dim) alongside the existing Bedrock / OpenAI / Ollama / Simple providers. Implementation mirrors OpenAIProvider: lazy dimension detection with a known-models lookup, chunked batch requests, defensive index sort, and a 429-aware retry decorator. In the same change, ProviderRegistry switches from os.getenv to the dynaconf-backed Settings dataclass so all five providers share a single configuration path. config.py gains the previously-uncovered Bedrock keys, the new Mistral keys, the missing OPENAI_GENERATION_MODEL / OLLAMA_GENERATION_MODEL, and SIMPLE_EMBEDDING_DIMENSION. Auto-detection priority: Bedrock → OpenAI → Mistral → Ollama → Simple. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
22ed9e99a0 |
feat(webdav): add tag-based file exclusion (#710)
Hide sensitive files/folders from the WebDAV MCP tool surface by tagging them with a configured Nextcloud system tag. Defence-in-depth control for users who connect LLMs to accounts holding contracts, medical records, credentials, etc. A new EXCLUDED_TAGS env var (comma-separated tag names, empty by default) gates an exclusion layer that runs at the start of every WebDAV tool call: tag names are resolved to tag IDs, those IDs are expanded to the set of tagged paths, then listings/searches are filtered and read/write/delete/move/copy operations on excluded paths raise ToolError. Tagged folders exclude their descendants via prefix match. Empty EXCLUDED_TAGS disables the feature entirely. The threat model is preventing accidental data exfiltration via the LLM tool surface — not hiding files from a determined operator. The docs explicitly recommend creating exclusion tags with user_assignable=false so the credentials the MCP server uses cannot remove the tag. Implementation: - config.py: add `excluded_tags` to _DEFAULTS, Settings, and the _field_map alongside other comma-separated env vars. - client/webdav.py: get_files_by_tag now requests <d:resourcetype/> and surfaces is_directory so tagged directories can recursively exclude descendants. - server/tag_exclusion.py (new): get_excluded_tag_names, get_excluded_file_paths, is_path_excluded. - server/webdav.py: exclusion guards in all 11 WebDAV tools; read/write/create/delete/move/copy raise ToolError, list/search tools silently filter excluded entries. Existing f-string log calls converted to lazy %-style. - tests: 17 new unit tests covering path-matching edge cases (shared-prefix non-match, descendants of excluded dirs), tag-name parsing, and get_excluded_file_paths with mocked WebDAV; 1 new client test asserting <d:resourcetype/> -> is_directory parsing. - docs/configuration.md: new "Tag-Based File Exclusion" section with per-tool effect table, security guidance, and per-call cost note. - README.md: feature mention under Key Features. Closes #710. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
ce80a36877 |
fix(auth): address PR #757 round-3 review feedback
Three review items from the third-round review on PR #757:
- scope_authorization: split the combined logger.warning(error_msg) in
the require_scopes decorator's missing-app-password branch into two
lazy %-style logger calls (one per branch), keeping the f-string
error_msg for the exception only. The else branch also logs the
elicit_result for diagnostics. Bypassing lazy %-interpolation in
security-sensitive code formatted the message regardless of log level
and matched the repo-wide lazy-logging preference; the new code now
conforms.
- config + browser_oauth_routes: wire COOKIE_SECURE through Settings
(cookie_secure: bool | None = None) so _should_use_secure_cookies()
reads it via get_settings() rather than os.getenv. Completes the
consolidation pass that touched this file in commit
|
||
|
Chris CoutinhoandClaude Opus 4.7
|
f3256e515e |
refactor(config): consolidate NEXTCLOUD_PUBLIC_ISSUER_URL through Settings
Lift NEXTCLOUD_PUBLIC_ISSUER_URL out of raw os.getenv reads into Settings.nextcloud_public_issuer_url across all 8 production call sites (app.py x2, oauth_routes.py x2, browser_oauth_routes.py, provision_routes.py, userinfo_routes.py, elicitation.py). cli.py remains the env-write source so the existing config-by-flag pipeline still works. Also addresses remaining PR #757 review nits: - elicitation.py: align URL-present/absent wording on "open in your browser" so users don't try clicking in the terminal - test_scope_authorization_stored.py: lock in the deliberately-shared fall-through branch with explicit declined/cancelled decorator tests - test_elicitation.py: switch from monkeypatch.setenv to patch(get_settings) since Settings is now the canonical surface Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
ffcca23a7b |
refactor(search): address PR #750 round 5 review feedback
Tightens verifier consistency, closes test gaps, hardens the fire-and-forget eviction snapshot, and routes the new concurrency knob through Settings. - Pre-flight ``int()`` guard in ``_verify_notes`` mirrors ``_verify_deck_cards``, so a non-numeric note id produces a type-specific log line instead of falling through to the generic "unexpected error" branch. - Adds explicit 403 tests for the file and news verifiers (symmetry with the existing notes/deck 403 tests) plus a ``non_numeric_id_keeps`` test. - ``AppContext`` and ``OAuthAppContext`` no longer snapshot ``_vector_sync_state.eviction_task_group`` at lifespan-yield time. Both expose it as a ``@property`` that reads the singleton dynamically, removing the order-sensitive race where a future startup-ordering change could silently degrade fire-and-forget eviction to inline forever. - Adds ``verification_concurrency`` (env var ``VERIFICATION_CONCURRENCY``, default 20) to ``Settings`` with a dynaconf validator; ``verify_search_results`` resolves the cap lazily from settings when the caller doesn't override it. - Enriches the news verifier TODO to call out that ``batch_size=-1`` is intentional — a numeric ceiling would silently break correctness because any item beyond the cap would be missing from ``present_ids`` and dropped. - Updates ``Optional[TaskGroup]`` to ``TaskGroup | None`` per project style. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
c1368b9a7f |
refactor(webhooks): bound queue waits, route URLs through dynaconf
Addresses round-3 review feedback on PR #747: - webhook_receiver: wrap send_stream.send() in anyio.fail_after(1.0) and return 503 with reason="queue full" if the queue is saturated. Avoids pinning the handler until NC's outbound timeout fires; the 503 retry contract is the same as the existing "sync not running" branch. - webhook_receiver: revise the compare_digest comment to match what the function actually guarantees — it avoids the per-character short-circuit of `==` but is not fully constant-time across length differences. - _get_webhook_uri: read WEBHOOK_INTERNAL_URL and NEXTCLOUD_MCP_SERVER_URL via dynaconf so operators using settings.toml (rather than env vars) aren't silently routed into the docker/localhost fallback. Adds webhook_internal_url to Settings/_DEFAULTS/_field_map; nextcloud_mcp_server_url already existed. Docker-detection markers stay on os.getenv since they're container-runtime signals, not user-facing config. - webhook_routes: sweep remaining f-string logger calls to lazy %s formatting per CLAUDE.md. - client/webhooks: modernise full file's type hints to dict / list / | None per CLAUDE.md. Tests: - New test_returns_503_when_queue_is_full exercises the timeout branch with a saturated buffer and a shortened deadline. - test_webhook_uri tests now patch get_settings (matching the auth-pair tests in the same file) instead of monkeypatching env vars directly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.7
|
224428fca5 |
fix(webhooks): authenticate deliveries via WEBHOOK_SECRET; review nits
Adds optional shared-secret authentication for /webhooks/nextcloud, addressing the security follow-up flagged in #747. Behavior: - WEBHOOK_SECRET set: registrations pass authMethod="header" with authData={"Authorization": "Bearer <secret>"} (encrypted at-rest in Nextcloud's DB and forwarded on every delivery). The receiver validates the same header with hmac.compare_digest before parsing any payload; missing/invalid → 401. - WEBHOOK_SECRET unset: registrations stay on authMethod="none" and the receiver accepts unauthenticated POSTs (logging a one-time startup warning). Backward compatible — operators can roll out at their own pace. Implementation notes: - WebhooksClient.create_webhook gains an `auth_data` parameter mapped to NC's `authData` body field; this is distinct from the existing `headers` parameter (`headers` is plaintext static request headers, `authData` is encrypted at-rest in NC and only emitted when authMethod="header"). The previous `auth_method="bearer"` mention in the docstring was incorrect — NC supports only "none" and "header". - A small `webhook_auth_pair()` helper in auth/webhook_routes.py centralises the secret→(auth_method, auth_data) resolution so the preset flow and the Astrolabe-facing /api/v1/webhooks endpoint stay in sync. Also addresses the smaller review points from #747: - f-string → lazy %s formatting in webhook_receiver.py and webhook_routes.py. - Move `int(time)` inside webhook_parser's try/except so a malformed `time` field returns None instead of raising ValueError. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.6
|
512de1f6b0 |
test: address PR #707 reviewer feedback on config path helpers
- _resolve_settings_files() now raises FileNotFoundError when NEXTCLOUD_MCP_SETTINGS_FILE points to a missing file, instead of silently falling back to defaults (footgun on typos). - .secrets.toml is now looked for alongside the explicit settings file when NEXTCLOUD_MCP_SETTINGS_FILE is set, matching user expectation for /etc-style deployments. Unset behaviour (cwd lookup) is unchanged. - get_token_db_path() drops the redundant os.environ.get() short-circuit; TOKEN_STORAGE_DB is already bound through dynaconf because the key is declared in _DEFAULTS. - is_ephemeral_token_db() docstring documents the "must call get_token_db_path() first" precondition. - alembic.ini comment clarifies the ./tokens.db placeholder is cwd-relative by design and points readers at the -x database_url escape hatch. - New tests/unit/test_config_paths.py (12 tests) covering the ephemeral tempfile lifecycle, the TOKEN_STORAGE_DB override path, and all six _resolve_settings_files() cases including the two new behaviours. Full unit suite now at 476 passed (464 + 12 new). Ruff + ty clean. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Chris CoutinhoandClaude Opus 4.6
|
146b622ebf |
fix: enable uvx/PyPI deployments without Docker assumptions
Two bugs made `uvx --from . nextcloud-mcp-server run` (and any pip install) unusable outside Docker: 1. Dynaconf was configured with ignore_unknown_envvars=True and relied on settings.toml to declare the key schema. With no settings.toml in a wheel install, every env var (NEXTCLOUD_HOST, MCP_DEPLOYMENT_MODE, ...) was silently dropped. Moved the schema into a Python _DEFAULTS dict passed directly to Dynaconf, kept settings.toml as an optional external override (renamed to settings.toml.example, gitignored), and pointed docker-compose at the example file. 2. Token SQLite DB defaulted to /app/data/tokens.db in multiple places (auth/storage.py, migrations.py, alembic/env.py, cli.py db subcommands), which blew up at uvicorn startup with FileNotFoundError on non-Docker hosts. Replaced with a new config.get_token_db_path() helper that resolves TOKEN_STORAGE_DB if explicitly set, otherwise allocates a per-process tempfile cleaned up at interpreter exit via atexit — mirroring the "ephemeral by default" pattern used for QDRANT_LOCATION=:memory:. Containers are unaffected: docker-compose services now explicitly set TOKEN_STORAGE_DB=/app/data/tokens.db (the fourth service that was missing this pin has been brought in line with the other three). Verified end-to-end in an isolated /tmp venv: env-var-only startup, Alembic migrations run against the tempfile, Application startup complete, /health/live returns 200, tempfile deleted on SIGTERM. Unit tests (464) + ruff + ty pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |