feat(observability): astrolabe_* metrics + traces for the document pipeline

Make per-tier bottlenecks in the document-processing pipeline
(scan -> fetch -> parse -> chunk -> embed -> Qdrant upsert) visible via
metrics, traces, and structured logs. Today the document_processors layer
emits only a logger.info line: no metric, no span, and page counts live only
inside a log string. The single processing-duration histogram is unlabeled and
whole-document, so it cannot isolate parse vs embed vs upsert.

New astrolabe_* metric family (distinct from the mcp_* protocol metrics):
- astrolabe_document_parse_{duration_seconds,total} + pages/chars/bytes counters
  recorded at the ProcessorRegistry.process() boundary (covers all current and
  future processors uniformly)
- astrolabe_document_escalation_total (dormant; tiered-pipeline readiness)
- astrolabe_embedding_{duration_seconds,requests_total,chunks_total,chars_total}
- astrolabe_document_chunks_total, astrolabe_documents_indexed_total{source,status}

Tracing: new document_processor.parse child span + enriched embed/chunk span
attributes (provider/model/batch_size/chunk_count). Structured logs gain a
consistent field vocabulary (doc_id, doc_type, processor, tier, pages, chars,
byte_size, chunks, duration_ms, status) so Loki can aggregate without regex.

Tier-readiness: processor/tier are labels from day one and a tier property is
added to DocumentProcessor, so adding docling/OCR/LLM tiers later is additive
(new label values, never new metrics). Tenant comes from the kube namespace
label; mime_type/model are span attributes only (cardinality). Existing
mcp_vector_sync_*/mcp_qdrant_* are left untouched.

Refs Deck #175 (superset of #173 Phase 2). Dashboard/recording-rules follow-up
tracked on #175 for homelab-argocd.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

nextcloud_mcp_server/document_processors/base.py

@@ -56,6 +56,19 @@ class DocumentProcessor(ABC):
         """Unique identifier for this processor (e.g., 'unstructured', 'tesseract')."""
         pass
 
+    @property
+    def tier(self) -> str:
+        """Extraction tier this processor belongs to (escalation ladder).
+
+        Used as the ``tier`` label/attribute in observability so that adding new
+        extraction tiers later (docling, OCR, LLM) is purely additive. Vocabulary
+        (cheapest first): ``fast`` -> ``structured`` -> ``ocr`` -> ``llm``.
+
+        Defaults to ``"fast"``; override in processors that belong to a higher
+        tier.
+        """
+        return "fast"
+
     @property
     @abstractmethod
     def supported_mime_types(self) -> set[str]:

nextcloud_mcp_server/document_processors/pymupdf.py

@@ -181,6 +181,14 @@ class PyMuPDFProcessor(DocumentProcessor):
                 metadata["page_count"],
                 len(md_text),
                 metadata.get("image_count", 0),
+                extra={
+                    "processor": self.name,
+                    "tier": self.tier,
+                    "pages": metadata["page_count"],
+                    "chars": len(md_text),
+                    "images": metadata.get("image_count", 0),
+                    "byte_size": len(content),
+                },
             )
 
             return ProcessingResult(

nextcloud_mcp_server/document_processors/registry.py

@@ -1,9 +1,13 @@
 """Central registry for document processors."""
 
 import logging
+import time
 from collections.abc import Awaitable, Callable
 from typing import Any, Optional
 
+from nextcloud_mcp_server.observability.metrics import record_document_parse
+from nextcloud_mcp_server.observability.tracing import trace_operation
+
 from .base import DocumentProcessor, ProcessingResult, ProcessorError
 
 logger = logging.getLogger(__name__)
@@ -152,13 +156,86 @@ class ProcessorRegistry:
                     f"Registered processors: {', '.join(self.list_processors())}"
                 )
 
-        logger.info("Processing with '%s' processor", processor.name)
-
-        # Process
-        return await processor.process(
-            content, content_type, filename, options, progress_callback
+        tier = processor.tier
+        logger.info(
+            "Processing with '%s' processor",
+            processor.name,
+            extra={
+                "processor": processor.name,
+                "tier": tier,
+                "mime_type": content_type,
+            },
         )
 
+        # Process (instrumented: per-processor span + parse metrics).
+        # NOTE: when the tiered pipeline (docling/OCR/LLM) lands, escalation
+        # decisions are recorded here via record_document_escalation() and an
+        # add_span_event("document.escalation", ...) -- the escalated=False
+        # attribute and the metric are wired ahead of that.
+        byte_size = len(content)
+        start_time = time.time()
+        with trace_operation(
+            "document_processor.parse",
+            attributes={
+                "processor.name": processor.name,
+                "processor.tier": tier,
+                "mime_type": content_type,
+                "byte_size": byte_size,
+                "escalated": False,
+            },
+        ) as span:
+            try:
+                result = await processor.process(
+                    content, content_type, filename, options, progress_callback
+                )
+            except Exception:
+                duration = time.time() - start_time
+                record_document_parse(
+                    processor.name,
+                    tier,
+                    duration,
+                    byte_size=byte_size,
+                    status="error",
+                )
+                raise
+
+            duration = time.time() - start_time
+            pages = int(result.metadata.get("page_count", 0) or 0)
+            chars = len(result.text)
+            status = "success" if result.success else "error"
+            record_document_parse(
+                processor.name,
+                tier,
+                duration,
+                pages=pages,
+                chars=chars,
+                byte_size=byte_size,
+                status=status,
+            )
+            if span is not None:
+                span.set_attribute("page_count", pages)
+                span.set_attribute("char_count", chars)
+                span.set_attribute("processor.success", result.success)
+
+            logger.info(
+                "Parsed %s with '%s': %s pages, %s chars in %.2fs",
+                filename or "<bytes>",
+                processor.name,
+                pages,
+                chars,
+                duration,
+                extra={
+                    "processor": processor.name,
+                    "tier": tier,
+                    "pages": pages,
+                    "chars": chars,
+                    "byte_size": byte_size,
+                    "duration_ms": round(duration * 1000, 1),
+                    "status": status,
+                },
+            )
+            return result
+
 
 # Global registry instance
 _registry = ProcessorRegistry()