Chris CoutinhoandClaude Opus 4.8
6ef7786cec
Fixes MCP reconnect timeouts on tenant servers (Deck #302). Three changes: - /health/ready now gates only on local config. Nextcloud/Qdrant health is refreshed by a background loop, cached, and reported but NON-gating, so a single-replica tenant Pod is no longer pulled from its Service on a transient dependency blip (which dropped every MCP streamable-HTTP session and caused reconnect timeouts). The probe path performs no external I/O. - Refactor starlette_lifespan: collapse the four near-identical per-mode task-group + session + yield + teardown skeletons into one shared task group that also runs the readiness refresh loop; each mode contributes a (start, teardown) pair. eviction_task_group is now always present. - Migrate app.py off os.getenv: all config is read through dynaconf Settings (adds health_ready_refresh_interval, oidc_token_type, oidc_scopes, port). Inline/dynamic defaults preserved at each call site. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
76 lines
2.7 KiB
Python
76 lines
2.7 KiB
Python
"""Non-blocking readiness dependency-health cache.
|
|
|
|
Kubernetes readiness probes must be cheap and must not gate a (typically
|
|
single-replica) tenant Pod out of its Service on transient external-dependency
|
|
latency. Doing so converts a *degraded* shared dependency (Nextcloud, Qdrant)
|
|
into a *total* outage: the only Pod is removed from the Service, the gateway
|
|
has no upstream, and connected MCP clients see their streamable-HTTP sessions
|
|
drop and fail to reconnect (Deck #302).
|
|
|
|
A background loop refreshes this snapshot off the probe path; the readiness
|
|
handler only ever reads ``snapshot()`` (no I/O), so probe latency is decoupled
|
|
from upstream latency. Dependency results are reported for observability but are
|
|
intentionally *non-gating*.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import time
|
|
|
|
from pydantic import BaseModel, Field
|
|
|
|
|
|
class DependencyStatus(BaseModel):
|
|
"""Last observed health of a single external dependency.
|
|
|
|
``healthy`` is ``None`` until the first check completes; ``detail`` carries
|
|
the human-readable string the readiness handler reports verbatim
|
|
(``"ok"`` / ``"embedded"`` / ``"pending"`` / ``"error: ..."``).
|
|
"""
|
|
|
|
name: str
|
|
healthy: bool | None = None
|
|
detail: str = "pending"
|
|
checked_at: float = 0.0
|
|
|
|
|
|
class ReadinessCache(BaseModel):
|
|
"""Time-bounded snapshot of external dependency health.
|
|
|
|
Written only by the background refresh loop and read only by the readiness
|
|
handler. No locking: assignment into ``statuses`` is atomic under the GIL,
|
|
and a reader tolerates seeing the previous value for one entry.
|
|
"""
|
|
|
|
ttl_seconds: float = 30.0
|
|
statuses: dict[str, DependencyStatus] = Field(default_factory=dict)
|
|
|
|
def snapshot(self) -> dict[str, DependencyStatus]:
|
|
"""Return a shallow copy of the current per-dependency statuses."""
|
|
return dict(self.statuses)
|
|
|
|
def update(
|
|
self, name: str, healthy: bool, detail: str, *, now: float | None = None
|
|
) -> None:
|
|
"""Record the outcome of a dependency check."""
|
|
self.statuses[name] = DependencyStatus(
|
|
name=name,
|
|
healthy=healthy,
|
|
detail=detail,
|
|
checked_at=time.monotonic() if now is None else now,
|
|
)
|
|
|
|
def is_stale(self, *, now: float | None = None) -> bool:
|
|
"""True when there is no data yet or any entry is older than the TTL.
|
|
|
|
Exposed for observability/diagnostics; the refresh loop runs on a fixed
|
|
cadence rather than polling this.
|
|
"""
|
|
if not self.statuses:
|
|
return True
|
|
current = time.monotonic() if now is None else now
|
|
return any(
|
|
current - status.checked_at >= self.ttl_seconds
|
|
for status in self.statuses.values()
|
|
)
|