fix: address PR #836 round-2 review (connect/timeout/observability)
🟡 Document why ProcrastinateTaskProducer.connect() uses `await app.open_async()` (AwaitableContext: await opens a long-lived pool, closed by drain()) and add a connect()/drain() lifecycle unit test (InMemoryConnector) asserting the pool is opened by connect and closed by drain — previously untested. 🟡 get_procrastinate_conninfo: forward connect_timeout from DATABASE_URL or default 10s so an unreachable DB can't hang worker/API startup indefinitely; warn only on other dropped query params. + tests. 🟢 INGEST_DELETE_SUCCEEDED_JOBS (default true) makes the worker's succeeded-job deletion configurable for audit retention. 🟢 Worker startup logs via logger.info (structured/OTel) instead of click.echo. 🟢 INGEST_STALLED_JOB_SECONDS (default 300) makes the crash-reclaim threshold tunable for slow embedding backends; reclaim reads it per-run. The broad `except` in _apply_ingest_queue_schema_open is kept deliberately: procrastinate wraps psycopg errors, so narrowing to psycopg.errors.* would miss the wrapped DDL-conflict and turn a benign concurrent-apply race into a failure; the presence re-check re-raises genuine errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
cfdef3c2c5
commit
820b98dac1
@@ -1,3 +1,4 @@
|
||||
import logging
|
||||
import os
|
||||
from importlib.metadata import version
|
||||
|
||||
@@ -21,6 +22,8 @@ from nextcloud_mcp_server.server import AVAILABLE_APPS
|
||||
|
||||
from .app import get_app
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
@@ -332,16 +335,24 @@ def worker(concurrency: int | None):
|
||||
# open/close cycle on startup.
|
||||
async with app.open_async():
|
||||
await apply_ingest_queue_schema(app, manage_connection=False)
|
||||
click.echo(
|
||||
f"Ingest worker started: queue={INGEST_QUEUE_NAME} concurrency={workers}"
|
||||
# Structured log (not click.echo) so it lands in the JSON / OTel
|
||||
# pipeline like every other startup message.
|
||||
logger.info(
|
||||
"Ingest worker started: queue=%s concurrency=%s delete_succeeded=%s",
|
||||
INGEST_QUEUE_NAME,
|
||||
workers,
|
||||
settings.ingest_delete_succeeded_jobs,
|
||||
)
|
||||
await app.run_worker_async(
|
||||
queues=[INGEST_QUEUE_NAME],
|
||||
concurrency=workers,
|
||||
install_signal_handlers=True,
|
||||
# Drop succeeded jobs so the queue table stays lean and the KEDA
|
||||
# queue-depth metric reflects only outstanding work.
|
||||
delete_jobs="successful",
|
||||
# Drop succeeded jobs (default) so the queue table stays lean and
|
||||
# the KEDA queue-depth metric reflects only outstanding work; set
|
||||
# INGEST_DELETE_SUCCEEDED_JOBS=false to retain them for audit.
|
||||
delete_jobs="successful"
|
||||
if settings.ingest_delete_succeeded_jobs
|
||||
else "never",
|
||||
)
|
||||
|
||||
anyio.run(_run)
|
||||
|
||||
Reference in New Issue
Block a user