mcp-nextcloud/docs/ADR-024-dynaconf-configuration-management.md

ADR-024: Dynaconf Configuration Management

Status: Proposed Date: 2026-04-04 Deciders: Development Team Related: ADR-020 (Deployment Modes), ADR-021 (Configuration Consolidation), ADR-022 (Login Flow v2)

Context

The nextcloud-mcp-server configuration system has grown to ~80+ environment variables across five deployment modes. All configuration is loaded via manual os.getenv() calls in config.py (~60 calls in get_settings() alone) and providers/registry.py. This creates several problems:

Problems Identified

1. No file-based configuration option: Every deployment requires setting environment variables. For complex deployments with 20+ variables (e.g., Keycloak + semantic search + observability), this is unwieldy and error-prone. There is no way to ship a "configuration profile" as a file.

2. Configuration sprawl across multiple locations: Environment variables are read in at least three places:

   • config.py:get_settings() — Main settings (~60 vars)
   • config.py:get_document_processor_config() — Document processing (~20 vars)
   • providers/registry.py:ProviderRegistry.create_provider() — Embedding providers (~15 vars)

3. No configuration file for local development: Developers must either maintain a .env file and export $(grep -v '^#' .env | xargs), or rely solely on docker-compose environment blocks. A structured settings file with defaults per deployment mode would simplify onboarding.

4. Manual type coercion is repetitive and error-prone: The codebase is littered with patterns like:

   os.getenv("SOME_BOOL", "false").lower() == "true"
   int(os.getenv("SOME_INT", "300"))
   float(os.getenv("SOME_FLOAT", "1.0"))
   

   Each is a potential ValueError if a user provides a non-numeric string for an integer field.

5. No structured validation at load time: While config_validators.py validates mode requirements after loading, there is no validation of individual field types, ranges, or mutual exclusivity at parse time. Invalid values (e.g., METRICS_PORT=abc) only fail when first used.

6. Secrets mixed with configuration: TOKEN_ENCRYPTION_KEY, NEXTCLOUD_PASSWORD, OPENAI_API_KEY, and other secrets are treated identically to non-sensitive configuration, with no separation mechanism.

Current Configuration Surface

Category	Approx. Vars	Example
Core Nextcloud	6	NEXTCLOUD_HOST, NEXTCLOUD_USERNAME, NEXTCLOUD_VERIFY_SSL
OAuth/OIDC	12	OIDC_DISCOVERY_URL, NEXTCLOUD_OIDC_CLIENT_ID, JWKS_URI
Mode Selection	4	MCP_DEPLOYMENT_MODE, ENABLE_LOGIN_FLOW, ENABLE_TOKEN_EXCHANGE
Token Storage	3	TOKEN_ENCRYPTION_KEY, TOKEN_STORAGE_DB
Semantic Search	6	ENABLE_SEMANTIC_SEARCH, VECTOR_SYNC_SCAN_INTERVAL
Qdrant	4	QDRANT_URL, QDRANT_LOCATION, QDRANT_API_KEY
Embedding Providers	15	OLLAMA_BASE_URL, OPENAI_API_KEY, BEDROCK_*
Document Processing	18	ENABLE_UNSTRUCTURED, TESSERACT_CMD, PYMUPDF_*
Observability	10	OTEL_EXPORTER_OTLP_ENDPOINT, LOG_FORMAT, METRICS_PORT
Webhooks/Internal	4	WEBHOOK_INTERNAL_URL, NEXTCLOUD_MCP_SERVICE_NAME
Total	~82

Decision

Adopt dynaconf as the configuration management layer, enabling TOML file-based configuration alongside existing environment variable support.

Why Dynaconf

Criterion	Dynaconf	Pydantic Settings	python-dotenv
File-based config (TOML/YAML)	Yes (native)	No (needs extra)	.env only
Environment sections/profiles	Yes ([default], [production])	No	No
Env var override (12-factor)	Yes (built-in, highest priority)	Yes	Yes
Type coercion	Automatic (TOML parser)	Via type hints	No
Validators	Declarative + conditional	Via Pydantic	No
Secrets file separation	Yes (.secrets.toml)	No built-in	Separate .env
Local overrides	Yes (settings.local.toml auto-loaded)	No	No
Zero-prefix env vars	Yes (envvar_prefix=False)	Custom	N/A
Dependency	Pure Python, well-maintained	Pydantic (already in project for models)	Minimal

Architecture

1. Dynaconf Instance Configuration

# nextcloud_mcp_server/config.py
from pathlib import Path
from dynaconf import Dynaconf, Validator

settings = Dynaconf(
    settings_files=["settings.toml", ".secrets.toml"],
    root_path=Path(__file__).parent.parent,
    environments=True,
    env_switcher="MCP_DEPLOYMENT_MODE",
    envvar_prefix=False,
    load_dotenv=False,
    ignore_unknown_envvars=True,
    post_hooks=[handle_deprecations, resolve_dependencies],
    validators=[...],  # See Section 4
)

Key choices:

• envvar_prefix=False: Existing env vars (NEXTCLOUD_HOST, ENABLE_SEMANTIC_SEARCH, etc.) work without any prefix. No renaming required.
• env_switcher="MCP_DEPLOYMENT_MODE": Reuses the existing ADR-021 variable. Setting MCP_DEPLOYMENT_MODE=single_user_basic loads the [single_user_basic] TOML section on top of [default]. Note: dynaconf's environments feature is designed for lifecycle environments (dev/staging/prod), but custom environment names are a supported pattern — see tests_functional/legacy/simple_ini_example/ in the dynaconf repo for a precedent using environments=["ansible", "puppet"].
• ignore_unknown_envvars=True: Only env vars matching keys defined in settings.toml or defaults are loaded. System env vars (HOME, PATH, LANG) are ignored.
• root_path=Path(__file__).parent.parent: Anchors settings file lookup to the project root regardless of working directory. This ensures consistent behavior whether running via uv run from the repo root, inside a container with WORKDIR /app, or during test execution.
• post_hooks=[...]: Deprecation remapping and dependency resolution run after all sources are loaded (see Sections 5 and 6).
• load_dotenv=False: We don't auto-load .env files to avoid surprising behavior. Users who want dotenv can use direnv or shell-level loading.

2. Settings File Structure

settings.toml — Shipped with the project, checked into git:

[default]
# === Nextcloud Connection ===
# nextcloud_host — Required, set via env var or .secrets.toml. No default.
nextcloud_verify_ssl = true
nextcloud_ca_bundle = "@none"

# === Deployment Mode ===
# Auto-detected if not set. Valid: single_user_basic, multi_user_basic,
# oauth_single_audience, login_flow, keycloak
# mcp_deployment_mode = ""

# === Authentication Toggles ===
enable_multi_user_basic_auth = false
enable_login_flow = false
enable_token_exchange = false

# === Token Storage ===
token_storage_db = "/tmp/tokens.db"

# === Semantic Search ===
enable_semantic_search = false
vector_sync_scan_interval = 300
vector_sync_processor_workers = 3
vector_sync_queue_max_size = 10000
vector_sync_user_poll_interval = 60

# === Qdrant ===
qdrant_location = ":memory:"
qdrant_collection = "nextcloud_content"

# === Embedding Providers ===
ollama_embedding_model = "nomic-embed-text"
ollama_verify_ssl = true
openai_embedding_model = "text-embedding-3-small"

# === Document Chunking ===
document_chunk_size = 2048
document_chunk_overlap = 200

# === Document Processing ===
enable_document_processing = false
document_processor = "unstructured"
enable_pymupdf = true
pymupdf_extract_images = true
enable_unstructured = false
unstructured_api_url = "http://unstructured:8000"
unstructured_timeout = 120
unstructured_strategy = "auto"
unstructured_languages = "eng,deu"
enable_tesseract = false
tesseract_lang = "eng"
enable_custom_processor = false
custom_processor_name = "custom"
custom_processor_types = "application/pdf"
custom_processor_timeout = 60

# === Observability ===
metrics_enabled = true
metrics_port = 9090
otel_service_name = "nextcloud-mcp-server"
otel_traces_sampler = "always_on"
otel_traces_sampler_arg = 1.0
otel_exporter_verify_ssl = false
log_format = "text"
log_level = "INFO"
log_include_trace_context = true

# === Webhooks ===
nextcloud_mcp_service_name = "mcp"
nextcloud_mcp_port = 8000

# ─────────────────────────────────────────────
# Deployment Mode Overrides
# ─────────────────────────────────────────────

[single_user_basic]
# Credentials provided via env vars or .secrets.toml
# nextcloud_username = ""  (in .secrets.toml)
# nextcloud_password = ""  (in .secrets.toml)

[multi_user_basic]
enable_multi_user_basic_auth = true
token_storage_db = "/app/data/tokens.db"

[login_flow]
enable_login_flow = true
token_storage_db = "/app/data/tokens.db"

[keycloak]
enable_token_exchange = true
token_storage_db = "/app/data/tokens.db"
token_exchange_cache_ttl = 300

[oauth_single_audience]
token_storage_db = "/app/data/tokens.db"

.secrets.toml.example — Template, checked into git (actual .secrets.toml is gitignored):

[default]
# token_encryption_key = ""

[single_user_basic]
# nextcloud_username = ""
# nextcloud_password = ""
# nextcloud_app_password = ""

[keycloak]
# nextcloud_oidc_client_id = ""
# nextcloud_oidc_client_secret = ""
# token_encryption_key = ""

[login_flow]
# token_encryption_key = ""

settings.local.toml — Personal overrides, gitignored, auto-loaded by dynaconf:

# Example developer overrides
[default]
log_level = "DEBUG"
ollama_base_url = "http://localhost:11434"

3. Configuration Loading Priority

Dynaconf merges configuration in this order (last wins):

1. settings.toml [default] section          ← base defaults
2. settings.toml [<mode>] section           ← mode-specific overrides
3. .secrets.toml [default] section          ← base secrets
4. .secrets.toml [<mode>] section           ← mode-specific secrets
5. settings.local.toml (all sections)       ← developer overrides
6. Environment variables                    ← highest priority (12-factor)

This means:

• File-based config is optional — env vars alone still work (they override everything)
• Mode-specific defaults reduce boilerplate — [login_flow] sets enable_login_flow=true and token_storage_db=/app/data/tokens.db so deployers don't need to
• Secrets are separated — .secrets.toml holds TOKEN_ENCRYPTION_KEY, passwords, API keys
• Local dev overrides don't pollute — settings.local.toml is gitignored

4. Dynaconf Validators

Replace repetitive __post_init__ checks with declarative validators:

validators = [
    # Required for all modes
    Validator("NEXTCLOUD_HOST", must_exist=True, when=Validator("MCP_DEPLOYMENT_MODE", ne="")),

    # Type and range validation
    Validator("METRICS_PORT", gte=1, lte=65535),
    Validator("VECTOR_SYNC_SCAN_INTERVAL", gte=1),
    Validator("VECTOR_SYNC_PROCESSOR_WORKERS", gte=1),
    Validator("DOCUMENT_CHUNK_SIZE", gte=128),
    Validator("DOCUMENT_CHUNK_OVERLAP", gte=0),
    Validator("OTEL_TRACES_SAMPLER_ARG", gte=0.0, lte=1.0),

    # Enum validation
    Validator("LOG_FORMAT", is_in=["text", "json"]),
    Validator("LOG_LEVEL", is_in=["DEBUG", "INFO", "WARNING", "ERROR"]),
    Validator("OTEL_TRACES_SAMPLER", is_in=["always_on", "always_off", "parentbased_always_on", "parentbased_always_off", "traceidratio", "parentbased_traceidratio"]),

    # Mutual exclusivity: QDRANT_URL and non-default QDRANT_LOCATION cannot both be set.
    # QDRANT_LOCATION defaults to ":memory:", so check for non-default values.
    Validator("QDRANT_URL", must_exist=False, when=Validator("QDRANT_LOCATION", ne=":memory:")),
]

5. Backward Compatibility: Deprecation Hooks

Deprecated env var names (VECTOR_SYNC_ENABLED, ENABLE_OFFLINE_ACCESS) are handled via a post-hook that runs after all sources are loaded. Hooks are registered via Dynaconf(post_hooks=[...]) (see Section 1):

def handle_deprecations(settings):
    """Map deprecated variable names to current names (ADR-021 compatibility)."""
    # VECTOR_SYNC_ENABLED -> ENABLE_SEMANTIC_SEARCH
    if settings.exists("VECTOR_SYNC_ENABLED") and not settings.exists("ENABLE_SEMANTIC_SEARCH"):
        settings.set("ENABLE_SEMANTIC_SEARCH", settings.VECTOR_SYNC_ENABLED)
        logger.warning("VECTOR_SYNC_ENABLED is deprecated. Use ENABLE_SEMANTIC_SEARCH instead.")

    # ENABLE_OFFLINE_ACCESS -> ENABLE_BACKGROUND_OPERATIONS
    if settings.exists("ENABLE_OFFLINE_ACCESS") and not settings.exists("ENABLE_BACKGROUND_OPERATIONS"):
        settings.set("ENABLE_BACKGROUND_OPERATIONS", settings.ENABLE_OFFLINE_ACCESS)
        logger.warning("ENABLE_OFFLINE_ACCESS is deprecated. Use ENABLE_BACKGROUND_OPERATIONS instead.")

6. Smart Dependency Resolution

The auto-enablement of ENABLE_BACKGROUND_OPERATIONS when semantic search is active in multi-user modes (existing behavior from ADR-021) is preserved as a post-hook:

def resolve_dependencies(settings):
    """Auto-enable background operations for semantic search in multi-user modes."""
    is_multi_user = (
        settings.get("ENABLE_MULTI_USER_BASIC_AUTH", False)
        or settings.get("ENABLE_TOKEN_EXCHANGE", False)
        or (not settings.get("NEXTCLOUD_USERNAME") and not settings.get("NEXTCLOUD_PASSWORD"))
    )

    if settings.get("ENABLE_SEMANTIC_SEARCH", False) and is_multi_user:
        if not settings.get("ENABLE_BACKGROUND_OPERATIONS", False):
            settings.set("ENABLE_BACKGROUND_OPERATIONS", True)
            logger.info("Auto-enabled background operations for semantic search in multi-user mode.")

7. Adapter Layer (Migration Bridge)

During migration, get_settings() continues to return the Settings dataclass, populated from dynaconf:

from dynaconf import Dynaconf

_dynaconf = Dynaconf(...)  # As configured above

def get_settings() -> Settings:
    """Get application settings — backed by dynaconf."""
    return Settings(
        deployment_mode=_dynaconf.get("MCP_DEPLOYMENT_MODE"),
        nextcloud_host=_dynaconf.get("NEXTCLOUD_HOST"),
        nextcloud_username=_dynaconf.get("NEXTCLOUD_USERNAME"),
        # ... all fields populated from _dynaconf.get() instead of os.getenv()
    )

This is a zero-risk change: every consumer of get_settings() sees the same Settings type. The dataclass can be removed in a later phase once all consumers migrate to _dynaconf directly.

8. Mode Detection Preserved

config_validators.py is unchanged in this phase. detect_auth_mode() and validate_configuration() continue to operate on the Settings dataclass. The business logic for mode detection, conditional requirements, and forbidden variables is too complex for declarative validators and benefits from remaining as explicit Python code.

9. Document Processor Config Integration

get_document_processor_config() currently reads ~20 env vars independently. It will be migrated to read from the same dynaconf instance, with document processor settings nested under the [default] section alongside all other settings.

10. Provider Registry

providers/registry.py:ProviderRegistry.create_provider() reads ~15 env vars directly. It will be updated to accept a settings object or read from the dynaconf instance, consolidating all configuration into a single source.

11. Test Isolation

Tests must not be affected by settings.toml or .secrets.toml being present in the repository. The test configuration strategy:

# conftest.py
import pytest

@pytest.fixture(autouse=True)
def isolated_settings(tmp_path, monkeypatch):
    """Ensure tests use a clean dynaconf instance with no file-based config."""
    monkeypatch.setenv("SETTINGS_FILE_FOR_DYNACONF", str(tmp_path / "empty.toml"))
    (tmp_path / "empty.toml").write_text("[default]\n")
    # Reset the dynaconf instance to pick up the override
    from nextcloud_mcp_server.config import _dynaconf
    _dynaconf.reload()

Tests that need specific configuration values continue to use monkeypatch.setenv() as today, which will override any file-based defaults (env vars have highest priority in dynaconf).

Docker Compose Impact

Zero breaking changes. All existing environment: blocks in docker-compose.yml continue to work because envvar_prefix=False means env vars map directly to setting keys.

Optional enhancement: Users can mount settings files for cleaner configuration:

mcp:
  volumes:
    - ./settings.toml:/app/settings.toml:ro
    - ./.secrets.toml:/app/.secrets.toml:ro
  environment:
    # Only override what differs from settings.toml
    - MCP_DEPLOYMENT_MODE=single_user_basic
    - LOG_LEVEL=DEBUG

Migration Strategy

Phase 1: Add Dynaconf Foundation

• Add dynaconf dependency to pyproject.toml
• Create settings.toml with [default] values matching current defaults
• Create .secrets.toml.example template
• Add .secrets.toml and settings.local.toml to .gitignore
• Initialize Dynaconf instance in config.py

Phase 2: Wire Adapter

• Replace os.getenv() calls in get_settings() with _dynaconf.get() calls
• Replace os.getenv() calls in get_document_processor_config() similarly
• Settings dataclass and all consumers unchanged
• All tests pass without modification

Phase 3: Add Validators

• Add dynaconf Validator instances for type checking, range validation, and enum constraints
• Remove corresponding manual checks from Settings.__post_init__

Phase 4: Deprecation and Dependency Hooks (Optional, Future)

• Move _get_semantic_search_enabled(), _get_background_operations_enabled(), and _is_multi_user_mode() logic into dynaconf post-hooks
• Remove standalone helper functions
• Risk note: These functions contain nuanced multi-variable logic (e.g., the ENABLE_SEMANTIC_SEARCH + VECTOR_SYNC_ENABLED OR pattern, the username/password presence check for mode detection). Running them as dynaconf post-hooks changes their execution context and ordering guarantees relative to config_validators.py. This phase should only proceed after Phases 1-3 are stable and well-tested.

Phase 5: Direct Dynaconf Access (Optional, Future)

• Gradually replace get_settings().field with settings.FIELD in consumers
• Remove Settings dataclass once all consumers migrated
• This is a larger refactor touching ~30 files and can be deferred

Phase 6: Provider Registry Consolidation (Optional, Future)

• Update ProviderRegistry.create_provider() to read from dynaconf
• Eliminates the last pocket of direct os.getenv() calls

Consequences

Positive

• File-based configuration enables shipping deployment profiles, reducing per-deployment env var count from 15-25 to 1-3 overrides
• Automatic type coercion eliminates ~30 manual int(), float(), .lower() == "true" patterns and their potential ValueError exceptions
• Declarative validation catches invalid configuration at startup with clear error messages
• Secret separation via .secrets.toml provides a standard pattern for credential management
• Local overrides via settings.local.toml simplify developer workflows without polluting git
• 12-factor compliant — env vars always win, files are optional
• Zero breaking changes in Phases 1-3. Phase 4 is optional and carries moderate risk due to complex multi-variable logic.

Negative

• New dependency — dynaconf is a runtime dependency (~50KB, pure Python, well-maintained)
• Two configuration systems during migration — Phases 1-3 run dynaconf alongside the existing Settings dataclass
• Learning curve — Contributors must understand dynaconf's merge semantics and environment sections
• envvar_prefix=False risk — Without a prefix, any env var matching a setting key is loaded. Mitigated by ignore_unknown_envvars=True which restricts to pre-defined keys only

Neutral

• config_validators.py unchanged — Mode detection and conditional validation remain as Python business logic. Dynaconf validators handle structural checks only.
• Docker Compose files unchanged — Existing environment: blocks work as-is. File mounting is optional.

Alternatives Considered

1. Pydantic Settings

Pydantic v2's BaseSettings provides type validation and env var loading. However, it lacks native file-based configuration (TOML sections, environment switching, secrets files), which is the primary motivation for this change. While Pydantic v2 is already used in the project for response models (nextcloud_mcp_server/models/), Pydantic Settings still lacks native TOML sections, environment switching, and secrets file separation.

2. python-decouple

Supports .env and .ini files with type casting. Lacks environment sections, validators, secrets separation, and TOML support. Too limited for our needs.

3. Custom TOML Loader

Build a minimal TOML loader using tomllib (stdlib in Python 3.11+). This avoids a dependency but requires implementing validation, env var override, secrets separation, and environment switching from scratch — essentially rebuilding dynaconf.

4. Status Quo (Env Vars Only)

Continue with os.getenv(). Acceptable for small projects, but with 80+ variables across 5 deployment modes, the lack of file-based configuration, validation, and defaults per mode is a growing maintenance burden.

References

• Dynaconf Documentation
• 12-Factor App: Config
• ADR-020: Deployment Modes and Configuration Validation
• ADR-021: Configuration Consolidation and Simplification
• ADR-022: Login Flow v2