feat(vector): Add configurable chunk size and overlap for document embedding
Enable users to tune document chunking parameters to match their embedding model and content type by adding DOCUMENT_CHUNK_SIZE and DOCUMENT_CHUNK_OVERLAP environment variables. - **config.py**: Added `document_chunk_size` (default: 512) and `document_chunk_overlap` (default: 50) configuration fields with validation: - Ensures overlap < chunk_size - Warns if chunk_size < 100 words - Prevents negative overlap values - **processor.py**: Updated DocumentChunker instantiation to use config settings instead of hardcoded values (line 174-177) - **tests/unit/test_config.py**: Added TestChunkConfigValidation class with 9 tests covering: - Default values - Valid configurations - Validation errors (overlap >= chunk_size, negative overlap) - Warning for small chunk sizes - Environment variable loading - **docs/configuration.md**: Added comprehensive "Document Chunking Configuration" section with: - Chunk size selection guidance (256-384 vs 512 vs 768-1024 words) - Overlap recommendations (10-20% of chunk size) - Configuration examples for different use cases - Added env vars to reference table - **docs/semantic-search-architecture.md**: Added "Document Chunking Strategy" section with: - Chunking process explanation - Example showing sliding window behavior - Search behavior with chunks - Tuning recommendations - **env.sample**: Added complete "Semantic Search & Vector Sync Configuration" section with: - Vector sync settings - Qdrant configuration (3 modes) - Ollama embedding service - Document chunking configuration - **docker-compose.yml**: Added commented examples for DOCUMENT_CHUNK_SIZE and DOCUMENT_CHUNK_OVERLAP with usage notes \`\`\`bash DOCUMENT_CHUNK_SIZE=512 DOCUMENT_CHUNK_OVERLAP=50 \`\`\` 1. \`overlap\` must be less than \`chunk_size\` 2. \`overlap\` cannot be negative 3. Warning issued if \`chunk_size\` < 100 words **Precise matching** (small notes, specific queries): \`\`\`bash DOCUMENT_CHUNK_SIZE=256 DOCUMENT_CHUNK_OVERLAP=25 \`\`\` **Balanced** (default, general purpose): \`\`\`bash DOCUMENT_CHUNK_SIZE=512 DOCUMENT_CHUNK_OVERLAP=50 \`\`\` **Contextual** (long documents, broader topics): \`\`\`bash DOCUMENT_CHUNK_SIZE=1024 DOCUMENT_CHUNK_OVERLAP=100 \`\`\` ✅ **User control** - Tune chunking to match embedding model capabilities ✅ **Experimentation** - Test different chunk sizes for optimal results ✅ **Model alignment** - Match chunk size to embedding context window ✅ **Backward compatible** - Defaults maintain existing behavior ✅ **Well validated** - Comprehensive tests prevent misconfiguration All 22 config validation tests pass (9 new tests for chunking): - Default values work correctly - Validation prevents invalid configurations - Environment variables load properly - Warning system works as expected With configurable chunk sizes, users can now experiment with different Ollama embedding models and tune chunk parameters for optimal semantic search quality. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -177,6 +177,53 @@ Currently only `NotesScanner` is implemented. Future: `CalendarScanner`, `DeckSc
|
||||
- `user_id`: Multi-tenancy filtering (each user's vectors isolated)
|
||||
- `doc_type`: App identifier ("note", "event", "card", etc.)
|
||||
- `etag`: Change detection for incremental updates
|
||||
- `chunk_index`: Position of this chunk within the document (0-indexed)
|
||||
- `total_chunks`: Total number of chunks for this document
|
||||
- `excerpt`: First 200 characters of chunk (for display)
|
||||
|
||||
### Document Chunking Strategy
|
||||
|
||||
Documents are chunked before embedding to handle content larger than the embedding model's context window and to improve search precision.
|
||||
|
||||
**Configuration:**
|
||||
```dotenv
|
||||
DOCUMENT_CHUNK_SIZE=512 # Words per chunk (default)
|
||||
DOCUMENT_CHUNK_OVERLAP=50 # Overlapping words between chunks (default)
|
||||
```
|
||||
|
||||
**Chunking Process:**
|
||||
1. **Text combination**: Document title + content (e.g., `"Note Title\n\nNote content..."`)
|
||||
2. **Word-based splitting**: Simple whitespace tokenization
|
||||
3. **Sliding window**: Create overlapping chunks
|
||||
4. **Individual embedding**: Each chunk gets its own vector
|
||||
5. **Separate storage**: Each chunk stored as distinct point in Qdrant
|
||||
|
||||
**Example:**
|
||||
```
|
||||
Document (1000 words):
|
||||
→ Chunk 0: words 0-511
|
||||
→ Chunk 1: words 462-973 (overlaps by 50 words)
|
||||
→ Chunk 2: words 924-999 (last chunk, partial)
|
||||
|
||||
Each chunk stored as separate vector with metadata:
|
||||
- chunk_index: 0, 1, 2
|
||||
- total_chunks: 3
|
||||
- excerpt: First 200 chars of each chunk
|
||||
```
|
||||
|
||||
**Search Behavior:**
|
||||
- **Vector search** operates on chunks (not whole documents)
|
||||
- **Deduplication** collapses multiple matching chunks from same document
|
||||
- **Best match** returns highest-scoring chunk's excerpt
|
||||
- **Access verification** still performed at document level
|
||||
|
||||
**Tuning Recommendations:**
|
||||
- **Small chunks (256-384 words)**: More precise, less context, more storage
|
||||
- **Large chunks (768-1024 words)**: More context, less precise, less storage
|
||||
- **Overlap (10-20% of chunk size)**: Preserves context across boundaries
|
||||
- **Match to embedding model**: Consider model's context window when sizing
|
||||
|
||||
**Important**: Changing chunk size requires re-embedding all documents. Use the collection naming strategy to manage different chunking configurations.
|
||||
|
||||
### Collection Naming and Model Switching
|
||||
|
||||
|
||||
Reference in New Issue
Block a user