Chris CoutinhoandClaude
fffe483c02
Previously, pymupdf4llm.to_markdown() was called twice - once in PyMuPDFProcessor during indexing and again in PDFHighlighter during visualization. Different image path lengths caused different character offsets, leading to highlighted pages not matching their chunks. Also fixed issue where all chunks on the same page showed all highlights instead of just their own highlight. Now restores original page contents between chunks using xref stream caching. Changes: - Add PDFHighlighter class requiring pre-computed page_boundaries and full_text from document processor (no fallback extraction) - Pass pre-computed data from processor to highlighter - Extract page-relative portion of chunk text for cross-page chunks - Add bounding box highlighting using text anchor search - Run highlight generation in parallel with embedding/BM25 - Cache and restore page contents to isolate highlights per chunk Results: Highlighting success rate improved from 51% to 95% (121/128). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
181 lines
11 KiB
HTML
181 lines
11 KiB
HTML
<div x-data="vizApp()">
|
|
<div class="viz-layout">
|
|
<!-- Top: Search Controls -->
|
|
<div class="viz-card viz-controls-card">
|
|
<form @submit.prevent="executeSearch">
|
|
<div class="viz-controls-grid">
|
|
<div class="viz-control-group">
|
|
<label>Search Query</label>
|
|
<input type="text" x-model="query" placeholder="Enter search query..." required />
|
|
</div>
|
|
|
|
<div class="viz-control-group">
|
|
<label>Algorithm</label>
|
|
<select x-model="algorithm">
|
|
<option value="semantic">Semantic (Dense)</option>
|
|
<option value="bm25_hybrid" selected>BM25 Hybrid</option>
|
|
</select>
|
|
</div>
|
|
|
|
<div class="viz-control-group">
|
|
<label>Fusion</label>
|
|
<select x-model="fusion" :disabled="algorithm !== 'bm25_hybrid'" :style="algorithm !== 'bm25_hybrid' ? 'opacity: 0.5; cursor: not-allowed;' : ''">
|
|
<option value="rrf" selected>RRF</option>
|
|
<option value="dbsf">DBSF</option>
|
|
</select>
|
|
</div>
|
|
|
|
<div class="viz-control-group">
|
|
<label> </label>
|
|
<button type="submit" class="viz-btn">Search</button>
|
|
</div>
|
|
|
|
<div class="viz-control-group">
|
|
<label> </label>
|
|
<button type="button" class="viz-btn-secondary" @click="showAdvanced = !showAdvanced">
|
|
<span x-text="showAdvanced ? 'Hide' : 'Advanced'"></span>
|
|
</button>
|
|
</div>
|
|
</div>
|
|
|
|
<!-- Advanced Options (Collapsible) -->
|
|
<div x-show="showAdvanced" style="margin-top: 16px;">
|
|
<div class="viz-controls-grid" style="grid-template-columns: repeat(auto-fit, minmax(150px, 1fr));">
|
|
<div class="viz-control-group">
|
|
<label>Document Types</label>
|
|
<div style="display: grid; grid-template-columns: 1fr 1fr; gap: 8px; font-size: 13px;">
|
|
<label style="display: flex; align-items: center; cursor: pointer; font-weight: normal;">
|
|
<input type="checkbox" x-model="docTypes" value="" style="margin-right: 4px;">
|
|
<span>All</span>
|
|
</label>
|
|
<label style="display: flex; align-items: center; cursor: pointer; font-weight: normal;">
|
|
<input type="checkbox" x-model="docTypes" value="note" style="margin-right: 4px;">
|
|
<span>Notes</span>
|
|
</label>
|
|
<label style="display: flex; align-items: center; cursor: pointer; font-weight: normal;">
|
|
<input type="checkbox" x-model="docTypes" value="file" style="margin-right: 4px;">
|
|
<span>Files</span>
|
|
</label>
|
|
<label style="display: flex; align-items: center; cursor: pointer; font-weight: normal;">
|
|
<input type="checkbox" x-model="docTypes" value="calendar" style="margin-right: 4px;">
|
|
<span>Calendar</span>
|
|
</label>
|
|
<label style="display: flex; align-items: center; cursor: pointer; font-weight: normal;">
|
|
<input type="checkbox" x-model="docTypes" value="contact" style="margin-right: 4px;">
|
|
<span>Contacts</span>
|
|
</label>
|
|
<label style="display: flex; align-items: center; cursor: pointer; font-weight: normal;">
|
|
<input type="checkbox" x-model="docTypes" value="deck" style="margin-right: 4px;">
|
|
<span>Deck</span>
|
|
</label>
|
|
</div>
|
|
</div>
|
|
|
|
<div class="viz-control-group">
|
|
<label>Score Threshold</label>
|
|
<input type="number" x-model.number="scoreThreshold" min="0" max="1" step="any" />
|
|
</div>
|
|
|
|
<div class="viz-control-group">
|
|
<label>Result Limit</label>
|
|
<input type="number" x-model.number="limit" min="1" max="1000" />
|
|
</div>
|
|
|
|
<div class="viz-control-group">
|
|
<label>Display Options</label>
|
|
<label style="display: flex; align-items: center; cursor: pointer; font-weight: normal; margin-top: 4px;">
|
|
<input type="checkbox" x-model="showQueryPoint" @change="updatePlot()" style="margin-right: 6px;">
|
|
<span>Show Query Point</span>
|
|
</label>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</form>
|
|
</div>
|
|
|
|
<!-- Plot -->
|
|
<div class="viz-card viz-card-plot">
|
|
<div id="viz-plot-container">
|
|
<div x-show="loading" class="viz-loading-overlay" x-transition.opacity.duration.200ms>
|
|
Executing search and computing PCA projection...
|
|
</div>
|
|
<div id="viz-plot" x-show="!loading" x-transition.opacity.duration.200ms></div>
|
|
</div>
|
|
</div>
|
|
|
|
<!-- Results -->
|
|
<div class="viz-card" style="flex: 0 0 auto;">
|
|
<h3 style="margin-top: 0;">Search Results (<span x-text="loading ? '...' : results.length"></span>)</h3>
|
|
|
|
<div x-show="loading" class="viz-loading" x-transition.opacity.duration.200ms>
|
|
Loading results...
|
|
</div>
|
|
|
|
<div x-show="!loading && results.length === 0" class="viz-no-results" x-transition.opacity.duration.200ms>
|
|
No results found. Try a different query or adjust your search parameters.
|
|
</div>
|
|
|
|
<template x-if="!loading && results.length > 0">
|
|
<div x-transition.opacity.duration.200ms>
|
|
<template x-for="result in results" :key="`${result.doc_type}_${result.id}_${result.chunk_start_offset || 0}`">
|
|
<div style="padding: 12px; border-bottom: 1px solid #eee;">
|
|
<a :href="getNextcloudUrl(result)" target="_blank" style="font-weight: 500; color: #0066cc; text-decoration: none;">
|
|
<span x-text="result.title"></span>
|
|
</a>
|
|
<div style="font-size: 14px; color: #666; margin-top: 4px;"
|
|
x-text="result.excerpt.length > 200 ? result.excerpt.substring(0, 200) + '...' : result.excerpt"></div>
|
|
<div style="font-size: 12px; color: #999; margin-top: 4px;">
|
|
Raw Score: <span x-text="result.original_score.toFixed(3)"></span>
|
|
(<span x-text="(result.score * 100).toFixed(0)"></span>% relative) |
|
|
Type: <span x-text="result.doc_type"></span>
|
|
</div>
|
|
|
|
<!-- Show Chunk button (only if chunk position is available) -->
|
|
<template x-if="hasChunkPosition(result)">
|
|
<button
|
|
class="chunk-toggle-btn"
|
|
@click="toggleChunk(result)"
|
|
x-text="isChunkExpanded(`${result.doc_type}_${result.id}_${result.chunk_start_offset || 0}`) ? 'Hide Chunk' : 'Show Chunk'"
|
|
></button>
|
|
</template>
|
|
|
|
<!-- Chunk context (expanded inline) -->
|
|
<template x-if="isChunkExpanded(`${result.doc_type}_${result.id}_${result.chunk_start_offset || 0}`)">
|
|
<div class="chunk-context" x-transition.opacity.duration.200ms>
|
|
<template x-if="chunkLoading[`${result.doc_type}_${result.id}_${result.chunk_start_offset || 0}`]">
|
|
<div style="color: #666; font-style: italic;">Loading chunk...</div>
|
|
</template>
|
|
<template x-if="!chunkLoading[`${result.doc_type}_${result.id}_${result.chunk_start_offset || 0}`]">
|
|
<div>
|
|
<!-- Highlighted page image for PDFs -->
|
|
<template x-if="expandedChunks[`${result.doc_type}_${result.id}_${result.chunk_start_offset || 0}`]?.highlighted_page_image">
|
|
<div class="chunk-image-container">
|
|
<div class="chunk-image-header">
|
|
<span>Page <span x-text="expandedChunks[`${result.doc_type}_${result.id}_${result.chunk_start_offset || 0}`]?.page_number"></span></span>
|
|
</div>
|
|
<img
|
|
:src="'data:image/png;base64,' + expandedChunks[`${result.doc_type}_${result.id}_${result.chunk_start_offset || 0}`]?.highlighted_page_image"
|
|
:alt="'Page ' + expandedChunks[`${result.doc_type}_${result.id}_${result.chunk_start_offset || 0}`]?.page_number"
|
|
class="chunk-highlighted-image"
|
|
/>
|
|
</div>
|
|
</template>
|
|
<!-- Text context -->
|
|
<template x-if="expandedChunks[`${result.doc_type}_${result.id}_${result.chunk_start_offset || 0}`]?.has_more_before">
|
|
<span class="chunk-ellipsis">...</span>
|
|
</template>
|
|
<span class="chunk-text" x-text="expandedChunks[`${result.doc_type}_${result.id}_${result.chunk_start_offset || 0}`]?.before_context"></span><span class="chunk-matched" x-text="expandedChunks[`${result.doc_type}_${result.id}_${result.chunk_start_offset || 0}`]?.chunk_text"></span><span class="chunk-text" x-text="expandedChunks[`${result.doc_type}_${result.id}_${result.chunk_start_offset || 0}`]?.after_context"></span><template x-if="expandedChunks[`${result.doc_type}_${result.id}_${result.chunk_start_offset || 0}`]?.has_more_after">
|
|
<span class="chunk-ellipsis">...</span>
|
|
</template>
|
|
</div>
|
|
</template>
|
|
</div>
|
|
</template>
|
|
</div>
|
|
</template>
|
|
</div>
|
|
</template>
|
|
</div><!-- Search Results -->
|
|
</div><!-- .viz-layout -->
|
|
</div><!-- x-data="vizApp()" -->
|