fix(usage): embed query once across doc_types; address review round 1
Round-1 claude-review findings: - 🔴 Multi-doc_type search billed N embedding calls as 1. nc_semantic_search loops search() once per doc_type on one BM25HybridSearchAlgorithm instance, and each call re-embedded the query, so only the last query_token_count was recorded. Cache the dense embedding per query on the (per-request) instance so the query is embedded — and metered — exactly once regardless of how many doc_types are searched. This also removes the redundant per-type embed work and avoids billing a user N× for one logical query. - 🟡 Ollama embed() now delegates to embed_with_usage() so single and batch embeds use the same /api/embed endpoint (was the legacy /api/embeddings), keeping _detect_dimension and other embed() callers consistent. - 🟢 round() instead of truncating int() when coercing provider-reported token counts (forward-compatible if a provider ever returns a float). Tests: per-instance query-embedding cache (embedded once across 3 doc_types; re-embeds on a different query). Deferred (stated on the PR): mistral/openai single-embed dual path (changes tested error/request semantics on the cloud-critical path — separate refactor), bedrock boto3 sync-in-async (pre-existing; no new invoke_model calls per doc). Deck #67. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
64318f0b25
commit
a0bb5642cb
@@ -537,9 +537,10 @@ def configure_semantic_tools(mcp: FastMCP):
|
||||
#
|
||||
# query_token_count is set by BM25HybridSearchAlgorithm during the
|
||||
# search() above. The doc_types loop reuses one search_algo instance
|
||||
# for the same query string, so the final value is the single query
|
||||
# embedding's cost (matches the prior one-query semantics). Falls
|
||||
# back to 0 only if the embedding never ran (e.g. a pre-embed error).
|
||||
# for the same query, and the algorithm caches the dense embedding
|
||||
# per query, so the query is embedded — and metered — exactly once
|
||||
# regardless of how many doc_types were searched. Falls back to 0
|
||||
# only if the embedding never ran (e.g. a pre-embed error).
|
||||
#
|
||||
# Privacy note: user_id stays tenant-local. The CP rollup
|
||||
# aggregates GROUP BY (day, metric) into usage_daily, which has no
|
||||
|
||||
Reference in New Issue
Block a user