fix(usage): embed query once across doc_types; address review round 1
Round-1 claude-review findings: - 🔴 Multi-doc_type search billed N embedding calls as 1. nc_semantic_search loops search() once per doc_type on one BM25HybridSearchAlgorithm instance, and each call re-embedded the query, so only the last query_token_count was recorded. Cache the dense embedding per query on the (per-request) instance so the query is embedded — and metered — exactly once regardless of how many doc_types are searched. This also removes the redundant per-type embed work and avoids billing a user N× for one logical query. - 🟡 Ollama embed() now delegates to embed_with_usage() so single and batch embeds use the same /api/embed endpoint (was the legacy /api/embeddings), keeping _detect_dimension and other embed() callers consistent. - 🟢 round() instead of truncating int() when coercing provider-reported token counts (forward-compatible if a provider ever returns a float). Tests: per-instance query-embedding cache (embedded once across 3 doc_types; re-embeds on a different query). Deferred (stated on the PR): mistral/openai single-embed dual path (changes tested error/request semantics on the cloud-critical path — separate refactor), bedrock boto3 sync-in-async (pre-existing; no new invoke_model calls per doc). Deck #67. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
64318f0b25
commit
a0bb5642cb
@@ -233,7 +233,7 @@ class OpenAIProvider(Provider):
|
||||
# gives an int, but test doubles / partial responses can surface a
|
||||
# non-numeric attribute — fall back to the estimate there.
|
||||
tokens = (
|
||||
int(total_tokens)
|
||||
round(total_tokens)
|
||||
if isinstance(total_tokens, (int, float))
|
||||
else self._estimate_tokens(batch)
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user