Round-2 review nit (PR #879): lock the intentional record ordering
(tokens_embedded before the conditional pages_embedded) with an
assertion in test_parsed_file_records_pages_and_tokens, so a refactor
that reverses it fails a test rather than only contradicting a comment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-1 review follow-ups (PR #879):
- Gate pages_embedded on `page_count and page_count > 0` so a malformed
negative count meters as "no pages" rather than emitting a negative
billing row (matches the documented call-site intent).
- Exclude bool at the call-site narrowing (`isinstance(int) and not
isinstance(bool)`) — bool is an int subclass, so a stray page_count=True
would otherwise record pages=1.
- Document chunk_count's role (empty-batch no-op guard) and the
intentional tokens-before-pages ordering in the docstring/comments.
- Add test_negative_pages_skips_pages.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`pages_embedded` carried an interim chunk count (`len(chunk_texts)`,
TODO #282). Reframe it as a charge for *parsing* (PDF page extraction /
OCR) rather than a normalized content size:
- Parsed files (PDFs) record `pages_embedded` = real `page_count` from
the document processor metadata.
- Text content (notes, deck cards, news items) is never parsed, carries
no `page_count`, and records no `pages_embedded` row — only
`tokens_embedded`. There is deliberately no chars/tokens-per-page
constant; pages map 1:1 to parsed document pages.
`record_indexing_usage` now takes `page_count` and records the two
dimensions independently, gating `pages_embedded` on a truthy page count
(not the doc_type) so a future non-PDF parsed type stays correct. Stays
flag-gated + best-effort. Tests cover parsed-file, text-only, and
zero-page cases.
Deck #282 (board 8). Billing-model ADR corrected in
astrolabe-cloud-website docs/control-plane/usage-metering.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Billing product model finalized (Deck #281): bill pages externally, record
tokens internally. Rename the data-plane metric literals to match the now-
canonical contract (Deck #284) — the control plane's METRIC_EVENT_NAMES is
already renamed, so the old names would be unmapped and never sync to Stripe.
Rename (values unchanged):
- embeddings_queries → tokens_embedded (value = real token count, already
emitted by this PR; the unit upstream providers bill on).
- pages_chunks → pages_embedded (value kept as len(chunk_texts) interim;
TODO(#282): real normalized "pages indexed" count — real pages for paginated
types, chars/tokens-per-page constant otherwise — is deferred to the
instrumentation card, this only lands the name/contract).
- All literals, log strings, docstrings, comments, the migration comment, and
tests renamed; grep confirms zero old strings remain.
Observability (new): export embedding token cost to Prometheus as
astrolabe_embedding_tokens_total{provider,operation} (operation = index|query)
so the billed cost unit is visible in Grafana, not just the per-tenant billing
DB. Dedicated counter (doesn't inflate the existing chunk/request metrics) and
always-on (independent of USAGE_METERING_ENABLED, so OSS/self-host gets it).
Wired on both the indexing batch embed and the search query embed (query inside
the per-request cache-miss branch, so reused embeddings aren't double-counted).
Note: the rename orphans any pre-existing embeddings_queries/pages_chunks rows
in tenant app DBs (CP no longer maps them) — acceptable; pipeline is inert with
throwaway dev/sandbox data.
Deck #284 (folded into PR #875).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-2 claude-review findings:
- 🟡 Base-class recursion invariant: documented on embed_with_usage /
embed_batch_with_usage that a provider overriding embed()/embed_batch() to
delegate to the *_with_usage variant MUST also override that variant, or the
two recurse. (No recursion today; the shipped providers pair the overrides.)
- 🟡 Processor metering had no unit test: extracted the two-event recording
into a module-level record_indexing_usage() helper and added
tests/unit/test_processor_metering.py (value mapping, flag/zero-chunk no-ops,
best-effort failure swallowed).
- 🟡 SonarQube hotspots (python:S5332) were 3 http:// URLs in the new test
fixtures (mock hosts, never contacted) blocking the quality gate
(new_security_hotspots_reviewed). Switched them to https:// so no hotspot is
raised.
- 🟢 Zero-chunk guard: record_indexing_usage() no-ops when chunk_count == 0, so
an empty document no longer writes zero-value billing rows.
Deferred (stated on the PR): Mistral x.index-or-0 sort key (pre-existing,
equivalent), CHANGELOG note for the Ollama /api/embed switch (CHANGELOG is
commitizen-generated from commit bodies, which document it), class-var
query_token_count (safe under the per-request instance pattern).
Deck #67.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
