Fix cardinality judged against sample size instead of real event volume

classify_field_shape() received a deduplicated set of observed values with
no real event-volume context, so cardinality was always measured against
itself — trivially 100%, making any field with more than ~16 distinct
values look "high cardinality" no matter how large the real traffic volume
was. This misclassified genuinely low-cardinality fields (e.g. username: 82
distinct values across 1,601 events) as free text, which the new shape-veto
work then surfaced as false-positive "remove this field" warnings in
production. Fixed by passing real event coverage as an explicit
total_events parameter; omitting it keeps prior behavior for other callers.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
larssand
2026-09-23 22:02:44 +02:00
parent 83ffef3570
commit caa37de98e
5 changed files with 61 additions and 6 deletions

View File

@@ -181,7 +181,18 @@ This is now the shipped behavior, not just the target.
targets, and existing stored profiles are periodically re-validated
against current observed shapes (flagged for review, never silently
rewritten — consistent with the drift-flagging precedent already used for
auto-applied field mappings).
auto-applied field mappings). Turning this on live surfaced a real,
pre-existing calibration bug in `classify_field_shape()`: callers pass a
*deduplicated* set of observed values (capped at 200 distinct entries),
whose own length was being used as the cardinality denominator — since a
deduplicated set is always 100% cardinality against itself, any field with
more than ~16 distinct values looked "high cardinality" regardless of true
traffic volume, misclassifying genuinely low-cardinality fields like
`username` (82 distinct values out of 1,601 events) as free text. Fixed by
passing the real event coverage (`coverage[field]`) as an explicit
`total_events` parameter, so cardinality is judged against actual volume;
omitting it preserves the old sample-as-population behavior for other
callers.
- **Shared, entity-scoped baselines** for entities that shared-field
discovery already recognizes as the same concept across multiple enabled
streams, instead of one cold baseline per `(stream, entity, field)` tuple.