Fix cardinality judged against sample size instead of real event volume
classify_field_shape() received a deduplicated set of observed values with no real event-volume context, so cardinality was always measured against itself — trivially 100%, making any field with more than ~16 distinct values look "high cardinality" no matter how large the real traffic volume was. This misclassified genuinely low-cardinality fields (e.g. username: 82 distinct values across 1,601 events) as free text, which the new shape-veto work then surfaced as false-positive "remove this field" warnings in production. Fixed by passing real event coverage as an explicit total_events parameter; omitting it keeps prior behavior for other callers. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -181,7 +181,18 @@ This is now the shipped behavior, not just the target.
|
||||
targets, and existing stored profiles are periodically re-validated
|
||||
against current observed shapes (flagged for review, never silently
|
||||
rewritten — consistent with the drift-flagging precedent already used for
|
||||
auto-applied field mappings).
|
||||
auto-applied field mappings). Turning this on live surfaced a real,
|
||||
pre-existing calibration bug in `classify_field_shape()`: callers pass a
|
||||
*deduplicated* set of observed values (capped at 200 distinct entries),
|
||||
whose own length was being used as the cardinality denominator — since a
|
||||
deduplicated set is always 100% cardinality against itself, any field with
|
||||
more than ~16 distinct values looked "high cardinality" regardless of true
|
||||
traffic volume, misclassifying genuinely low-cardinality fields like
|
||||
`username` (82 distinct values out of 1,601 events) as free text. Fixed by
|
||||
passing the real event coverage (`coverage[field]`) as an explicit
|
||||
`total_events` parameter, so cardinality is judged against actual volume;
|
||||
omitting it preserves the old sample-as-population behavior for other
|
||||
callers.
|
||||
- **Shared, entity-scoped baselines** for entities that shared-field
|
||||
discovery already recognizes as the same concept across multiple enabled
|
||||
streams, instead of one cold baseline per `(stream, entity, field)` tuple.
|
||||
|
||||
Reference in New Issue
Block a user