Also veto timestamp-shaped fields from categorical/entity/relationship roles

Found live: a baseline-maintenance vacuum surfaced eventtime/timestamp
sitting in a stream's categorical_fields, hitting the max-values-per-field
cap at 2001 distinct values. Same failure mode as the original context
incident — a timestamp is near-unique on essentially every event, so
tracking it as a categorical or relationship field guarantees a "new"
finding every time. Extends _SHAPE_VETO (entity/categorical/relationship)
to also exclude "timestamp" shape, reusing the existing veto machinery and
its default 60-confidence threshold (timestamp classification isn't capped
low the way free_text is, so the stricter default applies correctly).
Existing profiles with this issue get flagged automatically via
detect_shape_mismatches, which already reuses the same veto check.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
larssand
2026-09-24 18:27:45 +02:00
parent e61403d7f6
commit bc4d2f465f
3 changed files with 35 additions and 9 deletions

View File

@@ -252,12 +252,16 @@ even when it was not selected into the applied profile.
A name match never overrides value shape, even for names on the trusted A name match never overrides value shape, even for names on the trusted
priority lists such as `context`. If the observed values for a name-matched priority lists such as `context`. If the observed values for a name-matched
field confidently look like free text (a raw log line or URL, not a field confidently look like free text (a raw log line or URL, not a
low-cardinality category) it is excluded from entity, categorical, and low-cardinality category) or like a timestamp, it is excluded from entity,
relationship-target selection regardless of its name, both for a fresh categorical, and relationship-target selection regardless of its name, both
recommendation and as a second, independent check when a field reaches for a fresh recommendation and as a second, independent check when a field
relationship-candidate selection through cross-stream field sharing. A field reaches relationship-candidate selection through cross-stream field sharing.
with too few samples to classify is never excluded on that basis alone; the The timestamp case matters for the same reason free text does: a timestamp
veto only fires on a confident, contradicting shape. is near-unique on essentially every event, so tracking it as a categorical
or relationship field is guaranteed to produce a "never seen before" finding
every single time, exactly the failure mode that made `context` a problem.
A field with too few samples to classify is never excluded on that basis
alone; the veto only fires on a confident, contradicting shape.
Windows-like streams are recognized from stream names such as `Windows`, Windows-like streams are recognized from stream names such as `Windows`,
`Winlog`, `Security Event Log`, `Sysmon`, and `Powershell`, or from Windows event `Winlog`, `Security Event Log`, `Sysmon`, and `Powershell`, or from Windows event

View File

@@ -12,10 +12,10 @@ from .normalization import FIELD_ALIASES
SHAPE_ENTITY_CONFIDENCE = 60 SHAPE_ENTITY_CONFIDENCE = 60
_SHAPE_VETO = { _SHAPE_VETO = {
"entity": {"free_text"}, "entity": {"free_text", "timestamp"},
"categorical": {"free_text"}, "categorical": {"free_text", "timestamp"},
"numeric": {"free_text", "categorical_low_cardinality"}, "numeric": {"free_text", "categorical_low_cardinality"},
"relationship": {"free_text"}, "relationship": {"free_text", "timestamp"},
} }
# classify_field_shape() gives free_text a fixed 0.4 fit ratio and no name # classify_field_shape() gives free_text a fixed 0.4 fit ratio and no name

View File

@@ -393,6 +393,28 @@ class ProfileSuggestionTests(unittest.TestCase):
self.assertNotIn("context", relationship_fields) self.assertNotIn("context", relationship_fields)
self.assertIn("srcip", profile["entity_fields"]) self.assertIn("srcip", profile["entity_fields"])
def test_timestamp_shaped_field_is_vetoed_from_categorical_and_relationships(self):
events = [
parse_log_line(
f"fgai_stream_id=squid fgai_stream=Squid srcip=10.0.0.{index % 7} "
f"gl2_processing_finished=2026-06-29T10:{index % 60:02d}:00Z "
f"action=allow timestamp=2026-06-29T10:00:{index % 60:02d}Z"
)
for index in range(1, 40)
]
suggestion = suggest_stream_profiles(events)[0]
profile = suggestion["profile"]
self.assertEqual(suggestion["field_shapes"]["gl2_processing_finished"], "timestamp")
self.assertNotIn("gl2_processing_finished", profile["entity_fields"])
self.assertNotIn("gl2_processing_finished", profile["categorical_fields"])
relationship_fields = {item["left"] for item in profile["relationship_fields"]} | {
item["right"] for item in profile["relationship_fields"]
}
self.assertNotIn("gl2_processing_finished", relationship_fields)
self.assertIn("srcip", profile["entity_fields"])
def test_field_shape_classification_uses_real_event_volume_not_sample_size(self): def test_field_shape_classification_uses_real_event_volume_not_sample_size(self):
# 24 distinct usernames repeating across 300 events is genuinely low # 24 distinct usernames repeating across 300 events is genuinely low
# cardinality (8%) even though 24 distinct values alone would look # cardinality (8%) even though 24 distinct values alone would look