Public/fgAI
SHA256
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Also veto timestamp-shaped fields from categorical/entity/relationship roles

Browse Source 
Found live: a baseline-maintenance vacuum surfaced eventtime/timestamp
sitting in a stream's categorical_fields, hitting the max-values-per-field
cap at 2001 distinct values. Same failure mode as the original context
incident — a timestamp is near-unique on essentially every event, so
tracking it as a categorical or relationship field guarantees a "new"
finding every time. Extends _SHAPE_VETO (entity/categorical/relationship)
to also exclude "timestamp" shape, reusing the existing veto machinery and
its default 60-confidence threshold (timestamp classification isn't capped
low the way free_text is, so the stricter default applies correctly).
Existing profiles with this issue get flagged automatically via
detect_shape_mismatches, which already reuses the same veto check.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
larssand
2026-09-24 18:27:45 +02:00
parent e61403d7f6
commit bc4d2f465f
3 changed files with 35 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
tests/test_profile_suggestions.py
View File

@@ -393,6 +393,28 @@ class ProfileSuggestionTests(unittest.TestCase):
self.assertNotIn("context", relationship_fields)
self.assertIn("srcip", profile["entity_fields"])
def test_timestamp_shaped_field_is_vetoed_from_categorical_and_relationships(self):
events = [
parse_log_line(
f"fgai_stream_id=squid fgai_stream=Squid srcip=10.0.0.{index % 7} "
f"gl2_processing_finished=2026-06-29T10:{index % 60:02d}:00Z "
f"action=allow timestamp=2026-06-29T10:00:{index % 60:02d}Z"
)
for index in range(1, 40)
]
suggestion = suggest_stream_profiles(events)[0]
profile = suggestion["profile"]
self.assertEqual(suggestion["field_shapes"]["gl2_processing_finished"], "timestamp")
self.assertNotIn("gl2_processing_finished", profile["entity_fields"])
self.assertNotIn("gl2_processing_finished", profile["categorical_fields"])
relationship_fields = {item["left"] for item in profile["relationship_fields"]} | {
item["right"] for item in profile["relationship_fields"]
}
self.assertNotIn("gl2_processing_finished", relationship_fields)
self.assertIn("srcip", profile["entity_fields"])
def test_field_shape_classification_uses_real_event_volume_not_sample_size(self):
# 24 distinct usernames repeating across 300 events is genuinely low
# cardinality (8%) even though 24 distinct values alone would look