Public/fgAI
SHA256
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add new structure

Browse Source
This commit is contained in:
larssand
2026-09-23 17:06:05 +02:00
parent 7cbec64b0e
commit 8c155ec9cf
8 changed files with 343 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files

33
README.md
View File

@@ -159,6 +159,20 @@ Interpret findings by maturity:
- `event_rate_burst`, `auth_failure_burst`, `dns_query_burst`, and
`deny_action_burst` compare the current window to historical buckets.
Field deviations that share the same detector and field are clustered before
scoring, not after. If the same `(detector, field)` signature appears across
five or more entities in one poll (`false_positive`/`expected`-suppressed
deviations don't count toward that total), every deviation in the cluster is
tagged with a `cluster_size` and its score is capped at 25, and the Evidence
column shows `[part of a N-entity pattern]` ahead of the reason. This is a
deliberate inversion of the usual "more occurrences = more severe" rule: a
pattern hitting many unrelated entities identically almost always means a
detector or profile problem (a field that shouldn't have been picked as a
relationship target, for example) rather than N separate critical security
events, so it is scored at or below an isolated occurrence instead of above
it. A small, isolated deviation keeps its full score. See
[`ARCHITECTURE.md`](ARCHITECTURE.md) for the incident that motivated this.
The review buttons are part of the detection loop:
- `Expected`: known behavior that should stay visible but not keep creating
@@ -217,6 +231,16 @@ dashboard status payload carries the underlying per-field shape and confidence
values, so a `field_confidence`/`field_shapes` map is available for any field
even when it was not selected into the applied profile.
A name match never overrides value shape, even for names on the trusted
priority lists such as `context`. If the observed values for a name-matched
field confidently look like free text (a raw log line or URL, not a
low-cardinality category) it is excluded from entity, categorical, and
relationship-target selection regardless of its name, both for a fresh
recommendation and as a second, independent check when a field reaches
relationship-candidate selection through cross-stream field sharing. A field
with too few samples to classify is never excluded on that basis alone; the
veto only fires on a confident, contradicting shape.
Windows-like streams are recognized from stream names such as `Windows`,
`Winlog`, `Security Event Log`, `Sysmon`, and `Powershell`, or from Windows event
fields. The MCP search asks for common Winlogbeat/ECS names such as
@@ -337,6 +361,15 @@ After the baseline has learned those relationships, a new `username -> srcip` or
`username -> hostname` pair is reported as `new_relationship` with sample events.
The dashboard profile editor exposes this as `Behavior relationships (JSON)`.
An existing stored profile's `categorical_fields` and `relationship_fields` are
re-checked against current observed shapes on every poll, not just when the
profile was first created — a field can look fine at profile-creation time and
drift into free text later, or the classifier's read of it can simply have been
wrong initially. A mismatch never rewrites the stored profile automatically;
it only shows up under `Recommended Stream Profiles` as a warning naming the
field, the stream, and the confidence behind the mismatch, with a button that
opens that stream's profile editor so a person decides whether to remove it.
SignalScope keeps a common alias map for fields such as source IP, destination
IP, ports, action, severity, service/protocol, DNS query, URL, message, and event
type. This lets Related Activity and correlations work with firewall/proxy/DNS

34
ROADMAP.md
View File

@@ -29,18 +29,18 @@ parts of sections 1 and 6 below (baseline confidence tooling and compact
incident cards both depend on findings being clustered correctly first) — do
this section before spending more effort on either.
- [ ] Validate name-matched fields against observed value shape too, not just
- [x] Validate name-matched fields against observed value shape too, not just
fields that had no name match. A field that matches a trusted name (e.g.
`context`) but is actually free-text (a full raw log line, confirmed in 12
of 25 audited stream profiles) must not be trusted just because its name
matched a known-useful alias.
- [ ] Make relationship-candidate selection shape-aware: never propose a
- [x] Make relationship-candidate selection shape-aware: never propose a
free-text-shaped field as a relationship target, regardless of how it
entered `categorical_fields`. This alone was responsible for 17 of 25
entries in one real triage queue snapshot (91 individual `new_relationship`
deviations from a single misconfigured `*->context` pairing, repeated
across 5 streams).
- [ ] Re-validate existing stored profiles' `relationship_fields` and
- [x] Re-validate existing stored profiles' `relationship_fields` and
`categorical_fields` against current observed shapes on an ongoing basis,
not just at profile-creation time — flag (never silently rewrite) fields
that no longer look fit for their assigned role, and surface a one-click
@@ -50,18 +50,22 @@ this section before spending more effort on either.
(for example a source IP seen in both a firewall stream and an IIS
stream), instead of a cold, isolated baseline per stream per field. A value
already known-normal in one contributing stream must not register as "new"
the first time a different, related stream happens to see it.
- [ ] Cluster field deviations by `(detector, field, value-pattern)`
signature *before* scoring, not after. A pattern recurring identically
across many entities must surface as one clustered item with an occurrence
count and a "how many entities" breakdown, not N independently-scored
"critical" rows that all have to be discovered, selected, and bulk-cleared
by hand after the fact.
- [ ] Recalibrate severity so cluster size is itself a signal: a pattern
hitting many unrelated entities identically is usually a
configuration/detector problem, not N separate critical incidents, and
should score at or below an isolated occurrence, never above it.
- [ ] Update README usage documentation alongside each item above as it
the first time a different, related stream happens to see it. Deferred:
needs a `baseline.py` schema change and its own dedicated planning pass.
- [x] Cluster field deviations by `(detector, field)` signature *before*
scoring, not after (value-pattern was dropped from the original signature —
detector+field alone already matched the real incident's cross-stream
pattern, and adding value-pattern would have under-clustered near-identical
but not-byte-identical raw messages). A pattern recurring identically
across many entities surfaces as one clustered item with an occurrence
count (`cluster_size`) shown inline in the Evidence column, not N
independently-scored "critical" rows that all have to be discovered,
selected, and bulk-cleared by hand after the fact.
- [x] Recalibrate severity so cluster size is itself a signal: a pattern
hitting five or more entities identically is capped at score 25 regardless
of its uncapped score, so it never outranks an isolated, uncapped
occurrence.
- [x] Update README usage documentation alongside each item above as it
ships — a behavior change is not done until it is documented.
Acceptance: replaying the dataset that produced the original 25-entry,

11
src/fgai/dashboard.py
View File

File diff suppressed because one or more lines are too long

42
src/fgai/monitor.py
View File

@@ -5,6 +5,7 @@ import json
import signal
import threading
import time
from collections import defaultdict
from pathlib import Path
from .anomaly import anomaly_summary, detect_source_anomalies
@@ -251,6 +252,40 @@ def _detect_profile_drift(
return drift
CLUSTER_SIZE_DAMPEN_THRESHOLD = 5
CLUSTER_DAMPENED_SCORE_CAP = 25
def _cluster_field_deviations(field_deviations: dict[str, list[dict[str, object]]]) -> None:
"""Recognize the same (detector, field) signature repeating across many
entities as one systemic pattern instead of N independent findings.
Mutates deviations in place: tags cluster_size, and caps score for large
clusters — a signature hitting many entities identically usually means a
detector/profile problem, not N separate critical events. Signature
deliberately excludes stream_id: the incident this guards against (a
misconfigured relationship field) recurred identically across multiple
streams, and that cross-stream repetition is itself part of the pattern.
Already-suppressed deviations don't count toward another entity's cluster
size, so a false-positive rule doesn't inflate the appearance of a pattern.
"""
signature_entities: dict[tuple[str, str], set[str]] = defaultdict(set)
for entity, deviations in field_deviations.items():
for deviation in deviations:
if deviation.get("feedback") in SUPPRESSING_STATUSES:
continue
signature = (str(deviation.get("detector", "")), str(deviation.get("field", "")))
signature_entities[signature].add(entity)
for entity, deviations in field_deviations.items():
for deviation in deviations:
signature = (str(deviation.get("detector", "")), str(deviation.get("field", "")))
cluster_size = len(signature_entities.get(signature, ()))
deviation["cluster_size"] = cluster_size
if cluster_size >= CLUSTER_SIZE_DAMPEN_THRESHOLD:
deviation["score"] = min(int(deviation.get("score", 0) or 0), CLUSTER_DAMPENED_SCORE_CAP)
deviation["cluster_dampened"] = True
def _stream_name(stream_id: str, stream_titles: dict[str, str], profile: object | None = None) -> str:
return stream_titles.get(stream_id) or getattr(profile, "name", "") or stream_id
@@ -524,6 +559,7 @@ def build_status(
deviation["feedback_rule_id"] = match.get("id", "")
if match["status"] in SUPPRESSING_STATUSES:
deviation["score"] = 0
_cluster_field_deviations(field_deviations)
generalization_min_values = int(runtime_values.get("feedback_generalization_min_values", 3) or 3)
feedback_generalization_suggestions = suggest_generalizations(feedback, min_values=generalization_min_values)
anomalies = detect_source_anomalies(events, limit=anomaly_limit, baselines=profiles, field_deviations=field_deviations, feedback=feedback)
@@ -660,6 +696,11 @@ def build_status(
summary["total"] = int(mcp_status.get("aggregate_events", 0) or 0)
summary["raw_sample_total"] = len(events)
summary["aggregate_backed"] = True
profile_shape_warnings = [
{"stream_id": suggestion.get("stream_id", ""), "stream_name": suggestion.get("stream_name", ""), **warning}
for suggestion in profile_suggestions
for warning in suggestion.get("shape_warnings", [])
]
status = {
"status_schema": 2,
"stale": False,
@@ -676,6 +717,7 @@ def build_status(
"profile_drift": profile_drift,
"stream_coverage": stream_coverage,
"profile_suggestions": profile_suggestions,
"profile_shape_warnings": profile_shape_warnings,
"profile_readiness": profile_readiness,
"diagnostics": {
"top_source_ips": top_field_values(events, "srcip", limit=10),

84
src/fgai/profile_suggestions.py
View File

@@ -11,6 +11,68 @@ from .normalization import FIELD_ALIASES
SHAPE_ENTITY_CONFIDENCE = 60
_SHAPE_VETO = {
"entity": {"free_text"},
"categorical": {"free_text"},
"numeric": {"free_text", "categorical_low_cardinality"},
"relationship": {"free_text"},
}
# classify_field_shape() gives free_text a fixed 0.4 fit ratio and no name
# bonus (field_shapes.py), so a genuine free-text verdict tops out around
# 28-43 confidence — well under the 60 that a clean ip/categorical/numeric
# fit reaches. The fallback "nothing fit" case (classify_field_shape's
# `if not scores` branch) can't exceed 15. 20 cleanly separates a real
# free-text fit from that weak fallback while staying reachable.
_SHAPE_VETO_CONFIDENCE = {
"free_text": 20,
}
_DEFAULT_VETO_CONFIDENCE = 60
def _shape_vetoes(field: str, role: str, field_shapes: dict[str, str], field_confidence: dict[str, int]) -> bool:
"""True when observed values confidently contradict a name-matched field's role.
Only vetoes on positive evidence (a confident, wrong-looking shape) —
fields with no shape info yet (insufficient samples) are never vetoed.
"""
shape = field_shapes.get(field)
if shape is None or shape not in _SHAPE_VETO.get(role, set()):
return False
threshold = _SHAPE_VETO_CONFIDENCE.get(shape, _DEFAULT_VETO_CONFIDENCE)
return field_confidence.get(field, 0) >= threshold
def detect_shape_mismatches(existing: object, field_shapes: dict[str, str], field_confidence: dict[str, int]) -> list[dict[str, object]]:
"""Flag an existing stored profile's fields whose current observed shape no
longer fits their assigned role. Flag-only — never rewrites the stored
profile; a human reviews and removes the field from the dashboard warning.
"""
if existing is None:
return []
warnings: list[dict[str, object]] = []
seen: set[tuple[str, str]] = set()
def flag(field: str, role: str) -> None:
key = (field, role)
if not field or key in seen or not _shape_vetoes(field, role, field_shapes, field_confidence):
return
seen.add(key)
warnings.append({
"field": field,
"role": role,
"shape": field_shapes.get(field),
"confidence": field_confidence.get(field, 0),
})
for field in getattr(existing, "categorical_fields", ()):
flag(field, "categorical")
for relationship in getattr(existing, "relationship_fields", ()):
flag(relationship.left, "relationship")
flag(relationship.right, "relationship")
return warnings
ALIAS_CANONICAL_BY_FIELD = {
alias: canonical
for canonical, aliases in FIELD_ALIASES.items()
@@ -392,7 +454,7 @@ def _generic_entity_fields(coverage: Counter[str], unique_values: dict[str, set[
for field, count in coverage.most_common():
if len(output) >= 4:
break
if field in output or field in IGNORED_DISCOVERY_FIELDS:
if field in output or field in IGNORED_DISCOVERY_FIELDS or _shape_vetoes(field, "entity", field_shapes, field_confidence):
continue
shape_match = field_shapes.get(field) == "ip" and field_confidence.get(field, 0) >= SHAPE_ENTITY_CONFIDENCE
if not shape_match and not _looks_like_entity_field(field):
@@ -411,7 +473,7 @@ def _generic_categorical_fields(coverage: Counter[str], unique_values: dict[str,
for field, count in coverage.most_common():
if len(output) >= 12:
break
if field in output or field in IGNORED_DISCOVERY_FIELDS:
if field in output or field in IGNORED_DISCOVERY_FIELDS or _shape_vetoes(field, "categorical", field_shapes, field_confidence):
continue
coverage_ratio = count / max(1, total)
cardinality = len(unique_values[field])
@@ -428,7 +490,7 @@ def _generic_numeric_fields(coverage: Counter[str], numeric_counts: Counter[str]
for field, count in coverage.most_common():
if len(output) >= 8:
break
if field in output or field in IGNORED_DISCOVERY_FIELDS:
if field in output or field in IGNORED_DISCOVERY_FIELDS or _shape_vetoes(field, "numeric", field_shapes, field_confidence):
continue
shape_match = field_shapes.get(field) == "numeric" and field_confidence.get(field, 0) >= SHAPE_ENTITY_CONFIDENCE
if shape_match or (count / max(1, total) >= 0.1 and numeric_counts[field] / max(1, count) >= 0.95):
@@ -479,9 +541,11 @@ def _relationship_candidates(
numeric_fields: list[str],
*,
field_shapes: dict[str, str] | None = None,
field_confidence: dict[str, int] | None = None,
limit: int = 6,
) -> list[dict[str, str]]:
field_shapes = field_shapes or {}
field_confidence = field_confidence or {}
def usable(field: str) -> bool:
return (
bool(field)
@@ -513,6 +577,10 @@ def _relationship_candidates(
if _field_matches(field, {"dstip", "*_ip"}, {"dst", "dest", "destination", "server"})
]
destination_fields.extend(field for field in categorical if field.lower() in {"dstport", "destination_port", "dest_port", "service", "application", "app", "url", "query_domain", "dns_query"})
# Defense in depth: a field can reach here via shared_categorical or a future
# code path that bypasses the name-priority seed veto applied upstream, so
# re-check shape immediately before it can become a relationship target.
destination_fields = [field for field in destination_fields if not _shape_vetoes(field, "categorical", field_shapes, field_confidence)]
# Ambiguous-name tie-break: two ip-shaped entity fields that neither name-matched
# source nor destination are split using which one leans more private (source-like)
@@ -546,6 +614,7 @@ def _relationship_candidates(
if field not in entities
and not _looks_like_time_field(field)
and not any(token in field.lower() for token in ("gl2_", "message_id", "sort_field", "accounted_message_size", "processing_timestamp"))
and not _shape_vetoes(field, "categorical", field_shapes, field_confidence)
]
for left in entities:
for right in behavior_fields[:4]:
@@ -858,17 +927,20 @@ def suggest_stream_profiles(events: list[LogEvent], *, existing_profiles: dict[s
field_shapes[field] = shape.primary_shape
field_confidence[field] = shape.confidence
shared_categorical, shared_numeric, shared_stats = _shared_profile_fields(coverage, unique_values, numeric_counts, total, stream_field_counts, stream_semantic_counts, stream_semantic_fields)
entity_fields = _generic_entity_fields(coverage, unique_values, total, _pick_present(entity_priority, coverage, total, min_ratio=0.02, limit=12 if is_windows else 3), field_shapes=field_shapes, field_confidence=field_confidence)
entity_seed = [field for field in _pick_present(entity_priority, coverage, total, min_ratio=0.02, limit=12 if is_windows else 3) if not _shape_vetoes(field, "entity", field_shapes, field_confidence)]
entity_fields = _generic_entity_fields(coverage, unique_values, total, entity_seed, field_shapes=field_shapes, field_confidence=field_confidence)
timestamp = next(iter(_pick_present(TIME_PRIORITY, coverage, total, min_ratio=0.02, limit=1)), "")
if not timestamp:
timestamp = next((field for field, count in coverage.most_common() if _looks_like_time_field(field) and count / max(1, total) >= 0.02), "timestamp")
categorical_seed = list(dict.fromkeys([*_pick_present(categorical_priority, coverage, total, min_ratio=0.02, limit=10 if is_windows else 8), *shared_categorical]))
categorical_seed = [field for field in categorical_seed if not _shape_vetoes(field, "categorical", field_shapes, field_confidence)]
categorical = _generic_categorical_fields(coverage, unique_values, total, categorical_seed, field_shapes=field_shapes, field_confidence=field_confidence)
numeric_seed = [
field for field in numeric_priority
if coverage[field] and numeric_counts[field] / max(1, coverage[field]) >= 0.8
][:5]
numeric_seed = list(dict.fromkeys([*numeric_seed, *shared_numeric]))
numeric_seed = [field for field in numeric_seed if not _shape_vetoes(field, "numeric", field_shapes, field_confidence)]
numeric = numeric_seed if is_windows else _generic_numeric_fields(coverage, numeric_counts, total, numeric_seed, field_shapes=field_shapes, field_confidence=field_confidence)
if not entity_fields:
entity_fields = [field for field, _count in coverage.most_common() if field not in IGNORED_DISCOVERY_FIELDS][:1]
@@ -912,7 +984,7 @@ def suggest_stream_profiles(events: list[LogEvent], *, existing_profiles: dict[s
detectors.setdefault(name, hint)
if not detectors:
detectors = {"deny_action": {"enabled": True, "minimum": 10, "z_threshold": 3.0}}
relationship_fields = _relationship_candidates(coverage, unique_values, total, entity_fields, categorical, numeric, field_shapes=field_shapes)
relationship_fields = _relationship_candidates(coverage, unique_values, total, entity_fields, categorical, numeric, field_shapes=field_shapes, field_confidence=field_confidence)
high_coverage = [
{"field": field, "coverage": round(count / max(1, total), 2), "unique_values": len(unique_values[field])}
for field, count in coverage.most_common(12)
@@ -939,12 +1011,14 @@ def suggest_stream_profiles(events: list[LogEvent], *, existing_profiles: dict[s
],
}
existing = existing_profiles.get(stream_id)
shape_warnings = detect_shape_mismatches(existing, field_shapes, field_confidence)
score = min(100, 30 + len(entity_fields) * 15 + min(20, len(categorical) * 3) + min(15, len(detectors) * 5))
suggestions.append({
"stream_id": stream_id,
"stream_name": names.get(stream_id, stream_id),
"events": total,
"profile_exists": bool(existing),
"shape_warnings": shape_warnings,
"confidence": "high" if score >= 75 and total >= 100 else "medium" if score >= 55 and total >= 20 else "low",
"score": score,
"field_confidence": field_confidence,

11
tests/test_dashboard.py
View File

@@ -94,6 +94,17 @@ class DashboardTests(unittest.TestCase):
self.assertIn("openDetailIds", HTML)
self.assertIn("addEventListener('toggle'", HTML)
def test_dashboard_annotates_field_deviations_with_cluster_size(self):
self.assertIn("cluster_size", HTML)
self.assertIn("-entity pattern", HTML)
self.assertIn("cluster_dampened", HTML)
def test_dashboard_shows_shape_mismatch_warnings(self):
self.assertIn('id="profileShapeWarnings"', HTML)
self.assertIn("profile_shape_warnings", HTML)
self.assertIn("looks like free text", HTML)
self.assertIn("Open profile editor", HTML)
def test_dashboard_has_anomaly_bulk_bar(self):
self.assertIn('id="anomalyBulkBar"', HTML)
self.assertIn('anomaly-select', HTML)

71
tests/test_monitor.py
View File

@@ -8,7 +8,7 @@ from fgai.config import ConfigStore
from fgai.feedback import FeedbackStore
from fgai.history import StatusSnapshotStore
from fgai.history import FieldDiscoveryStore
from fgai.monitor import add_llm_assessment, build_status, cached_status_with_error, monitor_cycle_timeout_seconds, write_refreshing_status, write_status
from fgai.monitor import add_llm_assessment, build_status, cached_status_with_error, monitor_cycle_timeout_seconds, write_refreshing_status, write_status, _cluster_field_deviations
class MonitorTests(unittest.TestCase):
@@ -39,6 +39,75 @@ class MonitorTests(unittest.TestCase):
self.assertFalse(status["status_cache"]["served_from_cache"])
self.assertGreaterEqual(len(status["anomalies"]), 1)
def test_cluster_field_deviations_dampens_large_identical_clusters(self):
field_deviations = {
f"10.0.0.{index}": [{"detector": "relationship", "field": "client_ip->context", "score": 40}]
for index in range(1, 6)
}
_cluster_field_deviations(field_deviations)
for entity, deviations in field_deviations.items():
deviation = deviations[0]
self.assertEqual(deviation["cluster_size"], 5)
self.assertEqual(deviation["score"], 25)
self.assertTrue(deviation["cluster_dampened"])
def test_cluster_field_deviations_leaves_small_clusters_untouched(self):
field_deviations = {
f"10.0.0.{index}": [{"detector": "relationship", "field": "client_ip->context", "score": 40}]
for index in range(1, 4)
}
_cluster_field_deviations(field_deviations)
for entity, deviations in field_deviations.items():
deviation = deviations[0]
self.assertEqual(deviation["cluster_size"], 3)
self.assertEqual(deviation["score"], 40)
self.assertNotIn("cluster_dampened", deviation)
def test_cluster_field_deviations_excludes_suppressed_entities_from_cluster_size(self):
field_deviations = {
f"10.0.0.{index}": [{"detector": "relationship", "field": "client_ip->context", "score": 40}]
for index in range(1, 5)
}
field_deviations["10.0.0.99"] = [{"detector": "relationship", "field": "client_ip->context", "score": 40, "feedback": "false_positive"}]
_cluster_field_deviations(field_deviations)
self.assertEqual(field_deviations["10.0.0.1"][0]["cluster_size"], 4)
self.assertNotIn("cluster_dampened", field_deviations["10.0.0.1"][0])
self.assertEqual(field_deviations["10.0.0.99"][0]["score"], 40)
def test_cluster_field_deviations_dampen_large_clusters_in_build_status(self):
with tempfile.TemporaryDirectory() as tmp:
srcips = [f"203.0.113.{index}" for index in range(1, 6)]
log_path = Path(tmp) / "fg.log"
log_path.write_text(
"\n".join(
f"type=utm subtype=ips srcip={ip} dstip=10.0.0.10 service=https action=blocked severity=critical"
for ip in srcips
),
encoding="utf-8",
)
canned_deviations = {
ip: [{"detector": "relationship", "field": "client_ip->context", "stream_id": "utm", "score": 40, "reason": "new relationship: client_ip->context"}]
for ip in srcips
}
with patch("fgai.monitor.BaselineStore.profile_deviations", return_value=canned_deviations):
status = build_status(str(log_path), baseline_path=str(Path(tmp) / "baseline.sqlite3"), incident_path=str(Path(tmp) / "incidents.json"))
for ip in srcips:
deviation = status["field_deviations"][ip][0]
self.assertEqual(deviation["cluster_size"], 5)
self.assertEqual(deviation["score"], 25)
self.assertTrue(deviation["cluster_dampened"])
anomalies_by_subject = {item["subject"]: item for item in status["anomalies"]}
self.assertIn("new relationship: client_ip->context", anomalies_by_subject[srcips[0]]["reasons"])
def test_entity_scope_feedback_suppresses_anomaly_in_build_status(self):
with tempfile.TemporaryDirectory() as tmp:
log_path = Path(tmp) / "fg.log"

81
tests/test_profile_suggestions.py
View File

@@ -1,7 +1,8 @@
import unittest
from fgai.logs import parse_log_line
from fgai.profile_suggestions import apply_profile_advice, suggest_stream_profiles
from fgai.profile_suggestions import apply_profile_advice, detect_shape_mismatches, suggest_stream_profiles
from fgai.stream_profiles import RelationshipField, StreamProfile
class ProfileSuggestionTests(unittest.TestCase):
@@ -364,6 +365,84 @@ class ProfileSuggestionTests(unittest.TestCase):
self.assertIn("deny_action", detectors)
self.assertEqual(detectors["deny_action"].get("source"), "value_shape")
def test_name_matched_free_text_field_is_vetoed_from_categorical_and_relationships(self):
messages = [
"GET /checkout/cart/item-{0}?ref=promo&session=abc{0} HTTP/1.1 200",
"POST /api/v2/orders/{0}/confirm referrer=partner-{0} status=accepted",
"DELETE /users/{0}/sessions/active reason=timeout-{0} agent=curl/8.1",
"PUT /inventory/warehouse-{0}/restock qty={0} note=manual-adjustment",
]
events = [
parse_log_line(
f"fgai_stream_id=proxy fgai_stream=Proxy srcip=10.0.0.{index % 7} "
f"context='{messages[index % len(messages)].format(index)}' "
f"timestamp=2026-06-29T10:00:{index:02d}Z"
)
for index in range(1, 40)
]
suggestion = suggest_stream_profiles(events)[0]
profile = suggestion["profile"]
self.assertEqual(suggestion["field_shapes"]["context"], "free_text")
self.assertNotIn("context", profile["entity_fields"])
self.assertNotIn("context", profile["categorical_fields"])
relationship_fields = {item["left"] for item in profile["relationship_fields"]} | {
item["right"] for item in profile["relationship_fields"]
}
self.assertNotIn("context", relationship_fields)
self.assertIn("srcip", profile["entity_fields"])
def test_low_cardinality_name_matched_field_is_not_vetoed(self):
events = [
parse_log_line(
f"fgai_stream_id=firewall fgai_stream=Firewall srcip=10.0.0.{index % 7} "
f"action={'deny' if index % 2 else 'allow'} timestamp=2026-06-29T10:00:{index:02d}Z"
)
for index in range(1, 30)
]
suggestion = suggest_stream_profiles(events)[0]
profile = suggestion["profile"]
self.assertIn("action", profile["categorical_fields"])
def test_existing_profile_relationship_field_now_free_text_is_flagged(self):
field_shapes = {"context": "free_text", "srcip": "ip"}
field_confidence = {"context": 43, "srcip": 90}
existing = StreamProfile(
stream_id="proxy",
name="Proxy",
entity_field="srcip",
timestamp_field="timestamp",
categorical_fields=("context",),
relationship_fields=(RelationshipField(left="srcip", right="context", name="srcip -> context"),),
)
warnings = detect_shape_mismatches(existing, field_shapes, field_confidence)
flagged = {(item["field"], item["role"]) for item in warnings}
self.assertIn(("context", "categorical"), flagged)
self.assertIn(("context", "relationship"), flagged)
self.assertNotIn(("srcip", "relationship"), flagged)
def test_existing_profile_with_fitting_fields_produces_no_warnings(self):
field_shapes = {"action": "categorical_low_cardinality", "srcip": "ip"}
field_confidence = {"action": 85, "srcip": 90}
existing = StreamProfile(
stream_id="firewall",
name="Firewall",
entity_field="srcip",
timestamp_field="timestamp",
categorical_fields=("action",),
relationship_fields=(RelationshipField(left="srcip", right="action", name="srcip -> action"),),
)
self.assertEqual(detect_shape_mismatches(existing, field_shapes, field_confidence), [])
def test_missing_existing_profile_produces_no_warnings(self):
self.assertEqual(detect_shape_mismatches(None, {}, {}), [])
def test_every_suggestion_carries_field_confidence_and_shapes(self):
events = [
parse_log_line(f"fgai_stream_id=firewall fgai_stream=Firewall srcip=10.0.0.{index} dstip=8.8.8.8 dstport=443 action=deny timestamp=2026-06-29T10:00:{index:02d}Z hitcount={index}")