diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index ef8fd72..63ff960 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -107,6 +107,12 @@ calls. ## Known Structural Gaps (found auditing a real deployment) +**Status: all four gaps below are now fixed and verified live** against the +same deployment that produced them (see the Target Architecture section for +what shipped, and `ROADMAP.md` section 0 for the checklist). This section is +kept as the historical record of what the audit actually found, since the +numbers below are what "verified" is measured against. + A live 25-stream deployment (~1M events/day) was audited directly while debugging why its triage queue felt unmanageable. Findings, in order of impact: @@ -159,10 +165,16 @@ lets it become a relationship target, gap 3 means there's no shared state to dampen the noise across related streams, and gap 4 means nothing catches the resulting flood before it reaches the human. Fixing gap 2 alone would have prevented the specific incident that triggered this audit; fixing all four is -what makes the system hold up as more streams are added. +what makes the system hold up as more streams are added. Verified live after +all four shipped: the `context` relationship flood went from 91 deviations to +0, total field deviations from 117 to 17, and triage severity finally +differentiates (10 critical / 8 medium / 7 high, versus 25/25 critical +before). ## Target Architecture +This is now the shipped behavior, not just the target. + - **Shape-first validation everywhere a field is selected for a role**, including name-matched fields, not only as a fallback for unmatched ones. Free-text-shaped fields are structurally excluded from relationship @@ -173,16 +185,31 @@ what makes the system hold up as more streams are added. - **Shared, entity-scoped baselines** for entities that shared-field discovery already recognizes as the same concept across multiple enabled streams, instead of one cold baseline per `(stream, entity, field)` tuple. - A value known-normal in one contributing stream should not register as new - in another. + A value known-normal in one contributing stream does not register as new + in another. Implemented as an additive pooled scope in `baseline.py` + (`shared_entity_pools()` in `profile_suggestions.py` computes which + streams share an entity concept; pooled rows live under a synthetic + `shared_entity:` namespace in the *same* tables, so no schema + migration was needed) covering entity presence/age and known + relationship-target values specifically — the mechanism most directly + responsible for the audited flood. Per-stream distribution baselines + (event-rate/detector bursts, numeric baselines) stay siloed per stream, on + purpose: those distributions genuinely differ by log source. Pooling + arbitrary categorical `rare_value` newness is a deferred follow-up. + Building this also surfaced and fixed a latent, previously-harmless bug in + `ALIAS_CANONICAL_BY_FIELD`'s construction where a later, generic canonical + group (`context`) could silently steal an alias (`hostname`, `dns_query`) + from an earlier, more specific one. - **Cluster before score.** Field deviations get grouped by - `(detector, field, value-pattern)` signature before severity is assigned. - A pattern recurring identically across many entities becomes one clustered - triage entry with an occurrence count and entity breakdown, and — this is a - deliberate inversion of today's behavior — a large, identical cluster is a - signal to score *lower*, since it more often indicates a detector or - profile problem than a security event, while a small, isolated occurrence - keeps its current weight. + `(detector, field)` signature before severity is assigned (value-pattern + was dropped from the originally-planned signature — detector+field alone + already matched the real incident's cross-stream pattern). A pattern + recurring identically across many entities becomes one clustered triage + entry with an occurrence count (`cluster_size`), and — a deliberate + inversion of the old behavior — a cluster of 5 or more entities is capped + at score 25, since it more often indicates a detector or profile problem + than a security event, while a small, isolated occurrence keeps its full + weight. The feedback/suppression layer (`feedback.py`) is not being restructured — it is sound and stays as-is. It benefits automatically from cleaner input: fewer diff --git a/README.md b/README.md index f68f0e0..67a6554 100644 --- a/README.md +++ b/README.md @@ -289,6 +289,23 @@ If the Ollama advisor returns no usable profile for a stream, the row stays on the deterministic profile and is labeled as a heuristic fallback instead of pretending the whole recommendation failed. +When two or more enabled streams recognize the same real-world entity +concept through this alias grouping (for example `srcip` used as the entity +field in both a firewall stream and an IIS stream), SignalScope +automatically pools part of their baseline state instead of learning it +independently per stream: an entity's presence/age history and its known +relationship-target values are shared across the pool, so a value already +established as normal in one contributing stream does not get reported as +"new" the first time a different, related stream happens to observe it. +This is automatic — there is nothing to configure — and it only ever +*reduces* false "already new" flags for values genuinely seen elsewhere in +the pool; a value that is actually new everywhere is still reported exactly +as before. Per-stream statistical baselines (event-rate bursts, detector +bursts, numeric-field distributions) are not pooled and stay independent per +stream, since those distributions legitimately differ by log source. +Active pools are listed under `Diagnostics -> Shared Entity Baselines`, +showing which entity concept is shared and which streams contribute to it. + Recommended profiles can be re-applied to existing profiles. Enable `show existing profiles` and click `Update profile` to append newly discovered entity, categorical, numeric, and detector fields. Existing profile names, field weights, diff --git a/ROADMAP.md b/ROADMAP.md index b8c8010..8eb9cb5 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -45,13 +45,19 @@ this section before spending more effort on either. not just at profile-creation time — flag (never silently rewrite) fields that no longer look fit for their assigned role, and surface a one-click "remove this field from relationships" action from the flag. -- [ ] Pool baseline state by shared entity concept across streams that +- [x] Pool baseline state by shared entity concept across streams that shared-field discovery already recognizes as the same real-world entity (for example a source IP seen in both a firewall stream and an IIS stream), instead of a cold, isolated baseline per stream per field. A value already known-normal in one contributing stream must not register as "new" - the first time a different, related stream happens to see it. Deferred: - needs a `baseline.py` schema change and its own dedicated planning pass. + the first time a different, related stream happens to see it. Implemented + as an additive pooled scope (no schema migration — reuses the existing + per-`stream_id`-keyed tables under a synthetic `shared_entity:` + namespace) covering entity presence/age and known relationship-target + values; per-stream event-rate/detector-burst/numeric-distribution + baselines are intentionally left untouched since those legitimately + differ by log source. Pooling arbitrary categorical `rare_value` newness + is a deferred follow-up, not part of this pass. - [x] Cluster field deviations by `(detector, field)` signature *before* scoring, not after (value-pattern was dropped from the original signature — detector+field alone already matched the real incident's cross-stream @@ -179,10 +185,13 @@ Acceptance: adding a new Graylog stream normally requires a profile template and ## Suggested Next Milestone -Complete section 0 (Structural Rework) first. It was added after auditing a -real production deployment and finding that the existing Detection Quality -and Dashboard Usability work, while individually correct, sits on top of a -discovery/scoring pipeline that lets one misconfigured field flood the triage -queue with dozens of look-alike "critical" entries. Fixing the pipeline is -higher leverage than any further UI work until it's done — see -[`ARCHITECTURE.md`](ARCHITECTURE.md). +Section 0 (Structural Rework) is complete and verified against the real +25-stream production deployment that motivated it: the `context` +relationship flood dropped from 91 deviations to 0, total field deviations +from 117 to 17, and triage severity now differentiates (10 critical / 8 +medium / 7 high, versus 25/25 critical before). See +[`ARCHITECTURE.md`](ARCHITECTURE.md) for the full before/after. Sections 1 +(Detection Quality, in particular baseline confidence tooling) and 6 +(Dashboard Usability, compact incident cards) were previously blocked on +this and are the natural next targets, now that findings are clustered and +scored correctly. diff --git a/src/fgai/baseline.py b/src/fgai/baseline.py index b7b44d4..624ac94 100644 --- a/src/fgai/baseline.py +++ b/src/fgai/baseline.py @@ -10,7 +10,8 @@ from statistics import mean, pstdev from .logs import THREAT_ACTIONS, is_utm_event from .models import LogEvent -from .entities import profile_entities +from .entities import profile_entities, profile_entity_fields +from .profile_suggestions import _semantic_field_group from .detectors import DETECTOR_MINIMUMS, event_detector_categories from .field_shapes import parse_iso_timestamp from .normalization import canonical_value @@ -121,6 +122,15 @@ def _relationship_key(left: str, right: str) -> str: return f"relationship:{left.lower()}->{right.lower()}" +def _pooled_relationship_key(pool_id: str, right: str) -> str: + # Different streams name the same relationship differently (relationship:srcip->dstip + # vs relationship:client_ip->dstip), so the pooled key must be canonicalized on both + # sides — the pool's own group for the left/entity side, and the right side's + # semantic group — or contributions from different streams would land under + # different keys and never actually pool together. + return _relationship_key(pool_id.split(":", 1)[1], _semantic_field_group(right)) + + def _should_store_profile_value(field: str, value: str) -> bool: field = field.lower() if field.startswith("relationship:"): @@ -248,8 +258,10 @@ class BaselineStore: *, observed_at: int | None = None, max_values_per_field: int = DEFAULT_MAX_VALUES_PER_FIELD, + entity_pools: dict[str, dict[str, str]] | None = None, ) -> int: observed_at = observed_at or int(time.time()) + entity_pools = entity_pools or {} pending: dict[tuple[str, str, str, int], list[float]] = defaultdict(lambda: [0, 0.0, 0.0]) temporal_pending: dict[tuple[str, str, str, int, int, int], list[float]] = defaultdict(lambda: [0, 0.0, 0.0]) detector_pending: Counter[tuple[str, str, str, int]] = Counter() @@ -264,14 +276,20 @@ class BaselineStore: fingerprint = hashlib.sha256(f"{stream_id}|{event.raw}".encode("utf-8", errors="replace")).hexdigest() if connection.execute("insert or ignore into profile_seen_events(fingerprint, first_seen) values (?, ?)", (fingerprint, observed_at)).rowcount != 1: continue - entities = profile_entities(event, profile) - if not entities: + entity_pairs = profile_entity_fields(event, profile) + if not entity_pairs: continue + entities = tuple(value for _field, value in entity_pairs) timestamp = _event_epoch(event, observed_at) bucket = timestamp - (timestamp % self.bucket_seconds) moment = datetime.fromtimestamp(timestamp, tz=timezone.utc) fields = [*getattr(profile, "categorical_fields", ()), *getattr(profile, "numeric_fields", ())] numeric = {str(field).lower() for field in getattr(profile, "numeric_fields", ())} + stream_pools = entity_pools.get(stream_id, {}) + for entity_field, entity_value in entity_pairs: + pool_id = stream_pools.get(entity_field) + if pool_id: + pending[(pool_id, entity_value, "__entity_presence__", bucket)][0] += 1 for entity in entities: for detector in event_detector_categories(event): detector_pending[(stream_id, entity, detector, bucket)] += 1 @@ -283,6 +301,9 @@ class BaselineStore: right_value = event.fields.get(right) if left_value and right_value: pending_values[(stream_id, left_value, _relationship_key(left, right), right_value)] += 1 + pool_id = stream_pools.get(left) + if pool_id: + pending_values[(pool_id, left_value, _pooled_relationship_key(pool_id, right), right_value)] += 1 for field in fields: key = (stream_id, entity, str(field).lower(), bucket) value = _number(event.fields.get(key[2])) if key[2] in numeric else 0 @@ -330,7 +351,8 @@ class BaselineStore: on conflict(stream_id, entity, detector, weekday, hour, bucket_start) do update set events=events+excluded.events""", (stream_id, entity, detector, weekday, hour, bucket, count)) return len(pending) - def profile_deviations(self, events: list[LogEvent], profiles: dict[str, object], *, min_training_days: int = 0) -> dict[str, list[dict[str, object]]]: + def profile_deviations(self, events: list[LogEvent], profiles: dict[str, object], *, min_training_days: int = 0, entity_pools: dict[str, dict[str, str]] | None = None) -> dict[str, list[dict[str, object]]]: + entity_pools = entity_pools or {} current: dict[tuple[str, str, str], list[float]] = defaultdict(lambda: [0, 0.0]) entity_events: dict[tuple[str, str], list[LogEvent]] = defaultdict(list) detector_current: Counter[tuple[str, str, str]] = Counter() @@ -477,15 +499,18 @@ class BaselineStore: relationship_limit: Counter[tuple[str, str]] = Counter() for (stream, left_value, left, right, right_value), count in relationship_counts.items(): field = _relationship_key(left, right) - known = connection.execute("select seen_count from profile_values where stream_id=? and entity=? and field=? and value=?", (stream, left_value, field, right_value)).fetchone() - known_total = connection.execute("select coalesce(sum(seen_count), 0) from profile_values where stream_id=? and entity=? and field=?", (stream, left_value, field)).fetchone()[0] + pool_id = entity_pools.get(stream, {}).get(left) + lookup_scope = pool_id or stream + lookup_field = _pooled_relationship_key(pool_id, right) if pool_id else field + known = connection.execute("select seen_count from profile_values where stream_id=? and entity=? and field=? and value=?", (lookup_scope, left_value, lookup_field, right_value)).fetchone() + known_total = connection.execute("select coalesce(sum(seen_count), 0) from profile_values where stream_id=? and entity=? and field=?", (lookup_scope, left_value, lookup_field)).fetchone()[0] if known is not None or int(known_total or 0) < 12: continue if relationship_limit[(stream, left_value)] >= MAX_RARE_VALUES_PER_ENTITY: continue - oldest = connection.execute("select min(bucket_start) from profile_buckets where stream_id=? and entity=?", (stream, left_value)).fetchone()[0] + oldest = connection.execute("select min(bucket_start) from profile_buckets where stream_id=? and entity=?", (lookup_scope, left_value)).fetchone()[0] if oldest is None: - oldest = connection.execute("select min(last_seen) from profile_values where stream_id=? and entity=? and field=?", (stream, left_value, field)).fetchone()[0] + oldest = connection.execute("select min(last_seen) from profile_values where stream_id=? and entity=? and field=?", (lookup_scope, left_value, lookup_field)).fetchone()[0] baseline_age_days = _age_days(oldest, _event_epoch(relationship_samples[(stream, left_value, left, right, right_value)][0], int(time.time()))) if baseline_age_days < min_training_days: continue diff --git a/src/fgai/dashboard.py b/src/fgai/dashboard.py index 2c1ef17..1151e42 100644 --- a/src/fgai/dashboard.py +++ b/src/fgai/dashboard.py @@ -1066,6 +1066,7 @@ async function refresh() { const profileReadiness = (data.profile_readiness || []).map(item => ({...item, profile_name: item.profile_name || profileNames[item.stream_id] || item.stream_id, stream_title: item.stream_name || item.stream_title || streamTitles[item.stream_id] || item.stream_id})); document.getElementById('diagnostics').innerHTML = collapsibleSection('stream-coverage', 'Stream Coverage', table(streamCoverage, [{label:'Stream', key:'stream_name'}, {label:'Enabled', key:'enabled', render:r => r.enabled ? 'yes' : 'no'}, {label:'Profile', render:r => esc(r.profile || 'missing')}, {label:'Entity Field', key:'entity_field'}, {label:'Tracked Fields', key:'tracked_fields'}, {label:'Ready Fields', key:'readiness'}, {label:'Raw Events', key:'events_fetched'}, {label:'Aggregate Events', key:'aggregate_events'}, {label:'Aggregate', key:'aggregate_status'}, {label:'Aggregate Schema', key:'aggregate_schema_properties'}, {label:'Latest Event', key:'latest_event_time'}, {label:'Health', render:r => esc(`${r.health || ''}${r.health_detail ? ': ' + r.health_detail : ''}`)}, {label:'Aggregate Error', render:r => esc(r.aggregate_error || '-')}, {label:'Raw Error', render:r => esc(r.raw_error || '-')}], 'stream-coverage')) + + collapsibleSection('shared-entity-baselines', 'Shared Entity Baselines', (data.shared_entity_baselines || []).length ? table(data.shared_entity_baselines || [], [{label:'Entity concept', key:'group'}, {label:'Shared across streams', render:r => esc((r.stream_names || r.streams || []).join(', '))}], 'shared-entity-baselines') : '

No entity concept is currently recognized across two or more enabled streams. Baselines stay per-stream until that happens.

') + collapsibleSection('correlations', 'Cross-Source Correlations', table(correlations, [{label:'Entity', key:'entity', render:r => esc(`${correlationShortLabel(r)} (${r.entity_type || 'ip'})`)}, {label:'Streams', render:r => esc((r.streams || []).join(', '))}, {label:'Events', key:'events'}, {label:'Security Events', key:'security_events'}], 'correlations')) + collapsibleSection('entities', 'Entities', table(context.source_profiles || [], [{label:'Entity', key:'entity', render:r => entityCell(r)}, {label:'Events', key:'events'}, {label:'UTM', key:'utm_events'}, {label:'Deny', key:'deny_or_threat_actions'}, {label:'Destinations', key:'distinct_destinations'}, {label:'Actions', render:r => esc((r.top_actions || []).join(', '))}], 'entities')) + collapsibleSection('profile-readiness', 'Profile Baseline Readiness', table(profileReadiness, [{label:'Profile', key:'profile_name'}, {label:'Stream', key:'stream_title'}, {label:'Field', key:'field'}, {label:'Buckets', key:'buckets'}, {label:'Age days', key:'age_days'}, {label:'Training days', key:'training_days'}, {label:'Ready', key:'ready', render:r => r.ready ? 'ready' : 'learning'}, {label:'Action', render:r => r.stream_id ? `` : '-'}], 'profile-readiness')) + diff --git a/src/fgai/entities.py b/src/fgai/entities.py index 0aa159a..b1e758a 100644 --- a/src/fgai/entities.py +++ b/src/fgai/entities.py @@ -99,14 +99,20 @@ def profile_entity(event: LogEvent, field: str) -> str: return str(event.fields.get(field.lower(), "")).strip() -def profile_entities(event: LogEvent, profile: object) -> tuple[str, ...]: +def profile_entity_fields(event: LogEvent, profile: object) -> tuple[tuple[str, str], ...]: fields = tuple(str(field) for field in getattr(profile, "entity_fields", ()) if field) or (str(getattr(profile, "entity_field", "")),) - values = [] + pairs: list[tuple[str, str]] = [] + seen_values: set[str] = set() for field in fields: value = profile_entity(event, field) - if value and value not in values: - values.append(value) - return tuple(values) + if value and value not in seen_values: + seen_values.add(value) + pairs.append((field.lower(), value)) + return tuple(pairs) + + +def profile_entities(event: LogEvent, profile: object) -> tuple[str, ...]: + return tuple(value for _field, value in profile_entity_fields(event, profile)) def _timeline_sample(event: LogEvent) -> dict[str, object]: diff --git a/src/fgai/monitor.py b/src/fgai/monitor.py index e263304..e25341c 100644 --- a/src/fgai/monitor.py +++ b/src/fgai/monitor.py @@ -24,7 +24,7 @@ from .llm import ollama_dashboard_assessment, ollama_profile_advice from .logs import local_in_failures, read_events, summarize_events, top_field_values from .mitigation import parse_allowlist, suggest_block_candidates from .policies import audit_policies, read_policies -from .profile_suggestions import apply_profile_advice, suggest_stream_profiles +from .profile_suggestions import apply_profile_advice, shared_entity_pools, suggest_stream_profiles from .recommendations import build_recommendations from .sequences import detect_sequences from .threat_intel import ThreatIntelClient, enrich_ips, is_public_ip @@ -393,6 +393,7 @@ def build_status( runtime_values = config_store.read() if config_exists and config_store else {} runtime_config = _public_runtime_config(runtime_values) if config_store else {} stream_profiles = parse_profiles(runtime_values.get("graylog_stream_profiles", [])) + entity_pools = shared_entity_pools(stream_profiles) stream_titles = _stream_titles(runtime_values) events = read_events(log_path) if Path(log_path).exists() else [] mcp_status: dict[str, object] = {"status": "not_configured"} @@ -527,7 +528,7 @@ def build_status( baseline = BaselineStore(baseline_path) if baseline_path else None profiles = baseline.profiles({event.src_ip for event in events if event.src_ip}) if baseline else {} baseline_training_days = int(runtime_values.get("baseline_training_days", 7) or 7) - field_deviations = baseline.profile_deviations(events, stream_profiles, min_training_days=baseline_training_days) if baseline else {} + field_deviations = baseline.profile_deviations(events, stream_profiles, min_training_days=baseline_training_days, entity_pools=entity_pools) if baseline else {} sequence_findings = detect_sequences(events) for entity, findings in sequence_findings.items(): field_deviations.setdefault(entity, []).extend(findings) @@ -569,6 +570,7 @@ def build_status( events, stream_profiles, max_values_per_field=int(runtime_values.get("baseline_max_values_per_field", 500) or 500), + entity_pools=entity_pools, ) if baseline else 0 ) @@ -701,6 +703,18 @@ def build_status( for suggestion in profile_suggestions for warning in suggestion.get("shape_warnings", []) ] + pool_streams: dict[str, set[str]] = defaultdict(set) + for pool_stream_id, fields in entity_pools.items(): + for pool_id in fields.values(): + pool_streams[pool_id].add(pool_stream_id) + shared_entity_baselines = [ + { + "group": pool_id.split(":", 1)[1], + "streams": sorted(pool_streams[pool_id]), + "stream_names": sorted(_stream_name(sid, stream_titles, stream_profiles.get(sid)) for sid in pool_streams[pool_id]), + } + for pool_id in sorted(pool_streams) + ] status = { "status_schema": 2, "stale": False, @@ -718,6 +732,7 @@ def build_status( "stream_coverage": stream_coverage, "profile_suggestions": profile_suggestions, "profile_shape_warnings": profile_shape_warnings, + "shared_entity_baselines": shared_entity_baselines, "profile_readiness": profile_readiness, "diagnostics": { "top_source_ips": top_field_values(events, "srcip", limit=10), diff --git a/src/fgai/profile_suggestions.py b/src/fgai/profile_suggestions.py index 92023e0..ea92678 100644 --- a/src/fgai/profile_suggestions.py +++ b/src/fgai/profile_suggestions.py @@ -73,11 +73,19 @@ def detect_shape_mismatches(existing: object, field_shapes: dict[str, str], fiel return warnings -ALIAS_CANONICAL_BY_FIELD = { - alias: canonical - for canonical, aliases in FIELD_ALIASES.items() - for alias in aliases -} +def _build_alias_canonical_map() -> dict[str, str]: + # First-wins: FIELD_ALIASES lists specific groups before the generic + # "context" fallback, and some fallback aliases (hostname, dns_query) + # collide with a specific group's own alias — the specific group must + # win, or _semantic_field_group("hostname") would resolve to "context". + mapping: dict[str, str] = {} + for canonical, aliases in FIELD_ALIASES.items(): + for alias in aliases: + mapping.setdefault(alias, canonical) + return mapping + + +ALIAS_CANONICAL_BY_FIELD = _build_alias_canonical_map() ENTITY_PRIORITY = ( "srcip", @@ -717,6 +725,32 @@ def _shared_profile_fields( return list(dict.fromkeys(categorical))[:8], list(dict.fromkeys(numeric))[:5], stats[:16] +def shared_entity_pools(stream_profiles: dict[str, object]) -> dict[str, dict[str, str]]: + """Map each stream's entity fields to a pool id shared with other streams + that recognize the same real-world entity concept, via the same canonical + alias grouping used for cross-stream field-sharing recommendations. + + Returns {stream_id: {entity_field_lower: pool_id}}. A field is included + only when at least one other stream shares its canonical group, so a + stream with no matching peers gets nothing and baseline behavior for it + is unchanged. + """ + group_streams: dict[str, set[str]] = defaultdict(set) + stream_entity_groups: dict[str, dict[str, str]] = {} + for stream_id, profile in stream_profiles.items(): + fields = tuple(str(field).lower() for field in getattr(profile, "entity_fields", ()) if field) + groups = {field: _semantic_field_group(field) for field in fields} + stream_entity_groups[stream_id] = groups + for group in groups.values(): + group_streams[group].add(stream_id) + pools: dict[str, dict[str, str]] = {} + for stream_id, groups in stream_entity_groups.items(): + mapping = {field: f"shared_entity:{group}" for field, group in groups.items() if len(group_streams[group]) >= 2} + if mapping: + pools[stream_id] = mapping + return pools + + def _allowed_fields(suggestion: dict[str, object]) -> set[str]: profile = suggestion.get("profile", {}) if isinstance(suggestion.get("profile"), dict) else {} fields = {str(item.get("field", "")) for item in suggestion.get("common_fields", []) if isinstance(item, dict)} diff --git a/tests/test_baseline.py b/tests/test_baseline.py index a30dd83..aa4c88f 100644 --- a/tests/test_baseline.py +++ b/tests/test_baseline.py @@ -183,6 +183,57 @@ class BaselineTests(unittest.TestCase): self.assertNotIn("alice", store.profile_deviations(burst, profiles, min_training_days=7)) self.assertIn("alice", store.profile_deviations(burst, profiles, min_training_days=0)) + def test_entity_pools_share_known_relationship_values_across_streams(self): + with tempfile.TemporaryDirectory() as directory: + store = BaselineStore(str(Path(directory) / "baseline.sqlite3")) + profiles = parse_profiles([ + { + "stream_id": "firewall", + "entity_field": "srcip", + "entity_fields": ["srcip"], + "categorical_fields": ["action"], + "relationship_fields": [{"left": "srcip", "right": "dstip", "name": "src to dst"}], + }, + { + "stream_id": "iis", + "entity_field": "client_ip", + "entity_fields": ["client_ip"], + "categorical_fields": ["action"], + "relationship_fields": [{"left": "client_ip", "right": "dstip", "name": "client to dst"}], + }, + ]) + entity_pools = { + "firewall": {"srcip": "shared_entity:srcip"}, + "iis": {"client_ip": "shared_entity:srcip"}, + } + + for index in range(12): + # firewall establishes pooled knowledge of srcip=10.0.0.10 -> dstip=8.8.8.8 + fw_event = parse_log_line(f"fgai_stream_id=firewall srcip=10.0.0.10 dstip=8.8.8.8 action=allow baseline={index}") + store.ingest_profile_fields([fw_event], profiles, observed_at=1_700_000_000 + index * 300, entity_pools=entity_pools) + # iis independently establishes its own (unpooled) history for a different destination + iis_history_event = parse_log_line(f"fgai_stream_id=iis client_ip=10.0.0.10 dstip=9.9.9.9 action=allow baseline={index}") + store.ingest_profile_fields([iis_history_event], profiles, observed_at=1_700_000_000 + index * 300) + + iis_event = [parse_log_line("fgai_stream_id=iis client_ip=10.0.0.10 dstip=8.8.8.8 action=allow current=1")] + + # Pooled: dstip=8.8.8.8 is already known via the firewall stream's contribution to the shared pool. + pooled = store.profile_deviations(iis_event, profiles, entity_pools=entity_pools) + self.assertNotIn("new_relationship", [item["detector"] for item in pooled.get("10.0.0.10", [])]) + + # Unpooled: iis's own history only knows dstip=9.9.9.9, so 8.8.8.8 is still reported new — proves + # the feature is additive and today's per-stream behavior is unchanged without entity_pools. + unpooled = store.profile_deviations(iis_event, profiles) + relation = next(item for item in unpooled["10.0.0.10"] if item["detector"] == "new_relationship") + self.assertEqual(relation["field"], "relationship:client_ip->dstip") + self.assertEqual(relation["value"], "8.8.8.8") + + # Genuinely new everywhere: pooling never suppresses a value nobody has ever seen. + genuinely_new = [parse_log_line("fgai_stream_id=iis client_ip=10.0.0.10 dstip=1.2.3.4 current=1")] + still_new = store.profile_deviations(genuinely_new, profiles, entity_pools=entity_pools) + relation = next(item for item in still_new["10.0.0.10"] if item["detector"] == "new_relationship") + self.assertEqual(relation["value"], "1.2.3.4") + def test_custom_relationship_flags_new_user_source_ip(self): with tempfile.TemporaryDirectory() as directory: store = BaselineStore(str(Path(directory) / "baseline.sqlite3")) diff --git a/tests/test_dashboard.py b/tests/test_dashboard.py index dda8654..7f4e910 100644 --- a/tests/test_dashboard.py +++ b/tests/test_dashboard.py @@ -99,6 +99,11 @@ class DashboardTests(unittest.TestCase): self.assertIn("-entity pattern", HTML) self.assertIn("cluster_dampened", HTML) + def test_dashboard_shows_shared_entity_baselines(self): + self.assertIn("shared-entity-baselines", HTML) + self.assertIn("Shared Entity Baselines", HTML) + self.assertIn("shared_entity_baselines", HTML) + def test_dashboard_shows_shape_mismatch_warnings(self): self.assertIn('id="profileShapeWarnings"', HTML) self.assertIn("profile_shape_warnings", HTML) diff --git a/tests/test_monitor.py b/tests/test_monitor.py index bd9fda9..403c099 100644 --- a/tests/test_monitor.py +++ b/tests/test_monitor.py @@ -232,6 +232,49 @@ class MonitorTests(unittest.TestCase): self.assertEqual(status["profile_readiness"][0]["profile_name"], "Fortigate profile") self.assertEqual(status["stream_profiles"][0]["name"], "Fortigate profile") + def test_shared_entity_baselines_lists_streams_pooled_by_entity_concept(self): + with tempfile.TemporaryDirectory() as tmp: + config_path = Path(tmp) / "config.json" + config_path.write_text( + json.dumps( + { + "graylog_streams": [ + {"id": "firewall", "title": "Firewall", "enabled": True}, + {"id": "iis", "title": "IIS", "enabled": True}, + ], + "graylog_stream_profiles": [ + {"stream_id": "firewall", "name": "Firewall profile", "entity_field": "srcip", "categorical_fields": ["action"]}, + {"stream_id": "iis", "name": "IIS profile", "entity_field": "client_ip", "categorical_fields": ["action"]}, + ], + } + ), + encoding="utf-8", + ) + with patch("fgai.monitor.BaselineStore.profile_readiness", return_value=[]): + status = build_status(str(Path(tmp) / "missing.log"), baseline_path=str(Path(tmp) / "baseline.sqlite3"), config_path=str(config_path), incident_path=str(Path(tmp) / "incidents.json")) + + self.assertEqual(len(status["shared_entity_baselines"]), 1) + pool = status["shared_entity_baselines"][0] + self.assertEqual(pool["group"], "srcip") + self.assertEqual(set(pool["streams"]), {"firewall", "iis"}) + + def test_shared_entity_baselines_empty_for_single_stream(self): + with tempfile.TemporaryDirectory() as tmp: + config_path = Path(tmp) / "config.json" + config_path.write_text( + json.dumps( + { + "graylog_streams": [{"id": "firewall", "title": "Firewall", "enabled": True}], + "graylog_stream_profiles": [{"stream_id": "firewall", "name": "Firewall profile", "entity_field": "srcip", "categorical_fields": ["action"]}], + } + ), + encoding="utf-8", + ) + with patch("fgai.monitor.BaselineStore.profile_readiness", return_value=[]): + status = build_status(str(Path(tmp) / "missing.log"), baseline_path=str(Path(tmp) / "baseline.sqlite3"), config_path=str(config_path), incident_path=str(Path(tmp) / "incidents.json")) + + self.assertEqual(status["shared_entity_baselines"], []) + def test_stream_coverage_marks_missing_profiles(self): with tempfile.TemporaryDirectory() as tmp: config_path = Path(tmp) / "config.json" diff --git a/tests/test_profile_suggestions.py b/tests/test_profile_suggestions.py index 7c57832..cbc1d1a 100644 --- a/tests/test_profile_suggestions.py +++ b/tests/test_profile_suggestions.py @@ -1,7 +1,7 @@ import unittest from fgai.logs import parse_log_line -from fgai.profile_suggestions import apply_profile_advice, detect_shape_mismatches, suggest_stream_profiles +from fgai.profile_suggestions import ALIAS_CANONICAL_BY_FIELD, apply_profile_advice, detect_shape_mismatches, shared_entity_pools, suggest_stream_profiles from fgai.stream_profiles import RelationshipField, StreamProfile @@ -456,6 +456,37 @@ class ProfileSuggestionTests(unittest.TestCase): self.assertIn("srcip", suggestion["field_shapes"]) self.assertEqual(suggestion["field_shapes"]["srcip"], "ip") + def test_alias_canonical_map_prefers_specific_group_over_context_fallback(self): + self.assertEqual(ALIAS_CANONICAL_BY_FIELD["hostname"], "hostname") + self.assertEqual(ALIAS_CANONICAL_BY_FIELD["dns_query"], "dns_query") + self.assertEqual(ALIAS_CANONICAL_BY_FIELD["query_domain"], "dns_query") + self.assertEqual(ALIAS_CANONICAL_BY_FIELD["qh"], "dns_query") + + def test_shared_entity_pools_groups_streams_with_the_same_entity_concept(self): + stream_profiles = { + "firewall": StreamProfile(stream_id="firewall", name="Firewall", entity_field="srcip", timestamp_field="timestamp", entity_fields=("srcip",)), + "iis": StreamProfile(stream_id="iis", name="IIS", entity_field="client_ip", timestamp_field="timestamp", entity_fields=("client_ip",)), + "billing": StreamProfile(stream_id="billing", name="Billing", entity_field="actor_id", timestamp_field="timestamp", entity_fields=("actor_id",)), + } + + pools = shared_entity_pools(stream_profiles) + + self.assertEqual(pools["firewall"]["srcip"], pools["iis"]["client_ip"]) + self.assertTrue(pools["firewall"]["srcip"].startswith("shared_entity:")) + self.assertNotIn("billing", pools) + + def test_shared_entity_pools_keeps_source_and_destination_ip_separate(self): + stream_profiles = { + "firewall": StreamProfile(stream_id="firewall", name="Firewall", entity_field="srcip", timestamp_field="timestamp", entity_fields=("srcip", "dstip")), + "proxy": StreamProfile(stream_id="proxy", name="Proxy", entity_field="client_ip", timestamp_field="timestamp", entity_fields=("client_ip", "server_ip")), + } + + pools = shared_entity_pools(stream_profiles) + + self.assertEqual(pools["firewall"]["srcip"], pools["proxy"]["client_ip"]) + self.assertEqual(pools["firewall"]["dstip"], pools["proxy"]["server_ip"]) + self.assertNotEqual(pools["firewall"]["srcip"], pools["firewall"]["dstip"]) + if __name__ == "__main__": unittest.main()