diff --git a/README.md b/README.md index 35c6e64..f68f0e0 100644 --- a/README.md +++ b/README.md @@ -159,6 +159,20 @@ Interpret findings by maturity: - `event_rate_burst`, `auth_failure_burst`, `dns_query_burst`, and `deny_action_burst` compare the current window to historical buckets. +Field deviations that share the same detector and field are clustered before +scoring, not after. If the same `(detector, field)` signature appears across +five or more entities in one poll (`false_positive`/`expected`-suppressed +deviations don't count toward that total), every deviation in the cluster is +tagged with a `cluster_size` and its score is capped at 25, and the Evidence +column shows `[part of a N-entity pattern]` ahead of the reason. This is a +deliberate inversion of the usual "more occurrences = more severe" rule: a +pattern hitting many unrelated entities identically almost always means a +detector or profile problem (a field that shouldn't have been picked as a +relationship target, for example) rather than N separate critical security +events, so it is scored at or below an isolated occurrence instead of above +it. A small, isolated deviation keeps its full score. See +[`ARCHITECTURE.md`](ARCHITECTURE.md) for the incident that motivated this. + The review buttons are part of the detection loop: - `Expected`: known behavior that should stay visible but not keep creating @@ -217,6 +231,16 @@ dashboard status payload carries the underlying per-field shape and confidence values, so a `field_confidence`/`field_shapes` map is available for any field even when it was not selected into the applied profile. +A name match never overrides value shape, even for names on the trusted +priority lists such as `context`. If the observed values for a name-matched +field confidently look like free text (a raw log line or URL, not a +low-cardinality category) it is excluded from entity, categorical, and +relationship-target selection regardless of its name, both for a fresh +recommendation and as a second, independent check when a field reaches +relationship-candidate selection through cross-stream field sharing. A field +with too few samples to classify is never excluded on that basis alone; the +veto only fires on a confident, contradicting shape. + Windows-like streams are recognized from stream names such as `Windows`, `Winlog`, `Security Event Log`, `Sysmon`, and `Powershell`, or from Windows event fields. The MCP search asks for common Winlogbeat/ECS names such as @@ -337,6 +361,15 @@ After the baseline has learned those relationships, a new `username -> srcip` or `username -> hostname` pair is reported as `new_relationship` with sample events. The dashboard profile editor exposes this as `Behavior relationships (JSON)`. +An existing stored profile's `categorical_fields` and `relationship_fields` are +re-checked against current observed shapes on every poll, not just when the +profile was first created — a field can look fine at profile-creation time and +drift into free text later, or the classifier's read of it can simply have been +wrong initially. A mismatch never rewrites the stored profile automatically; +it only shows up under `Recommended Stream Profiles` as a warning naming the +field, the stream, and the confidence behind the mismatch, with a button that +opens that stream's profile editor so a person decides whether to remove it. + SignalScope keeps a common alias map for fields such as source IP, destination IP, ports, action, severity, service/protocol, DNS query, URL, message, and event type. This lets Related Activity and correlations work with firewall/proxy/DNS diff --git a/ROADMAP.md b/ROADMAP.md index 55c3662..b8c8010 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -29,18 +29,18 @@ parts of sections 1 and 6 below (baseline confidence tooling and compact incident cards both depend on findings being clustered correctly first) — do this section before spending more effort on either. -- [ ] Validate name-matched fields against observed value shape too, not just +- [x] Validate name-matched fields against observed value shape too, not just fields that had no name match. A field that matches a trusted name (e.g. `context`) but is actually free-text (a full raw log line, confirmed in 12 of 25 audited stream profiles) must not be trusted just because its name matched a known-useful alias. -- [ ] Make relationship-candidate selection shape-aware: never propose a +- [x] Make relationship-candidate selection shape-aware: never propose a free-text-shaped field as a relationship target, regardless of how it entered `categorical_fields`. This alone was responsible for 17 of 25 entries in one real triage queue snapshot (91 individual `new_relationship` deviations from a single misconfigured `*->context` pairing, repeated across 5 streams). -- [ ] Re-validate existing stored profiles' `relationship_fields` and +- [x] Re-validate existing stored profiles' `relationship_fields` and `categorical_fields` against current observed shapes on an ongoing basis, not just at profile-creation time — flag (never silently rewrite) fields that no longer look fit for their assigned role, and surface a one-click @@ -50,18 +50,22 @@ this section before spending more effort on either. (for example a source IP seen in both a firewall stream and an IIS stream), instead of a cold, isolated baseline per stream per field. A value already known-normal in one contributing stream must not register as "new" - the first time a different, related stream happens to see it. -- [ ] Cluster field deviations by `(detector, field, value-pattern)` - signature *before* scoring, not after. A pattern recurring identically - across many entities must surface as one clustered item with an occurrence - count and a "how many entities" breakdown, not N independently-scored - "critical" rows that all have to be discovered, selected, and bulk-cleared - by hand after the fact. -- [ ] Recalibrate severity so cluster size is itself a signal: a pattern - hitting many unrelated entities identically is usually a - configuration/detector problem, not N separate critical incidents, and - should score at or below an isolated occurrence, never above it. -- [ ] Update README usage documentation alongside each item above as it + the first time a different, related stream happens to see it. Deferred: + needs a `baseline.py` schema change and its own dedicated planning pass. +- [x] Cluster field deviations by `(detector, field)` signature *before* + scoring, not after (value-pattern was dropped from the original signature — + detector+field alone already matched the real incident's cross-stream + pattern, and adding value-pattern would have under-clustered near-identical + but not-byte-identical raw messages). A pattern recurring identically + across many entities surfaces as one clustered item with an occurrence + count (`cluster_size`) shown inline in the Evidence column, not N + independently-scored "critical" rows that all have to be discovered, + selected, and bulk-cleared by hand after the fact. +- [x] Recalibrate severity so cluster size is itself a signal: a pattern + hitting five or more entities identically is capped at score 25 regardless + of its uncapped score, so it never outranks an isolated, uncapped + occurrence. +- [x] Update README usage documentation alongside each item above as it ships — a behavior change is not done until it is documented. Acceptance: replaying the dataset that produced the original 25-entry, diff --git a/src/fgai/dashboard.py b/src/fgai/dashboard.py index dce26a3..2c1ef17 100644 --- a/src/fgai/dashboard.py +++ b/src/fgai/dashboard.py @@ -193,7 +193,7 @@ HTML = """

Events and Anomalies

Baseline and Stream Health

Operator Guidance

Waiting for monitor data.

Correlation Map

Waiting for correlated entities.

Investigation Incidents

Anomalies

Recommendations

AI Assessment

LLM assessment disabled.

Triage Queue

Field Baseline Deviations

Active Feedback Rules

Related Activity Across Sources
Block Candidates
Threat Intelligence
Policy Findings

Diagnostics

-

Recommended Stream Profiles

Waiting for observed stream data.

Installed Ollama Models

Loading local Ollama models.

Runtime Configuration

+

Recommended Stream Profiles

Waiting for observed stream data.

Installed Ollama Models

Loading local Ollama models.

Runtime Configuration

How To Use SignalScope

1. Normal workflow

  1. Settings: connect Graylog MCP and enable streams.
  2. Settings: apply recommended profiles for missing streams.
  3. Diagnostics: confirm raw samples, aggregate counts, and profile readiness.
  4. Overview: use Operator Guidance, incidents, trends, and correlation map.
  5. Findings: review only high-signal deviations first, then mark decisions.

2. Stream health

  • ready: profile exists, events are arriving, and baseline fields are ready.
  • learning: profile exists and events arrive, but baseline age or buckets are still too low.
  • missing_profile: stream is enabled but no profile exists. Apply or edit one.
  • no_events: stream is enabled but the current poll has no raw sample events.
  • partial_fetch: Graylog returned only part of the requested raw sample.

3. Ready fields

0/8 means 8 profile fields are tracked but none are mature enough yet. A field needs at least 12 baseline buckets and the configured Baseline training days before it is ready.

During learning, treat findings as signals to tune profiles, not as final alerts.

4. Profiles

  • Entity fields define who or what behavior is tracked, such as user, host, source IP, or application actor.
  • Baseline fields define the changing behavior to learn, such as action, event ID, service, URL, status, or counters.
  • Relationships learn pairs such as username to srcip or host to process.
  • Detectors add burst checks for auth failures, DNS queries, and deny actions.

5. Findings

  • Start with Triage Queue and incidents, not raw long tables.
  • Open evidence details before confirming a finding.
  • Use Expected for known behavior, False positive for bad signal, Confirmed for real investigation items.
  • Use expiry when a behavior is expected only temporarily.

6. High EPS / MCP

  • Use aggregate or auto fetch mode for high EPS streams.
  • Keep raw samples small enough for context; aggregate counts represent the full window.
  • Sample capped is normal in aggregate mode. Truncated raw mode means you may miss context.
  • If MCP is stale, the UI shows cached status so you can still inspect previous findings.

7. Ollama

  • Dashboard assessment summarizes current evidence.
  • Profile advisor maps unknown/custom fields and suggests relationships.
  • Ollama advice is constrained to fields discovered from Graylog; unknown fields are rejected.

8. Baseline DB size

  • Baseline DB size is the local SQLite behavior baseline.
  • Retention deletes old rows, but SQLite only returns disk space after manual VACUUM.
  • If the DB grows fast, lower bucket retention, value retention, or max values per entity field in Settings.
  • Stop the service before running signalscope baseline-maintenance --vacuum.

9. What to fix first

  1. No streams enabled.
  2. Enabled streams with missing profiles.
  3. Enabled streams with zero raw events.
  4. Profiles stuck at 0 ready fields after the training window.
  5. Too many repeated findings without review feedback.
  6. Baseline DB growing without maintenance.
@@ -677,6 +677,8 @@ async function refresh() { const pendingProfileSuggestions = window.profileSuggestions.filter(item => !item.profile_exists); const visibleProfileSuggestions = showExistingProfiles ? window.profileSuggestions : pendingProfileSuggestions; document.getElementById('profileSuggestionSummary').textContent = `${pendingProfileSuggestions.length} missing profile(s), ${window.profileSuggestions.length - pendingProfileSuggestions.length} existing.`; + const shapeWarnings = data.profile_shape_warnings || []; + document.getElementById('profileShapeWarnings').innerHTML = shapeWarnings.map(w => `

'${esc(w.field)}' in ${esc(w.stream_name || w.stream_id)}'s ${esc(w.role)} fields now looks like free text (${esc(String(w.confidence))}% confidence) — consider removing it from the profile.

`).join(''); document.getElementById('profileSuggestions').innerHTML = visibleProfileSuggestions.length ? table(visibleProfileSuggestions, [ {label:'Stream', key:'stream_name'}, {label:'Events', key:'events'}, @@ -894,7 +896,7 @@ async function refresh() { if (rawFieldRows.length) uiCache.fieldRows = rawFieldRows; document.getElementById('fieldDeviations').innerHTML = `${fieldRowsCached ? '

No current field deviations in this poll; showing cached findings from the last non-empty poll.

' : ''}` + table(fieldRows, [ {label:'', render:r => { const key = deviationKey(r); return ``; }}, - {label:'Entity', key:'entity', render:r => entityCell(r)}, {label:'Stream', key:'stream_title'}, {label:'Detector', key:'detector'}, {label:'Field', key:'field'}, {label:'Score', key:'score'}, {label:'Baseline', render:r => esc(`${r.confidence || '-'}; ${r.baseline_age_days ?? 0}d; ${r.baseline_samples || 0} samples`)}, {label:'Review', render:r => esc(r.feedback || 'unreviewed')}, {label:'Suggested', render:suggestReview}, {label:'Evidence', render:r => { const weighted = r.weight && r.weight !== 1 ? `; weighted ${r.base_score ?? r.score} x ${r.weight}` : ''; const summary=esc(`${r.reason}; current ${r.current ?? '-'} vs baseline ${r.baseline ?? '-'}; scope ${r.baseline_scope ?? '-'}${weighted}; values: ${(r.sample_values || []).join(', ') || '-'}`); const events=(r.sample_events || []).map(item => `${esc(`${item.timestamp} | ${item.source} -> ${item.destination} | ${item.action} ${item.service} | ${item.value} | ${item.message}`)}${item.graylog_query ? `
${esc(item.graylog_query)}${graylogEvidenceLink(item.graylog_query, configuration)}` : ''}`).join('
'); const id=`deviation:${r.entity}:${r.stream_id}:${r.field}:${r.value || ''}`; return events ? `
${summary}

${events}

` : summary; }}, {label:'Review action', render:r => `
`} + {label:'Entity', key:'entity', render:r => entityCell(r)}, {label:'Stream', key:'stream_title'}, {label:'Detector', key:'detector'}, {label:'Field', key:'field'}, {label:'Score', key:'score'}, {label:'Baseline', render:r => esc(`${r.confidence || '-'}; ${r.baseline_age_days ?? 0}d; ${r.baseline_samples || 0} samples`)}, {label:'Review', render:r => esc(r.feedback || 'unreviewed')}, {label:'Suggested', render:suggestReview}, {label:'Evidence', render:r => { const weighted = r.weight && r.weight !== 1 ? `; weighted ${r.base_score ?? r.score} x ${r.weight}` : ''; const clusterNote = r.cluster_size >= 5 ? `[part of a ${r.cluster_size}-entity pattern${r.cluster_dampened ? '; score capped' : ''}] ` : ''; const summary=esc(`${clusterNote}${r.reason}; current ${r.current ?? '-'} vs baseline ${r.baseline ?? '-'}; scope ${r.baseline_scope ?? '-'}${weighted}; values: ${(r.sample_values || []).join(', ') || '-'}`); const events=(r.sample_events || []).map(item => `${esc(`${item.timestamp} | ${item.source} -> ${item.destination} | ${item.action} ${item.service} | ${item.value} | ${item.message}`)}${item.graylog_query ? `
${esc(item.graylog_query)}${graylogEvidenceLink(item.graylog_query, configuration)}` : ''}`).join('
'); const id=`deviation:${r.entity}:${r.stream_id}:${r.field}:${r.value || ''}`; return events ? `
${summary}

${events}

` : summary; }}, {label:'Review action', render:r => `
`} ], 'field-deviations'); const bulkReviewBar = document.getElementById('bulkReviewBar'); const updateBulkReviewBar = () => { @@ -1362,6 +1364,11 @@ document.getElementById('streamPicker').addEventListener('click', event => { if (!button) return; editStreamProfile(button.dataset.id, button.dataset.title); }); +document.getElementById('profileShapeWarnings').addEventListener('click', event => { + const button = event.target.closest('.edit-profile'); + if (!button) return; + editStreamProfile(button.dataset.id, button.dataset.title); +}); document.getElementById('profileSuggestions').addEventListener('click', event => { const deleteButton = event.target.closest('.delete-stream-profile'); if (deleteButton) { diff --git a/src/fgai/monitor.py b/src/fgai/monitor.py index 3ff1178..e263304 100644 --- a/src/fgai/monitor.py +++ b/src/fgai/monitor.py @@ -5,6 +5,7 @@ import json import signal import threading import time +from collections import defaultdict from pathlib import Path from .anomaly import anomaly_summary, detect_source_anomalies @@ -251,6 +252,40 @@ def _detect_profile_drift( return drift +CLUSTER_SIZE_DAMPEN_THRESHOLD = 5 +CLUSTER_DAMPENED_SCORE_CAP = 25 + + +def _cluster_field_deviations(field_deviations: dict[str, list[dict[str, object]]]) -> None: + """Recognize the same (detector, field) signature repeating across many + entities as one systemic pattern instead of N independent findings. + + Mutates deviations in place: tags cluster_size, and caps score for large + clusters — a signature hitting many entities identically usually means a + detector/profile problem, not N separate critical events. Signature + deliberately excludes stream_id: the incident this guards against (a + misconfigured relationship field) recurred identically across multiple + streams, and that cross-stream repetition is itself part of the pattern. + Already-suppressed deviations don't count toward another entity's cluster + size, so a false-positive rule doesn't inflate the appearance of a pattern. + """ + signature_entities: dict[tuple[str, str], set[str]] = defaultdict(set) + for entity, deviations in field_deviations.items(): + for deviation in deviations: + if deviation.get("feedback") in SUPPRESSING_STATUSES: + continue + signature = (str(deviation.get("detector", "")), str(deviation.get("field", ""))) + signature_entities[signature].add(entity) + for entity, deviations in field_deviations.items(): + for deviation in deviations: + signature = (str(deviation.get("detector", "")), str(deviation.get("field", ""))) + cluster_size = len(signature_entities.get(signature, ())) + deviation["cluster_size"] = cluster_size + if cluster_size >= CLUSTER_SIZE_DAMPEN_THRESHOLD: + deviation["score"] = min(int(deviation.get("score", 0) or 0), CLUSTER_DAMPENED_SCORE_CAP) + deviation["cluster_dampened"] = True + + def _stream_name(stream_id: str, stream_titles: dict[str, str], profile: object | None = None) -> str: return stream_titles.get(stream_id) or getattr(profile, "name", "") or stream_id @@ -524,6 +559,7 @@ def build_status( deviation["feedback_rule_id"] = match.get("id", "") if match["status"] in SUPPRESSING_STATUSES: deviation["score"] = 0 + _cluster_field_deviations(field_deviations) generalization_min_values = int(runtime_values.get("feedback_generalization_min_values", 3) or 3) feedback_generalization_suggestions = suggest_generalizations(feedback, min_values=generalization_min_values) anomalies = detect_source_anomalies(events, limit=anomaly_limit, baselines=profiles, field_deviations=field_deviations, feedback=feedback) @@ -660,6 +696,11 @@ def build_status( summary["total"] = int(mcp_status.get("aggregate_events", 0) or 0) summary["raw_sample_total"] = len(events) summary["aggregate_backed"] = True + profile_shape_warnings = [ + {"stream_id": suggestion.get("stream_id", ""), "stream_name": suggestion.get("stream_name", ""), **warning} + for suggestion in profile_suggestions + for warning in suggestion.get("shape_warnings", []) + ] status = { "status_schema": 2, "stale": False, @@ -676,6 +717,7 @@ def build_status( "profile_drift": profile_drift, "stream_coverage": stream_coverage, "profile_suggestions": profile_suggestions, + "profile_shape_warnings": profile_shape_warnings, "profile_readiness": profile_readiness, "diagnostics": { "top_source_ips": top_field_values(events, "srcip", limit=10), diff --git a/src/fgai/profile_suggestions.py b/src/fgai/profile_suggestions.py index b037960..92023e0 100644 --- a/src/fgai/profile_suggestions.py +++ b/src/fgai/profile_suggestions.py @@ -11,6 +11,68 @@ from .normalization import FIELD_ALIASES SHAPE_ENTITY_CONFIDENCE = 60 +_SHAPE_VETO = { + "entity": {"free_text"}, + "categorical": {"free_text"}, + "numeric": {"free_text", "categorical_low_cardinality"}, + "relationship": {"free_text"}, +} + +# classify_field_shape() gives free_text a fixed 0.4 fit ratio and no name +# bonus (field_shapes.py), so a genuine free-text verdict tops out around +# 28-43 confidence — well under the 60 that a clean ip/categorical/numeric +# fit reaches. The fallback "nothing fit" case (classify_field_shape's +# `if not scores` branch) can't exceed 15. 20 cleanly separates a real +# free-text fit from that weak fallback while staying reachable. +_SHAPE_VETO_CONFIDENCE = { + "free_text": 20, +} +_DEFAULT_VETO_CONFIDENCE = 60 + + +def _shape_vetoes(field: str, role: str, field_shapes: dict[str, str], field_confidence: dict[str, int]) -> bool: + """True when observed values confidently contradict a name-matched field's role. + + Only vetoes on positive evidence (a confident, wrong-looking shape) — + fields with no shape info yet (insufficient samples) are never vetoed. + """ + shape = field_shapes.get(field) + if shape is None or shape not in _SHAPE_VETO.get(role, set()): + return False + threshold = _SHAPE_VETO_CONFIDENCE.get(shape, _DEFAULT_VETO_CONFIDENCE) + return field_confidence.get(field, 0) >= threshold + + +def detect_shape_mismatches(existing: object, field_shapes: dict[str, str], field_confidence: dict[str, int]) -> list[dict[str, object]]: + """Flag an existing stored profile's fields whose current observed shape no + longer fits their assigned role. Flag-only — never rewrites the stored + profile; a human reviews and removes the field from the dashboard warning. + """ + if existing is None: + return [] + warnings: list[dict[str, object]] = [] + seen: set[tuple[str, str]] = set() + + def flag(field: str, role: str) -> None: + key = (field, role) + if not field or key in seen or not _shape_vetoes(field, role, field_shapes, field_confidence): + return + seen.add(key) + warnings.append({ + "field": field, + "role": role, + "shape": field_shapes.get(field), + "confidence": field_confidence.get(field, 0), + }) + + for field in getattr(existing, "categorical_fields", ()): + flag(field, "categorical") + for relationship in getattr(existing, "relationship_fields", ()): + flag(relationship.left, "relationship") + flag(relationship.right, "relationship") + return warnings + + ALIAS_CANONICAL_BY_FIELD = { alias: canonical for canonical, aliases in FIELD_ALIASES.items() @@ -392,7 +454,7 @@ def _generic_entity_fields(coverage: Counter[str], unique_values: dict[str, set[ for field, count in coverage.most_common(): if len(output) >= 4: break - if field in output or field in IGNORED_DISCOVERY_FIELDS: + if field in output or field in IGNORED_DISCOVERY_FIELDS or _shape_vetoes(field, "entity", field_shapes, field_confidence): continue shape_match = field_shapes.get(field) == "ip" and field_confidence.get(field, 0) >= SHAPE_ENTITY_CONFIDENCE if not shape_match and not _looks_like_entity_field(field): @@ -411,7 +473,7 @@ def _generic_categorical_fields(coverage: Counter[str], unique_values: dict[str, for field, count in coverage.most_common(): if len(output) >= 12: break - if field in output or field in IGNORED_DISCOVERY_FIELDS: + if field in output or field in IGNORED_DISCOVERY_FIELDS or _shape_vetoes(field, "categorical", field_shapes, field_confidence): continue coverage_ratio = count / max(1, total) cardinality = len(unique_values[field]) @@ -428,7 +490,7 @@ def _generic_numeric_fields(coverage: Counter[str], numeric_counts: Counter[str] for field, count in coverage.most_common(): if len(output) >= 8: break - if field in output or field in IGNORED_DISCOVERY_FIELDS: + if field in output or field in IGNORED_DISCOVERY_FIELDS or _shape_vetoes(field, "numeric", field_shapes, field_confidence): continue shape_match = field_shapes.get(field) == "numeric" and field_confidence.get(field, 0) >= SHAPE_ENTITY_CONFIDENCE if shape_match or (count / max(1, total) >= 0.1 and numeric_counts[field] / max(1, count) >= 0.95): @@ -479,9 +541,11 @@ def _relationship_candidates( numeric_fields: list[str], *, field_shapes: dict[str, str] | None = None, + field_confidence: dict[str, int] | None = None, limit: int = 6, ) -> list[dict[str, str]]: field_shapes = field_shapes or {} + field_confidence = field_confidence or {} def usable(field: str) -> bool: return ( bool(field) @@ -513,6 +577,10 @@ def _relationship_candidates( if _field_matches(field, {"dstip", "*_ip"}, {"dst", "dest", "destination", "server"}) ] destination_fields.extend(field for field in categorical if field.lower() in {"dstport", "destination_port", "dest_port", "service", "application", "app", "url", "query_domain", "dns_query"}) + # Defense in depth: a field can reach here via shared_categorical or a future + # code path that bypasses the name-priority seed veto applied upstream, so + # re-check shape immediately before it can become a relationship target. + destination_fields = [field for field in destination_fields if not _shape_vetoes(field, "categorical", field_shapes, field_confidence)] # Ambiguous-name tie-break: two ip-shaped entity fields that neither name-matched # source nor destination are split using which one leans more private (source-like) @@ -546,6 +614,7 @@ def _relationship_candidates( if field not in entities and not _looks_like_time_field(field) and not any(token in field.lower() for token in ("gl2_", "message_id", "sort_field", "accounted_message_size", "processing_timestamp")) + and not _shape_vetoes(field, "categorical", field_shapes, field_confidence) ] for left in entities: for right in behavior_fields[:4]: @@ -858,17 +927,20 @@ def suggest_stream_profiles(events: list[LogEvent], *, existing_profiles: dict[s field_shapes[field] = shape.primary_shape field_confidence[field] = shape.confidence shared_categorical, shared_numeric, shared_stats = _shared_profile_fields(coverage, unique_values, numeric_counts, total, stream_field_counts, stream_semantic_counts, stream_semantic_fields) - entity_fields = _generic_entity_fields(coverage, unique_values, total, _pick_present(entity_priority, coverage, total, min_ratio=0.02, limit=12 if is_windows else 3), field_shapes=field_shapes, field_confidence=field_confidence) + entity_seed = [field for field in _pick_present(entity_priority, coverage, total, min_ratio=0.02, limit=12 if is_windows else 3) if not _shape_vetoes(field, "entity", field_shapes, field_confidence)] + entity_fields = _generic_entity_fields(coverage, unique_values, total, entity_seed, field_shapes=field_shapes, field_confidence=field_confidence) timestamp = next(iter(_pick_present(TIME_PRIORITY, coverage, total, min_ratio=0.02, limit=1)), "") if not timestamp: timestamp = next((field for field, count in coverage.most_common() if _looks_like_time_field(field) and count / max(1, total) >= 0.02), "timestamp") categorical_seed = list(dict.fromkeys([*_pick_present(categorical_priority, coverage, total, min_ratio=0.02, limit=10 if is_windows else 8), *shared_categorical])) + categorical_seed = [field for field in categorical_seed if not _shape_vetoes(field, "categorical", field_shapes, field_confidence)] categorical = _generic_categorical_fields(coverage, unique_values, total, categorical_seed, field_shapes=field_shapes, field_confidence=field_confidence) numeric_seed = [ field for field in numeric_priority if coverage[field] and numeric_counts[field] / max(1, coverage[field]) >= 0.8 ][:5] numeric_seed = list(dict.fromkeys([*numeric_seed, *shared_numeric])) + numeric_seed = [field for field in numeric_seed if not _shape_vetoes(field, "numeric", field_shapes, field_confidence)] numeric = numeric_seed if is_windows else _generic_numeric_fields(coverage, numeric_counts, total, numeric_seed, field_shapes=field_shapes, field_confidence=field_confidence) if not entity_fields: entity_fields = [field for field, _count in coverage.most_common() if field not in IGNORED_DISCOVERY_FIELDS][:1] @@ -912,7 +984,7 @@ def suggest_stream_profiles(events: list[LogEvent], *, existing_profiles: dict[s detectors.setdefault(name, hint) if not detectors: detectors = {"deny_action": {"enabled": True, "minimum": 10, "z_threshold": 3.0}} - relationship_fields = _relationship_candidates(coverage, unique_values, total, entity_fields, categorical, numeric, field_shapes=field_shapes) + relationship_fields = _relationship_candidates(coverage, unique_values, total, entity_fields, categorical, numeric, field_shapes=field_shapes, field_confidence=field_confidence) high_coverage = [ {"field": field, "coverage": round(count / max(1, total), 2), "unique_values": len(unique_values[field])} for field, count in coverage.most_common(12) @@ -939,12 +1011,14 @@ def suggest_stream_profiles(events: list[LogEvent], *, existing_profiles: dict[s ], } existing = existing_profiles.get(stream_id) + shape_warnings = detect_shape_mismatches(existing, field_shapes, field_confidence) score = min(100, 30 + len(entity_fields) * 15 + min(20, len(categorical) * 3) + min(15, len(detectors) * 5)) suggestions.append({ "stream_id": stream_id, "stream_name": names.get(stream_id, stream_id), "events": total, "profile_exists": bool(existing), + "shape_warnings": shape_warnings, "confidence": "high" if score >= 75 and total >= 100 else "medium" if score >= 55 and total >= 20 else "low", "score": score, "field_confidence": field_confidence, diff --git a/tests/test_dashboard.py b/tests/test_dashboard.py index 5d5fe98..dda8654 100644 --- a/tests/test_dashboard.py +++ b/tests/test_dashboard.py @@ -94,6 +94,17 @@ class DashboardTests(unittest.TestCase): self.assertIn("openDetailIds", HTML) self.assertIn("addEventListener('toggle'", HTML) + def test_dashboard_annotates_field_deviations_with_cluster_size(self): + self.assertIn("cluster_size", HTML) + self.assertIn("-entity pattern", HTML) + self.assertIn("cluster_dampened", HTML) + + def test_dashboard_shows_shape_mismatch_warnings(self): + self.assertIn('id="profileShapeWarnings"', HTML) + self.assertIn("profile_shape_warnings", HTML) + self.assertIn("looks like free text", HTML) + self.assertIn("Open profile editor", HTML) + def test_dashboard_has_anomaly_bulk_bar(self): self.assertIn('id="anomalyBulkBar"', HTML) self.assertIn('anomaly-select', HTML) diff --git a/tests/test_monitor.py b/tests/test_monitor.py index 4b8be39..bd9fda9 100644 --- a/tests/test_monitor.py +++ b/tests/test_monitor.py @@ -8,7 +8,7 @@ from fgai.config import ConfigStore from fgai.feedback import FeedbackStore from fgai.history import StatusSnapshotStore from fgai.history import FieldDiscoveryStore -from fgai.monitor import add_llm_assessment, build_status, cached_status_with_error, monitor_cycle_timeout_seconds, write_refreshing_status, write_status +from fgai.monitor import add_llm_assessment, build_status, cached_status_with_error, monitor_cycle_timeout_seconds, write_refreshing_status, write_status, _cluster_field_deviations class MonitorTests(unittest.TestCase): @@ -39,6 +39,75 @@ class MonitorTests(unittest.TestCase): self.assertFalse(status["status_cache"]["served_from_cache"]) self.assertGreaterEqual(len(status["anomalies"]), 1) + def test_cluster_field_deviations_dampens_large_identical_clusters(self): + field_deviations = { + f"10.0.0.{index}": [{"detector": "relationship", "field": "client_ip->context", "score": 40}] + for index in range(1, 6) + } + + _cluster_field_deviations(field_deviations) + + for entity, deviations in field_deviations.items(): + deviation = deviations[0] + self.assertEqual(deviation["cluster_size"], 5) + self.assertEqual(deviation["score"], 25) + self.assertTrue(deviation["cluster_dampened"]) + + def test_cluster_field_deviations_leaves_small_clusters_untouched(self): + field_deviations = { + f"10.0.0.{index}": [{"detector": "relationship", "field": "client_ip->context", "score": 40}] + for index in range(1, 4) + } + + _cluster_field_deviations(field_deviations) + + for entity, deviations in field_deviations.items(): + deviation = deviations[0] + self.assertEqual(deviation["cluster_size"], 3) + self.assertEqual(deviation["score"], 40) + self.assertNotIn("cluster_dampened", deviation) + + def test_cluster_field_deviations_excludes_suppressed_entities_from_cluster_size(self): + field_deviations = { + f"10.0.0.{index}": [{"detector": "relationship", "field": "client_ip->context", "score": 40}] + for index in range(1, 5) + } + field_deviations["10.0.0.99"] = [{"detector": "relationship", "field": "client_ip->context", "score": 40, "feedback": "false_positive"}] + + _cluster_field_deviations(field_deviations) + + self.assertEqual(field_deviations["10.0.0.1"][0]["cluster_size"], 4) + self.assertNotIn("cluster_dampened", field_deviations["10.0.0.1"][0]) + self.assertEqual(field_deviations["10.0.0.99"][0]["score"], 40) + + def test_cluster_field_deviations_dampen_large_clusters_in_build_status(self): + with tempfile.TemporaryDirectory() as tmp: + srcips = [f"203.0.113.{index}" for index in range(1, 6)] + log_path = Path(tmp) / "fg.log" + log_path.write_text( + "\n".join( + f"type=utm subtype=ips srcip={ip} dstip=10.0.0.10 service=https action=blocked severity=critical" + for ip in srcips + ), + encoding="utf-8", + ) + canned_deviations = { + ip: [{"detector": "relationship", "field": "client_ip->context", "stream_id": "utm", "score": 40, "reason": "new relationship: client_ip->context"}] + for ip in srcips + } + + with patch("fgai.monitor.BaselineStore.profile_deviations", return_value=canned_deviations): + status = build_status(str(log_path), baseline_path=str(Path(tmp) / "baseline.sqlite3"), incident_path=str(Path(tmp) / "incidents.json")) + + for ip in srcips: + deviation = status["field_deviations"][ip][0] + self.assertEqual(deviation["cluster_size"], 5) + self.assertEqual(deviation["score"], 25) + self.assertTrue(deviation["cluster_dampened"]) + + anomalies_by_subject = {item["subject"]: item for item in status["anomalies"]} + self.assertIn("new relationship: client_ip->context", anomalies_by_subject[srcips[0]]["reasons"]) + def test_entity_scope_feedback_suppresses_anomaly_in_build_status(self): with tempfile.TemporaryDirectory() as tmp: log_path = Path(tmp) / "fg.log" diff --git a/tests/test_profile_suggestions.py b/tests/test_profile_suggestions.py index 031108f..7c57832 100644 --- a/tests/test_profile_suggestions.py +++ b/tests/test_profile_suggestions.py @@ -1,7 +1,8 @@ import unittest from fgai.logs import parse_log_line -from fgai.profile_suggestions import apply_profile_advice, suggest_stream_profiles +from fgai.profile_suggestions import apply_profile_advice, detect_shape_mismatches, suggest_stream_profiles +from fgai.stream_profiles import RelationshipField, StreamProfile class ProfileSuggestionTests(unittest.TestCase): @@ -364,6 +365,84 @@ class ProfileSuggestionTests(unittest.TestCase): self.assertIn("deny_action", detectors) self.assertEqual(detectors["deny_action"].get("source"), "value_shape") + def test_name_matched_free_text_field_is_vetoed_from_categorical_and_relationships(self): + messages = [ + "GET /checkout/cart/item-{0}?ref=promo&session=abc{0} HTTP/1.1 200", + "POST /api/v2/orders/{0}/confirm referrer=partner-{0} status=accepted", + "DELETE /users/{0}/sessions/active reason=timeout-{0} agent=curl/8.1", + "PUT /inventory/warehouse-{0}/restock qty={0} note=manual-adjustment", + ] + events = [ + parse_log_line( + f"fgai_stream_id=proxy fgai_stream=Proxy srcip=10.0.0.{index % 7} " + f"context='{messages[index % len(messages)].format(index)}' " + f"timestamp=2026-06-29T10:00:{index:02d}Z" + ) + for index in range(1, 40) + ] + + suggestion = suggest_stream_profiles(events)[0] + profile = suggestion["profile"] + + self.assertEqual(suggestion["field_shapes"]["context"], "free_text") + self.assertNotIn("context", profile["entity_fields"]) + self.assertNotIn("context", profile["categorical_fields"]) + relationship_fields = {item["left"] for item in profile["relationship_fields"]} | { + item["right"] for item in profile["relationship_fields"] + } + self.assertNotIn("context", relationship_fields) + self.assertIn("srcip", profile["entity_fields"]) + + def test_low_cardinality_name_matched_field_is_not_vetoed(self): + events = [ + parse_log_line( + f"fgai_stream_id=firewall fgai_stream=Firewall srcip=10.0.0.{index % 7} " + f"action={'deny' if index % 2 else 'allow'} timestamp=2026-06-29T10:00:{index:02d}Z" + ) + for index in range(1, 30) + ] + + suggestion = suggest_stream_profiles(events)[0] + profile = suggestion["profile"] + + self.assertIn("action", profile["categorical_fields"]) + + def test_existing_profile_relationship_field_now_free_text_is_flagged(self): + field_shapes = {"context": "free_text", "srcip": "ip"} + field_confidence = {"context": 43, "srcip": 90} + existing = StreamProfile( + stream_id="proxy", + name="Proxy", + entity_field="srcip", + timestamp_field="timestamp", + categorical_fields=("context",), + relationship_fields=(RelationshipField(left="srcip", right="context", name="srcip -> context"),), + ) + + warnings = detect_shape_mismatches(existing, field_shapes, field_confidence) + + flagged = {(item["field"], item["role"]) for item in warnings} + self.assertIn(("context", "categorical"), flagged) + self.assertIn(("context", "relationship"), flagged) + self.assertNotIn(("srcip", "relationship"), flagged) + + def test_existing_profile_with_fitting_fields_produces_no_warnings(self): + field_shapes = {"action": "categorical_low_cardinality", "srcip": "ip"} + field_confidence = {"action": 85, "srcip": 90} + existing = StreamProfile( + stream_id="firewall", + name="Firewall", + entity_field="srcip", + timestamp_field="timestamp", + categorical_fields=("action",), + relationship_fields=(RelationshipField(left="srcip", right="action", name="srcip -> action"),), + ) + + self.assertEqual(detect_shape_mismatches(existing, field_shapes, field_confidence), []) + + def test_missing_existing_profile_produces_no_warnings(self): + self.assertEqual(detect_shape_mismatches(None, {}, {}), []) + def test_every_suggestion_carries_field_confidence_and_shapes(self): events = [ parse_log_line(f"fgai_stream_id=firewall fgai_stream=Firewall srcip=10.0.0.{index} dstip=8.8.8.8 dstport=443 action=deny timestamp=2026-06-29T10:00:{index:02d}Z hitcount={index}")