From 7cbec64b0ec0604857b194465fbdab10545d361c23e9d9cef88056433c4c1799 Mon Sep 17 00:00:00 2001 From: larssand Date: Wed, 23 Sep 2026 15:09:13 +0200 Subject: [PATCH] ny roadmap och arch --- ARCHITECTURE.md | 220 ++++++++++++++++++++++++++++++++++++++++++++++++ README.md | 4 +- ROADMAP.md | 61 +++++++++++++- 3 files changed, 281 insertions(+), 4 deletions(-) create mode 100644 ARCHITECTURE.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..ef8fd72 --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,220 @@ +# SignalScope Architecture + +This document explains how data moves through SignalScope today, the +structural gaps found by auditing a real 25-stream production deployment, +and the target architecture that [`ROADMAP.md`](ROADMAP.md) section 0 is +working toward. Read this before making changes to discovery, baselining, or +detection — several of today's problems come from small local decisions that +look correct in isolation but compound badly at real scale. + +## Principles + +- **Local-only, single-user.** No data leaves the machine. No auth beyond + binding to loopback (see the README's "Exposing the Dashboard Beyond + Loopback" section before changing that). +- **Never suppress something genuinely new.** Every suppression mechanism in + this codebase (the feedback engine's scope model, auto-apply's manual-field + protection, drift flagging instead of silent re-correction) exists to + uphold this. Any new feature that touches detection or suppression must be + checked against it before it ships. +- **Fix the source, not the symptom.** A UI feature that makes it easier to + clean up noise is worth less than a structural fix that stops the noise + from being generated in a misleading shape in the first place. Section 0 of + the roadmap exists because this session repeatedly reached for the former + when the latter was available. + +## Data Flow (current) + +``` +Graylog MCP or local syslog + | + v ++-------------------+ +| Ingestion | graylog_source.py / graylog_aggregate.py / syslog_server.py / logs.py ++-------------------+ + | + v ++-------------------+ +| Normalization & | normalization.py (name-alias canonical map) +| Discovery | field_shapes.py (value-shape classifier: ip/timestamp/port/ +| | numeric/categorical/free_text, confidence 0-100) +| | history.py FieldDiscoveryStore (accumulates observed +| | values across polls so discovery survives sampling) ++-------------------+ + | + v ++-------------------+ +| Profile Discovery | profile_suggestions.py: suggest_stream_profiles() +| (per stream) | - name-priority lists seed entity/categorical/numeric +| | candidates (ENTITY_PRIORITY, CATEGORICAL_PRIORITY, ...) +| | - field_shapes.py reinforces/extends selection for +| | fields that did NOT match by name +| | - _relationship_candidates() pairs entities against +| | categorical/numeric fields, name-token driven +| | - optional Ollama profile advisor refines only the +| | fields the deterministic pass is unsure about +| | stream_profiles.py: parsed StreamProfile, field_origin +| | (auto/manual) per field, defaults missing to "manual" ++-------------------+ + | + v ++-------------------+ +| Baselining | baseline.py: BaselineStore, five-minute buckets per +| (per stream) | (stream, entity, field), SQLite, UTC weekday/hour +| | aware comparison, retention/compaction ++-------------------+ + | + v ++-------------------+ +| Detection | baseline.py profile comparison -> field_deviations +| | detectors.py: burst/rare-value/sequence detectors +| | sequences.py: ordered cross-stream event sequences ++-------------------+ + | + v ++-------------------+ +| Aggregation | anomaly.py: detect_source_anomalies() sums per-source +| | signals (event-rate z-scores, deny-rate, UTM count, +| | new-destinations-vs-baseline, field_deviations, ...) +| | into one score per source; feedback-aware (entity- +| | scope suppression skips a source's finding entirely) +| | incidents.py: build_incidents() groups anomalies + +| | field_deviations by entity +| | triage.py: build_triage_queue() ranks incidents + +| | deviations by severity +| | recommendations.py: iterates the same anomalies list ++-------------------+ + | + v ++-------------------+ +| Feedback / | feedback.py: FeedbackStore, scope model (value/field/ +| Suppression | entity/stream), feedback_matches()/select_match() +| | (most-specific-wins), suggest_generalizations() +| | (suggest merging repeated narrow rules, never auto- +| | applies), mandatory expiry on entity/stream scope ++-------------------+ + | + v ++-------------------+ +| Dashboard | dashboard.py: single-file HTTP server + polled JS UI ++-------------------+ +``` + +`monitor.py` is the orchestrator that runs one full pass through this +pipeline every poll cycle (`build_status`) and drives the continuous loop +(`monitor_loop`), including auto-apply, drift flagging, and throttled Ollama +calls. + +## Known Structural Gaps (found auditing a real deployment) + +A live 25-stream deployment (~1M events/day) was audited directly while +debugging why its triage queue felt unmanageable. Findings, in order of +impact: + +**1. Name-matched fields never get shape-validated.** `_generic_entity_fields` +etc. in `profile_suggestions.py` only reinforce/extend selection with +`field_shapes.py` for fields that *failed* to match a name-priority list. A +field that already matches a trusted name — like `context`, which is in the +common alias map as a generically useful correlation field — is accepted +without ever checking whether its *observed values* actually look like the +shape that role requires. In the audited deployment, `context` resolves to a +full raw log line or URL (free-text) in most streams, not a low-cardinality +categorical value, but it was never re-checked against that. + +**2. Relationship-candidate selection has no shape-awareness at all.** +`_relationship_candidates` pairs every entity field against whatever already +landed in `categorical_fields`, purely via name-token matching (`src`/`dst` +tokens, semantic group membership). It never asks "does this field's value +shape make sense as a relationship target." Combined with gap 1, this let a +`*_ip -> context` relationship (comparing a source IP against the full raw +message) get selected in 12 of 25 audited stream profiles. Because a raw +message is unique on essentially every event by construction, this +relationship is *guaranteed* to look "new" every time — it produced 91 +individual `new_relationship` field deviations across 5 streams in one poll, +feeding 17 of 25 entries in the triage queue, all scored "critical." + +**3. Baselines are siloed per stream, even for entities shared-field +discovery already recognizes as the same real-world thing.** Shared-field +discovery (`profile_suggestions.py` `_shared_profile_fields`, used for +recommending profiles and for correlation) already knows that most of the 25 +audited streams share the concept of "a source IP entity." But baselining +(`baseline.py`) never uses that: each stream keeps a fully independent +baseline per `(stream, entity, field)`. A source IP behaving completely +normally in one stream is treated as never-seen-before the first time a +*different* stream happens to observe it, purely because of which stream +polled it first — an artifact of how state is partitioned, not a real +signal. + +**4. Findings are scored and surfaced individually, before any clustering.** +Nothing between "a field deviation was detected" and "it's a row in the +triage queue" asks whether many of today's deviations are actually the same +underlying pattern. Gap 2's 91 deviations are real per the letter of the +detector logic — but they are one signal, not 91. Scoring and severity +(`anomaly.py`, `triage.py`) currently have no notion of this, so a systemic +misconfiguration and a genuinely isolated critical event can look identical +in the UI: both max out at "critical." + +These compound: gap 1 lets an unsuitable field into candidate pools, gap 2 +lets it become a relationship target, gap 3 means there's no shared state to +dampen the noise across related streams, and gap 4 means nothing catches the +resulting flood before it reaches the human. Fixing gap 2 alone would have +prevented the specific incident that triggered this audit; fixing all four is +what makes the system hold up as more streams are added. + +## Target Architecture + +- **Shape-first validation everywhere a field is selected for a role**, + including name-matched fields, not only as a fallback for unmatched ones. + Free-text-shaped fields are structurally excluded from relationship + targets, and existing stored profiles are periodically re-validated + against current observed shapes (flagged for review, never silently + rewritten — consistent with the drift-flagging precedent already used for + auto-applied field mappings). +- **Shared, entity-scoped baselines** for entities that shared-field + discovery already recognizes as the same concept across multiple enabled + streams, instead of one cold baseline per `(stream, entity, field)` tuple. + A value known-normal in one contributing stream should not register as new + in another. +- **Cluster before score.** Field deviations get grouped by + `(detector, field, value-pattern)` signature before severity is assigned. + A pattern recurring identically across many entities becomes one clustered + triage entry with an occurrence count and entity breakdown, and — this is a + deliberate inversion of today's behavior — a large, identical cluster is a + signal to score *lower*, since it more often indicates a detector or + profile problem than a security event, while a small, isolated occurrence + keeps its current weight. + +The feedback/suppression layer (`feedback.py`) is not being restructured — it +is sound and stays as-is. It benefits automatically from cleaner input: fewer +and better-clustered findings to review, and less need for the bulk-clear and +rule-generalization UI to compensate for structural noise. + +## Module Index + +| Concern | File(s) | +|---|---| +| Log ingestion | `graylog_source.py`, `graylog_aggregate.py`, `graylog_mcp.py`, `syslog_server.py`, `logs.py` | +| Field name normalization | `normalization.py` | +| Value-shape classification | `field_shapes.py` | +| Field/stream discovery accumulation | `history.py` (`FieldDiscoveryStore`) | +| Profile suggestion & advisor merge | `profile_suggestions.py`, `llm.py` (`ollama_profile_advice`) | +| Stored profile schema | `stream_profiles.py` | +| Baselines | `baseline.py` | +| Detectors (burst/rare-value/sequence) | `detectors.py`, `sequences.py` | +| Per-source anomaly aggregation | `anomaly.py` | +| Incident grouping | `incidents.py` | +| Triage ranking | `triage.py` | +| Recommendations | `recommendations.py` | +| Feedback/suppression engine | `feedback.py` | +| Threat intel enrichment | `threat_intel.py` | +| Mitigation/blocking | `mitigation.py` | +| Metrics export | `metrics.py` | +| Orchestration (poll cycle, CLI monitor loop) | `monitor.py` | +| Dashboard (HTTP + UI) | `dashboard.py` | +| CLI entry points | `cli.py` | + +## See Also + +- [`README.md`](README.md) — day-to-day usage, dashboard workflow, settings reference. +- [`ROADMAP.md`](ROADMAP.md) — prioritized work, including section 0 for this rework. +- [`systemd/README.md`](systemd/README.md) — production deployment. diff --git a/README.md b/README.md index 1d76642..35c6e64 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,9 @@ FortiGate is one supported example. The same workflow applies to DNS/AdGuard, Wi The Python module and legacy `fgai` command remain available for compatibility. New installations can use `signalscope`. -The prioritized implementation plan is tracked in [ROADMAP.md](ROADMAP.md). +The prioritized implementation plan is tracked in [ROADMAP.md](ROADMAP.md). For +how data actually flows through the system and the structural rework +currently in progress, see [ARCHITECTURE.md](ARCHITECTURE.md). Autoblocking is dry-run by default. The tool will not block RFC1918, loopback, multicast, link-local, reserved, or allowlisted addresses unless you change the code. diff --git a/ROADMAP.md b/ROADMAP.md index 24e98db..55c3662 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -19,6 +19,57 @@ operational value and dependency, not by UI appeal. - [x] Prometheus-compatible low-cardinality metrics endpoint. - [x] Local dashboard for status, findings, incidents, diagnostics, and configuration. +## 0. Structural Rework: Shape-Validated Discovery, Shared Baselines, Clustered Findings + +Goal: fix root causes found by auditing a real 25-stream / ~1M-event/day +production deployment, instead of continuing to patch symptoms in the UI. See +[`ARCHITECTURE.md`](ARCHITECTURE.md) for the full data-flow diagram and the +detailed rationale behind each item below. This section blocks/supersedes +parts of sections 1 and 6 below (baseline confidence tooling and compact +incident cards both depend on findings being clustered correctly first) — do +this section before spending more effort on either. + +- [ ] Validate name-matched fields against observed value shape too, not just + fields that had no name match. A field that matches a trusted name (e.g. + `context`) but is actually free-text (a full raw log line, confirmed in 12 + of 25 audited stream profiles) must not be trusted just because its name + matched a known-useful alias. +- [ ] Make relationship-candidate selection shape-aware: never propose a + free-text-shaped field as a relationship target, regardless of how it + entered `categorical_fields`. This alone was responsible for 17 of 25 + entries in one real triage queue snapshot (91 individual `new_relationship` + deviations from a single misconfigured `*->context` pairing, repeated + across 5 streams). +- [ ] Re-validate existing stored profiles' `relationship_fields` and + `categorical_fields` against current observed shapes on an ongoing basis, + not just at profile-creation time — flag (never silently rewrite) fields + that no longer look fit for their assigned role, and surface a one-click + "remove this field from relationships" action from the flag. +- [ ] Pool baseline state by shared entity concept across streams that + shared-field discovery already recognizes as the same real-world entity + (for example a source IP seen in both a firewall stream and an IIS + stream), instead of a cold, isolated baseline per stream per field. A value + already known-normal in one contributing stream must not register as "new" + the first time a different, related stream happens to see it. +- [ ] Cluster field deviations by `(detector, field, value-pattern)` + signature *before* scoring, not after. A pattern recurring identically + across many entities must surface as one clustered item with an occurrence + count and a "how many entities" breakdown, not N independently-scored + "critical" rows that all have to be discovered, selected, and bulk-cleared + by hand after the fact. +- [ ] Recalibrate severity so cluster size is itself a signal: a pattern + hitting many unrelated entities identically is usually a + configuration/detector problem, not N separate critical incidents, and + should score at or below an isolated occurrence, never above it. +- [ ] Update README usage documentation alongside each item above as it + ships — a behavior change is not done until it is documented. + +Acceptance: replaying the dataset that produced the original 25-entry, +all-"critical" triage queue (17 of them the same misconfigured relationship) +produces a small number of distinct, correctly-ranked items, with the +systemic pattern visible as one clustered entry showing "91 occurrences +across 5 streams," not 17 separate critical rows. + ## 1. Detection Quality Goal: make findings more accurate before adding more integrations. @@ -124,6 +175,10 @@ Acceptance: adding a new Graylog stream normally requires a profile template and ## Suggested Next Milestone -Complete Detection Quality first: confidence scoring, burst/rare-value detectors, -per-stream thresholds, and historical replay. These features determine whether -the incident and dashboard work remains trustworthy as more log sources are added. +Complete section 0 (Structural Rework) first. It was added after auditing a +real production deployment and finding that the existing Detection Quality +and Dashboard Usability work, while individually correct, sits on top of a +discovery/scoring pipeline that lets one misconfigured field flood the triage +queue with dozens of look-alike "critical" entries. Fixing the pipeline is +higher leverage than any further UI work until it's done — see +[`ARCHITECTURE.md`](ARCHITECTURE.md).