Public/fgAI
SHA256
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ny roadmap och arch

Browse Source
This commit is contained in:
larssand
2026-09-23 15:09:13 +02:00
parent 5f8b1cd098
commit 7cbec64b0e
3 changed files with 281 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

220
ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,220 @@
# SignalScope Architecture
This document explains how data moves through SignalScope today, the
structural gaps found by auditing a real 25-stream production deployment,
and the target architecture that [`ROADMAP.md`](ROADMAP.md) section 0 is
working toward. Read this before making changes to discovery, baselining, or
detection — several of today's problems come from small local decisions that
look correct in isolation but compound badly at real scale.
## Principles
- **Local-only, single-user.** No data leaves the machine. No auth beyond
binding to loopback (see the README's "Exposing the Dashboard Beyond
Loopback" section before changing that).
- **Never suppress something genuinely new.** Every suppression mechanism in
this codebase (the feedback engine's scope model, auto-apply's manual-field
protection, drift flagging instead of silent re-correction) exists to
uphold this. Any new feature that touches detection or suppression must be
checked against it before it ships.
- **Fix the source, not the symptom.** A UI feature that makes it easier to
clean up noise is worth less than a structural fix that stops the noise
from being generated in a misleading shape in the first place. Section 0 of
the roadmap exists because this session repeatedly reached for the former
when the latter was available.
## Data Flow (current)
```
Graylog MCP or local syslog
|
v
+-------------------+
| Ingestion | graylog_source.py / graylog_aggregate.py / syslog_server.py / logs.py
+-------------------+
|
v
+-------------------+
| Normalization & | normalization.py (name-alias canonical map)
| Discovery | field_shapes.py (value-shape classifier: ip/timestamp/port/
| | numeric/categorical/free_text, confidence 0-100)
| | history.py FieldDiscoveryStore (accumulates observed
| | values across polls so discovery survives sampling)
+-------------------+
|
v
+-------------------+
| Profile Discovery | profile_suggestions.py: suggest_stream_profiles()
| (per stream) | - name-priority lists seed entity/categorical/numeric
| | candidates (ENTITY_PRIORITY, CATEGORICAL_PRIORITY, ...)
| | - field_shapes.py reinforces/extends selection for
| | fields that did NOT match by name
| | - _relationship_candidates() pairs entities against
| | categorical/numeric fields, name-token driven
| | - optional Ollama profile advisor refines only the
| | fields the deterministic pass is unsure about
| | stream_profiles.py: parsed StreamProfile, field_origin
| | (auto/manual) per field, defaults missing to "manual"
+-------------------+
|
v
+-------------------+
| Baselining | baseline.py: BaselineStore, five-minute buckets per
| (per stream) | (stream, entity, field), SQLite, UTC weekday/hour
| | aware comparison, retention/compaction
+-------------------+
|
v
+-------------------+
| Detection | baseline.py profile comparison -> field_deviations
| | detectors.py: burst/rare-value/sequence detectors
| | sequences.py: ordered cross-stream event sequences
+-------------------+
|
v
+-------------------+
| Aggregation | anomaly.py: detect_source_anomalies() sums per-source
| | signals (event-rate z-scores, deny-rate, UTM count,
| | new-destinations-vs-baseline, field_deviations, ...)
| | into one score per source; feedback-aware (entity-
| | scope suppression skips a source's finding entirely)
| | incidents.py: build_incidents() groups anomalies +
| | field_deviations by entity
| | triage.py: build_triage_queue() ranks incidents +
| | deviations by severity
| | recommendations.py: iterates the same anomalies list
+-------------------+
|
v
+-------------------+
| Feedback / | feedback.py: FeedbackStore, scope model (value/field/
| Suppression | entity/stream), feedback_matches()/select_match()
| | (most-specific-wins), suggest_generalizations()
| | (suggest merging repeated narrow rules, never auto-
| | applies), mandatory expiry on entity/stream scope
+-------------------+
|
v
+-------------------+
| Dashboard | dashboard.py: single-file HTTP server + polled JS UI
+-------------------+
```
`monitor.py` is the orchestrator that runs one full pass through this
pipeline every poll cycle (`build_status`) and drives the continuous loop
(`monitor_loop`), including auto-apply, drift flagging, and throttled Ollama
calls.
## Known Structural Gaps (found auditing a real deployment)
A live 25-stream deployment (~1M events/day) was audited directly while
debugging why its triage queue felt unmanageable. Findings, in order of
impact:
**1. Name-matched fields never get shape-validated.** `_generic_entity_fields`
etc. in `profile_suggestions.py` only reinforce/extend selection with
`field_shapes.py` for fields that *failed* to match a name-priority list. A
field that already matches a trusted name — like `context`, which is in the
common alias map as a generically useful correlation field — is accepted
without ever checking whether its *observed values* actually look like the
shape that role requires. In the audited deployment, `context` resolves to a
full raw log line or URL (free-text) in most streams, not a low-cardinality
categorical value, but it was never re-checked against that.
**2. Relationship-candidate selection has no shape-awareness at all.**
`_relationship_candidates` pairs every entity field against whatever already
landed in `categorical_fields`, purely via name-token matching (`src`/`dst`
tokens, semantic group membership). It never asks "does this field's value
shape make sense as a relationship target." Combined with gap 1, this let a
`*_ip -> context` relationship (comparing a source IP against the full raw
message) get selected in 12 of 25 audited stream profiles. Because a raw
message is unique on essentially every event by construction, this
relationship is *guaranteed* to look "new" every time — it produced 91
individual `new_relationship` field deviations across 5 streams in one poll,
feeding 17 of 25 entries in the triage queue, all scored "critical."
**3. Baselines are siloed per stream, even for entities shared-field
discovery already recognizes as the same real-world thing.** Shared-field
discovery (`profile_suggestions.py` `_shared_profile_fields`, used for
recommending profiles and for correlation) already knows that most of the 25
audited streams share the concept of "a source IP entity." But baselining
(`baseline.py`) never uses that: each stream keeps a fully independent
baseline per `(stream, entity, field)`. A source IP behaving completely
normally in one stream is treated as never-seen-before the first time a
*different* stream happens to observe it, purely because of which stream
polled it first — an artifact of how state is partitioned, not a real
signal.
**4. Findings are scored and surfaced individually, before any clustering.**
Nothing between "a field deviation was detected" and "it's a row in the
triage queue" asks whether many of today's deviations are actually the same
underlying pattern. Gap 2's 91 deviations are real per the letter of the
detector logic — but they are one signal, not 91. Scoring and severity
(`anomaly.py`, `triage.py`) currently have no notion of this, so a systemic
misconfiguration and a genuinely isolated critical event can look identical
in the UI: both max out at "critical."
These compound: gap 1 lets an unsuitable field into candidate pools, gap 2
lets it become a relationship target, gap 3 means there's no shared state to
dampen the noise across related streams, and gap 4 means nothing catches the
resulting flood before it reaches the human. Fixing gap 2 alone would have
prevented the specific incident that triggered this audit; fixing all four is
what makes the system hold up as more streams are added.
## Target Architecture
- **Shape-first validation everywhere a field is selected for a role**,
including name-matched fields, not only as a fallback for unmatched ones.
Free-text-shaped fields are structurally excluded from relationship
targets, and existing stored profiles are periodically re-validated
against current observed shapes (flagged for review, never silently
rewritten — consistent with the drift-flagging precedent already used for
auto-applied field mappings).
- **Shared, entity-scoped baselines** for entities that shared-field
discovery already recognizes as the same concept across multiple enabled
streams, instead of one cold baseline per `(stream, entity, field)` tuple.
A value known-normal in one contributing stream should not register as new
in another.
- **Cluster before score.** Field deviations get grouped by
`(detector, field, value-pattern)` signature before severity is assigned.
A pattern recurring identically across many entities becomes one clustered
triage entry with an occurrence count and entity breakdown, and — this is a
deliberate inversion of today's behavior — a large, identical cluster is a
signal to score *lower*, since it more often indicates a detector or
profile problem than a security event, while a small, isolated occurrence
keeps its current weight.
The feedback/suppression layer (`feedback.py`) is not being restructured — it
is sound and stays as-is. It benefits automatically from cleaner input: fewer
and better-clustered findings to review, and less need for the bulk-clear and
rule-generalization UI to compensate for structural noise.
## Module Index
| Concern | File(s) |
|---|---|
| Log ingestion | `graylog_source.py`, `graylog_aggregate.py`, `graylog_mcp.py`, `syslog_server.py`, `logs.py` |
| Field name normalization | `normalization.py` |
| Value-shape classification | `field_shapes.py` |
| Field/stream discovery accumulation | `history.py` (`FieldDiscoveryStore`) |
| Profile suggestion & advisor merge | `profile_suggestions.py`, `llm.py` (`ollama_profile_advice`) |
| Stored profile schema | `stream_profiles.py` |
| Baselines | `baseline.py` |
| Detectors (burst/rare-value/sequence) | `detectors.py`, `sequences.py` |
| Per-source anomaly aggregation | `anomaly.py` |
| Incident grouping | `incidents.py` |
| Triage ranking | `triage.py` |
| Recommendations | `recommendations.py` |
| Feedback/suppression engine | `feedback.py` |
| Threat intel enrichment | `threat_intel.py` |
| Mitigation/blocking | `mitigation.py` |
| Metrics export | `metrics.py` |
| Orchestration (poll cycle, CLI monitor loop) | `monitor.py` |
| Dashboard (HTTP + UI) | `dashboard.py` |
| CLI entry points | `cli.py` |
## See Also
- [`README.md`](README.md) — day-to-day usage, dashboard workflow, settings reference.
- [`ROADMAP.md`](ROADMAP.md) — prioritized work, including section 0 for this rework.
- [`systemd/README.md`](systemd/README.md) — production deployment.

4
README.md
View File

@@ -8,7 +8,9 @@ FortiGate is one supported example. The same workflow applies to DNS/AdGuard, Wi
The Python module and legacy `fgai` command remain available for compatibility. New installations can use `signalscope`. The Python module and legacy `fgai` command remain available for compatibility. New installations can use `signalscope`.
The prioritized implementation plan is tracked in [ROADMAP.md](ROADMAP.md). The prioritized implementation plan is tracked in [ROADMAP.md](ROADMAP.md). For
how data actually flows through the system and the structural rework
currently in progress, see [ARCHITECTURE.md](ARCHITECTURE.md).
Autoblocking is dry-run by default. The tool will not block RFC1918, loopback, multicast, link-local, reserved, or allowlisted addresses unless you change the code. Autoblocking is dry-run by default. The tool will not block RFC1918, loopback, multicast, link-local, reserved, or allowlisted addresses unless you change the code.

61
ROADMAP.md
View File

@@ -19,6 +19,57 @@ operational value and dependency, not by UI appeal.
- [x] Prometheus-compatible low-cardinality metrics endpoint. - [x] Prometheus-compatible low-cardinality metrics endpoint.
- [x] Local dashboard for status, findings, incidents, diagnostics, and configuration. - [x] Local dashboard for status, findings, incidents, diagnostics, and configuration.
## 0. Structural Rework: Shape-Validated Discovery, Shared Baselines, Clustered Findings
Goal: fix root causes found by auditing a real 25-stream / ~1M-event/day
production deployment, instead of continuing to patch symptoms in the UI. See
[`ARCHITECTURE.md`](ARCHITECTURE.md) for the full data-flow diagram and the
detailed rationale behind each item below. This section blocks/supersedes
parts of sections 1 and 6 below (baseline confidence tooling and compact
incident cards both depend on findings being clustered correctly first) — do
this section before spending more effort on either.
- [ ] Validate name-matched fields against observed value shape too, not just
fields that had no name match. A field that matches a trusted name (e.g.
`context`) but is actually free-text (a full raw log line, confirmed in 12
of 25 audited stream profiles) must not be trusted just because its name
matched a known-useful alias.
- [ ] Make relationship-candidate selection shape-aware: never propose a
free-text-shaped field as a relationship target, regardless of how it
entered `categorical_fields`. This alone was responsible for 17 of 25
entries in one real triage queue snapshot (91 individual `new_relationship`
deviations from a single misconfigured `*->context` pairing, repeated
across 5 streams).
- [ ] Re-validate existing stored profiles' `relationship_fields` and
`categorical_fields` against current observed shapes on an ongoing basis,
not just at profile-creation time — flag (never silently rewrite) fields
that no longer look fit for their assigned role, and surface a one-click
"remove this field from relationships" action from the flag.
- [ ] Pool baseline state by shared entity concept across streams that
shared-field discovery already recognizes as the same real-world entity
(for example a source IP seen in both a firewall stream and an IIS
stream), instead of a cold, isolated baseline per stream per field. A value
already known-normal in one contributing stream must not register as "new"
the first time a different, related stream happens to see it.
- [ ] Cluster field deviations by `(detector, field, value-pattern)`
signature *before* scoring, not after. A pattern recurring identically
across many entities must surface as one clustered item with an occurrence
count and a "how many entities" breakdown, not N independently-scored
"critical" rows that all have to be discovered, selected, and bulk-cleared
by hand after the fact.
- [ ] Recalibrate severity so cluster size is itself a signal: a pattern
hitting many unrelated entities identically is usually a
configuration/detector problem, not N separate critical incidents, and
should score at or below an isolated occurrence, never above it.
- [ ] Update README usage documentation alongside each item above as it
ships — a behavior change is not done until it is documented.
Acceptance: replaying the dataset that produced the original 25-entry,
all-"critical" triage queue (17 of them the same misconfigured relationship)
produces a small number of distinct, correctly-ranked items, with the
systemic pattern visible as one clustered entry showing "91 occurrences
across 5 streams," not 17 separate critical rows.
## 1. Detection Quality ## 1. Detection Quality
Goal: make findings more accurate before adding more integrations. Goal: make findings more accurate before adding more integrations.
@@ -124,6 +175,10 @@ Acceptance: adding a new Graylog stream normally requires a profile template and
## Suggested Next Milestone ## Suggested Next Milestone
Complete Detection Quality first: confidence scoring, burst/rare-value detectors, Complete section 0 (Structural Rework) first. It was added after auditing a
per-stream thresholds, and historical replay. These features determine whether real production deployment and finding that the existing Detection Quality
the incident and dashboard work remains trustworthy as more log sources are added. and Dashboard Usability work, while individually correct, sits on top of a
discovery/scoring pipeline that lets one misconfigured field flood the triage
queue with dozens of look-alike "critical" entries. Fixing the pipeline is
higher leverage than any further UI work until it's done — see
[`ARCHITECTURE.md`](ARCHITECTURE.md).