Public/fgAI
SHA256
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ny roadmap och arch

Browse Source
This commit is contained in:
larssand
2026-09-23 15:09:13 +02:00
parent 5f8b1cd098
commit 7cbec64b0e
3 changed files with 281 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

220
ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,220 @@
# SignalScope Architecture
This document explains how data moves through SignalScope today, the
structural gaps found by auditing a real 25-stream production deployment,
and the target architecture that [`ROADMAP.md`](ROADMAP.md) section 0 is
working toward. Read this before making changes to discovery, baselining, or
detection — several of today's problems come from small local decisions that
look correct in isolation but compound badly at real scale.
## Principles
- **Local-only, single-user.** No data leaves the machine. No auth beyond
binding to loopback (see the README's "Exposing the Dashboard Beyond
Loopback" section before changing that).
- **Never suppress something genuinely new.** Every suppression mechanism in
this codebase (the feedback engine's scope model, auto-apply's manual-field
protection, drift flagging instead of silent re-correction) exists to
uphold this. Any new feature that touches detection or suppression must be
checked against it before it ships.
- **Fix the source, not the symptom.** A UI feature that makes it easier to
clean up noise is worth less than a structural fix that stops the noise
from being generated in a misleading shape in the first place. Section 0 of
the roadmap exists because this session repeatedly reached for the former
when the latter was available.
## Data Flow (current)
```
Graylog MCP or local syslog
|
v
+-------------------+
| Ingestion | graylog_source.py / graylog_aggregate.py / syslog_server.py / logs.py
+-------------------+
|
v
+-------------------+
| Normalization & | normalization.py (name-alias canonical map)
| Discovery | field_shapes.py (value-shape classifier: ip/timestamp/port/
| | numeric/categorical/free_text, confidence 0-100)
| | history.py FieldDiscoveryStore (accumulates observed
| | values across polls so discovery survives sampling)
+-------------------+
|
v
+-------------------+
| Profile Discovery | profile_suggestions.py: suggest_stream_profiles()
| (per stream) | - name-priority lists seed entity/categorical/numeric
| | candidates (ENTITY_PRIORITY, CATEGORICAL_PRIORITY, ...)
| | - field_shapes.py reinforces/extends selection for
| | fields that did NOT match by name
| | - _relationship_candidates() pairs entities against
| | categorical/numeric fields, name-token driven
| | - optional Ollama profile advisor refines only the
| | fields the deterministic pass is unsure about
| | stream_profiles.py: parsed StreamProfile, field_origin
| | (auto/manual) per field, defaults missing to "manual"
+-------------------+
|
v
+-------------------+
| Baselining | baseline.py: BaselineStore, five-minute buckets per
| (per stream) | (stream, entity, field), SQLite, UTC weekday/hour
| | aware comparison, retention/compaction
+-------------------+
|
v
+-------------------+
| Detection | baseline.py profile comparison -> field_deviations
| | detectors.py: burst/rare-value/sequence detectors
| | sequences.py: ordered cross-stream event sequences
+-------------------+
|
v
+-------------------+
| Aggregation | anomaly.py: detect_source_anomalies() sums per-source
| | signals (event-rate z-scores, deny-rate, UTM count,
| | new-destinations-vs-baseline, field_deviations, ...)
| | into one score per source; feedback-aware (entity-
| | scope suppression skips a source's finding entirely)
| | incidents.py: build_incidents() groups anomalies +
| | field_deviations by entity
| | triage.py: build_triage_queue() ranks incidents +
| | deviations by severity
| | recommendations.py: iterates the same anomalies list
+-------------------+
|
v
+-------------------+
| Feedback / | feedback.py: FeedbackStore, scope model (value/field/
| Suppression | entity/stream), feedback_matches()/select_match()
| | (most-specific-wins), suggest_generalizations()
| | (suggest merging repeated narrow rules, never auto-
| | applies), mandatory expiry on entity/stream scope
+-------------------+
|
v
+-------------------+
| Dashboard | dashboard.py: single-file HTTP server + polled JS UI
+-------------------+
```
`monitor.py` is the orchestrator that runs one full pass through this
pipeline every poll cycle (`build_status`) and drives the continuous loop
(`monitor_loop`), including auto-apply, drift flagging, and throttled Ollama
calls.
## Known Structural Gaps (found auditing a real deployment)
A live 25-stream deployment (~1M events/day) was audited directly while
debugging why its triage queue felt unmanageable. Findings, in order of
impact:
**1. Name-matched fields never get shape-validated.** `_generic_entity_fields`
etc. in `profile_suggestions.py` only reinforce/extend selection with
`field_shapes.py` for fields that *failed* to match a name-priority list. A
field that already matches a trusted name — like `context`, which is in the
common alias map as a generically useful correlation field — is accepted
without ever checking whether its *observed values* actually look like the
shape that role requires. In the audited deployment, `context` resolves to a
full raw log line or URL (free-text) in most streams, not a low-cardinality
categorical value, but it was never re-checked against that.
**2. Relationship-candidate selection has no shape-awareness at all.**
`_relationship_candidates` pairs every entity field against whatever already
landed in `categorical_fields`, purely via name-token matching (`src`/`dst`
tokens, semantic group membership). It never asks "does this field's value
shape make sense as a relationship target." Combined with gap 1, this let a
`*_ip -> context` relationship (comparing a source IP against the full raw
message) get selected in 12 of 25 audited stream profiles. Because a raw
message is unique on essentially every event by construction, this
relationship is *guaranteed* to look "new" every time — it produced 91
individual `new_relationship` field deviations across 5 streams in one poll,
feeding 17 of 25 entries in the triage queue, all scored "critical."
**3. Baselines are siloed per stream, even for entities shared-field
discovery already recognizes as the same real-world thing.** Shared-field
discovery (`profile_suggestions.py` `_shared_profile_fields`, used for
recommending profiles and for correlation) already knows that most of the 25
audited streams share the concept of "a source IP entity." But baselining
(`baseline.py`) never uses that: each stream keeps a fully independent
baseline per `(stream, entity, field)`. A source IP behaving completely
normally in one stream is treated as never-seen-before the first time a
*different* stream happens to observe it, purely because of which stream
polled it first — an artifact of how state is partitioned, not a real
signal.
**4. Findings are scored and surfaced individually, before any clustering.**
Nothing between "a field deviation was detected" and "it's a row in the
triage queue" asks whether many of today's deviations are actually the same
underlying pattern. Gap 2's 91 deviations are real per the letter of the
detector logic — but they are one signal, not 91. Scoring and severity
(`anomaly.py`, `triage.py`) currently have no notion of this, so a systemic
misconfiguration and a genuinely isolated critical event can look identical
in the UI: both max out at "critical."
These compound: gap 1 lets an unsuitable field into candidate pools, gap 2
lets it become a relationship target, gap 3 means there's no shared state to
dampen the noise across related streams, and gap 4 means nothing catches the
resulting flood before it reaches the human. Fixing gap 2 alone would have
prevented the specific incident that triggered this audit; fixing all four is
what makes the system hold up as more streams are added.
## Target Architecture
- **Shape-first validation everywhere a field is selected for a role**,
including name-matched fields, not only as a fallback for unmatched ones.
Free-text-shaped fields are structurally excluded from relationship
targets, and existing stored profiles are periodically re-validated
against current observed shapes (flagged for review, never silently
rewritten — consistent with the drift-flagging precedent already used for
auto-applied field mappings).
- **Shared, entity-scoped baselines** for entities that shared-field
discovery already recognizes as the same concept across multiple enabled
streams, instead of one cold baseline per `(stream, entity, field)` tuple.
A value known-normal in one contributing stream should not register as new
in another.
- **Cluster before score.** Field deviations get grouped by
`(detector, field, value-pattern)` signature before severity is assigned.
A pattern recurring identically across many entities becomes one clustered
triage entry with an occurrence count and entity breakdown, and — this is a
deliberate inversion of today's behavior — a large, identical cluster is a
signal to score *lower*, since it more often indicates a detector or
profile problem than a security event, while a small, isolated occurrence
keeps its current weight.
The feedback/suppression layer (`feedback.py`) is not being restructured — it
is sound and stays as-is. It benefits automatically from cleaner input: fewer
and better-clustered findings to review, and less need for the bulk-clear and
rule-generalization UI to compensate for structural noise.
## Module Index
| Concern | File(s) |
|---|---|
| Log ingestion | `graylog_source.py`, `graylog_aggregate.py`, `graylog_mcp.py`, `syslog_server.py`, `logs.py` |
| Field name normalization | `normalization.py` |
| Value-shape classification | `field_shapes.py` |
| Field/stream discovery accumulation | `history.py` (`FieldDiscoveryStore`) |
| Profile suggestion & advisor merge | `profile_suggestions.py`, `llm.py` (`ollama_profile_advice`) |
| Stored profile schema | `stream_profiles.py` |
| Baselines | `baseline.py` |
| Detectors (burst/rare-value/sequence) | `detectors.py`, `sequences.py` |
| Per-source anomaly aggregation | `anomaly.py` |
| Incident grouping | `incidents.py` |
| Triage ranking | `triage.py` |
| Recommendations | `recommendations.py` |
| Feedback/suppression engine | `feedback.py` |
| Threat intel enrichment | `threat_intel.py` |
| Mitigation/blocking | `mitigation.py` |
| Metrics export | `metrics.py` |
| Orchestration (poll cycle, CLI monitor loop) | `monitor.py` |
| Dashboard (HTTP + UI) | `dashboard.py` |
| CLI entry points | `cli.py` |
## See Also
- [`README.md`](README.md) — day-to-day usage, dashboard workflow, settings reference.
- [`ROADMAP.md`](ROADMAP.md) — prioritized work, including section 0 for this rework.
- [`systemd/README.md`](systemd/README.md) — production deployment.

4
README.md
View File

@@ -8,7 +8,9 @@ FortiGate is one supported example. The same workflow applies to DNS/AdGuard, Wi
The Python module and legacy `fgai` command remain available for compatibility. New installations can use `signalscope`.
The prioritized implementation plan is tracked in [ROADMAP.md](ROADMAP.md).
The prioritized implementation plan is tracked in [ROADMAP.md](ROADMAP.md). For
how data actually flows through the system and the structural rework
currently in progress, see [ARCHITECTURE.md](ARCHITECTURE.md).
Autoblocking is dry-run by default. The tool will not block RFC1918, loopback, multicast, link-local, reserved, or allowlisted addresses unless you change the code.

61
ROADMAP.md
View File

@@ -19,6 +19,57 @@ operational value and dependency, not by UI appeal.
- [x] Prometheus-compatible low-cardinality metrics endpoint.
- [x] Local dashboard for status, findings, incidents, diagnostics, and configuration.
## 0. Structural Rework: Shape-Validated Discovery, Shared Baselines, Clustered Findings
Goal: fix root causes found by auditing a real 25-stream / ~1M-event/day
production deployment, instead of continuing to patch symptoms in the UI. See
[`ARCHITECTURE.md`](ARCHITECTURE.md) for the full data-flow diagram and the
detailed rationale behind each item below. This section blocks/supersedes
parts of sections 1 and 6 below (baseline confidence tooling and compact
incident cards both depend on findings being clustered correctly first) — do
this section before spending more effort on either.
- [ ] Validate name-matched fields against observed value shape too, not just
fields that had no name match. A field that matches a trusted name (e.g.
`context`) but is actually free-text (a full raw log line, confirmed in 12
of 25 audited stream profiles) must not be trusted just because its name
matched a known-useful alias.
- [ ] Make relationship-candidate selection shape-aware: never propose a
free-text-shaped field as a relationship target, regardless of how it
entered `categorical_fields`. This alone was responsible for 17 of 25
entries in one real triage queue snapshot (91 individual `new_relationship`
deviations from a single misconfigured `*->context` pairing, repeated
across 5 streams).
- [ ] Re-validate existing stored profiles' `relationship_fields` and
`categorical_fields` against current observed shapes on an ongoing basis,
not just at profile-creation time — flag (never silently rewrite) fields
that no longer look fit for their assigned role, and surface a one-click
"remove this field from relationships" action from the flag.
- [ ] Pool baseline state by shared entity concept across streams that
shared-field discovery already recognizes as the same real-world entity
(for example a source IP seen in both a firewall stream and an IIS
stream), instead of a cold, isolated baseline per stream per field. A value
already known-normal in one contributing stream must not register as "new"
the first time a different, related stream happens to see it.
- [ ] Cluster field deviations by `(detector, field, value-pattern)`
signature *before* scoring, not after. A pattern recurring identically
across many entities must surface as one clustered item with an occurrence
count and a "how many entities" breakdown, not N independently-scored
"critical" rows that all have to be discovered, selected, and bulk-cleared
by hand after the fact.
- [ ] Recalibrate severity so cluster size is itself a signal: a pattern
hitting many unrelated entities identically is usually a
configuration/detector problem, not N separate critical incidents, and
should score at or below an isolated occurrence, never above it.
- [ ] Update README usage documentation alongside each item above as it
ships — a behavior change is not done until it is documented.
Acceptance: replaying the dataset that produced the original 25-entry,
all-"critical" triage queue (17 of them the same misconfigured relationship)
produces a small number of distinct, correctly-ranked items, with the
systemic pattern visible as one clustered entry showing "91 occurrences
across 5 streams," not 17 separate critical rows.
## 1. Detection Quality
Goal: make findings more accurate before adding more integrations.
@@ -124,6 +175,10 @@ Acceptance: adding a new Graylog stream normally requires a profile template and
## Suggested Next Milestone
Complete Detection Quality first: confidence scoring, burst/rare-value detectors,
per-stream thresholds, and historical replay. These features determine whether
the incident and dashboard work remains trustworthy as more log sources are added.
Complete section 0 (Structural Rework) first. It was added after auditing a
real production deployment and finding that the existing Detection Quality
and Dashboard Usability work, while individually correct, sits on top of a
discovery/scoring pipeline that lets one misconfigured field flood the triage
queue with dozens of look-alike "critical" entries. Fixing the pipeline is
higher leverage than any further UI work until it's done — see
[`ARCHITECTURE.md`](ARCHITECTURE.md).