Public/fgAI
SHA256
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ny roadmap och arch

Browse Source
This commit is contained in:
larssand
2026-09-23 15:09:13 +02:00
parent 5f8b1cd098
commit 7cbec64b0e
3 changed files with 281 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

220
ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,220 @@
# SignalScope Architecture
This document explains how data moves through SignalScope today, the
structural gaps found by auditing a real 25-stream production deployment,
and the target architecture that [`ROADMAP.md`](ROADMAP.md) section 0 is
working toward. Read this before making changes to discovery, baselining, or
detection — several of today's problems come from small local decisions that
look correct in isolation but compound badly at real scale.
## Principles
- **Local-only, single-user.** No data leaves the machine. No auth beyond
binding to loopback (see the README's "Exposing the Dashboard Beyond
Loopback" section before changing that).
- **Never suppress something genuinely new.** Every suppression mechanism in
this codebase (the feedback engine's scope model, auto-apply's manual-field
protection, drift flagging instead of silent re-correction) exists to
uphold this. Any new feature that touches detection or suppression must be
checked against it before it ships.
- **Fix the source, not the symptom.** A UI feature that makes it easier to
clean up noise is worth less than a structural fix that stops the noise
from being generated in a misleading shape in the first place. Section 0 of
the roadmap exists because this session repeatedly reached for the former
when the latter was available.
## Data Flow (current)
```
Graylog MCP or local syslog
|
v
+-------------------+
| Ingestion | graylog_source.py / graylog_aggregate.py / syslog_server.py / logs.py
+-------------------+
|
v
+-------------------+
| Normalization & | normalization.py (name-alias canonical map)
| Discovery | field_shapes.py (value-shape classifier: ip/timestamp/port/
| | numeric/categorical/free_text, confidence 0-100)
| | history.py FieldDiscoveryStore (accumulates observed
| | values across polls so discovery survives sampling)
+-------------------+
|
v
+-------------------+
| Profile Discovery | profile_suggestions.py: suggest_stream_profiles()
| (per stream) | - name-priority lists seed entity/categorical/numeric
| | candidates (ENTITY_PRIORITY, CATEGORICAL_PRIORITY, ...)
| | - field_shapes.py reinforces/extends selection for
| | fields that did NOT match by name
| | - _relationship_candidates() pairs entities against
| | categorical/numeric fields, name-token driven
| | - optional Ollama profile advisor refines only the
| | fields the deterministic pass is unsure about
| | stream_profiles.py: parsed StreamProfile, field_origin
| | (auto/manual) per field, defaults missing to "manual"
+-------------------+
|
v
+-------------------+
| Baselining | baseline.py: BaselineStore, five-minute buckets per
| (per stream) | (stream, entity, field), SQLite, UTC weekday/hour
| | aware comparison, retention/compaction
+-------------------+
|
v
+-------------------+
| Detection | baseline.py profile comparison -> field_deviations
| | detectors.py: burst/rare-value/sequence detectors
| | sequences.py: ordered cross-stream event sequences
+-------------------+
|
v
+-------------------+
| Aggregation | anomaly.py: detect_source_anomalies() sums per-source
| | signals (event-rate z-scores, deny-rate, UTM count,
| | new-destinations-vs-baseline, field_deviations, ...)
| | into one score per source; feedback-aware (entity-
| | scope suppression skips a source's finding entirely)
| | incidents.py: build_incidents() groups anomalies +
| | field_deviations by entity
| | triage.py: build_triage_queue() ranks incidents +
| | deviations by severity
| | recommendations.py: iterates the same anomalies list
+-------------------+
|
v
+-------------------+
| Feedback / | feedback.py: FeedbackStore, scope model (value/field/
| Suppression | entity/stream), feedback_matches()/select_match()
| | (most-specific-wins), suggest_generalizations()
| | (suggest merging repeated narrow rules, never auto-
| | applies), mandatory expiry on entity/stream scope
+-------------------+
|
v
+-------------------+
| Dashboard | dashboard.py: single-file HTTP server + polled JS UI
+-------------------+
```
`monitor.py` is the orchestrator that runs one full pass through this
pipeline every poll cycle (`build_status`) and drives the continuous loop
(`monitor_loop`), including auto-apply, drift flagging, and throttled Ollama
calls.
## Known Structural Gaps (found auditing a real deployment)
A live 25-stream deployment (~1M events/day) was audited directly while
debugging why its triage queue felt unmanageable. Findings, in order of
impact:
**1. Name-matched fields never get shape-validated.** `_generic_entity_fields`
etc. in `profile_suggestions.py` only reinforce/extend selection with
`field_shapes.py` for fields that *failed* to match a name-priority list. A
field that already matches a trusted name — like `context`, which is in the
common alias map as a generically useful correlation field — is accepted
without ever checking whether its *observed values* actually look like the
shape that role requires. In the audited deployment, `context` resolves to a
full raw log line or URL (free-text) in most streams, not a low-cardinality
categorical value, but it was never re-checked against that.
**2. Relationship-candidate selection has no shape-awareness at all.**
`_relationship_candidates` pairs every entity field against whatever already
landed in `categorical_fields`, purely via name-token matching (`src`/`dst`
tokens, semantic group membership). It never asks "does this field's value
shape make sense as a relationship target." Combined with gap 1, this let a
`*_ip -> context` relationship (comparing a source IP against the full raw
message) get selected in 12 of 25 audited stream profiles. Because a raw
message is unique on essentially every event by construction, this
relationship is *guaranteed* to look "new" every time — it produced 91
individual `new_relationship` field deviations across 5 streams in one poll,
feeding 17 of 25 entries in the triage queue, all scored "critical."
**3. Baselines are siloed per stream, even for entities shared-field
discovery already recognizes as the same real-world thing.** Shared-field
discovery (`profile_suggestions.py` `_shared_profile_fields`, used for
recommending profiles and for correlation) already knows that most of the 25
audited streams share the concept of "a source IP entity." But baselining
(`baseline.py`) never uses that: each stream keeps a fully independent
baseline per `(stream, entity, field)`. A source IP behaving completely
normally in one stream is treated as never-seen-before the first time a
*different* stream happens to observe it, purely because of which stream
polled it first — an artifact of how state is partitioned, not a real
signal.
**4. Findings are scored and surfaced individually, before any clustering.**
Nothing between "a field deviation was detected" and "it's a row in the
triage queue" asks whether many of today's deviations are actually the same
underlying pattern. Gap 2's 91 deviations are real per the letter of the
detector logic — but they are one signal, not 91. Scoring and severity
(`anomaly.py`, `triage.py`) currently have no notion of this, so a systemic
misconfiguration and a genuinely isolated critical event can look identical
in the UI: both max out at "critical."
These compound: gap 1 lets an unsuitable field into candidate pools, gap 2
lets it become a relationship target, gap 3 means there's no shared state to
dampen the noise across related streams, and gap 4 means nothing catches the
resulting flood before it reaches the human. Fixing gap 2 alone would have
prevented the specific incident that triggered this audit; fixing all four is
what makes the system hold up as more streams are added.
## Target Architecture
- **Shape-first validation everywhere a field is selected for a role**,
including name-matched fields, not only as a fallback for unmatched ones.
Free-text-shaped fields are structurally excluded from relationship
targets, and existing stored profiles are periodically re-validated
against current observed shapes (flagged for review, never silently
rewritten — consistent with the drift-flagging precedent already used for
auto-applied field mappings).
- **Shared, entity-scoped baselines** for entities that shared-field
discovery already recognizes as the same concept across multiple enabled
streams, instead of one cold baseline per `(stream, entity, field)` tuple.
A value known-normal in one contributing stream should not register as new
in another.
- **Cluster before score.** Field deviations get grouped by
`(detector, field, value-pattern)` signature before severity is assigned.
A pattern recurring identically across many entities becomes one clustered
triage entry with an occurrence count and entity breakdown, and — this is a
deliberate inversion of today's behavior — a large, identical cluster is a
signal to score *lower*, since it more often indicates a detector or
profile problem than a security event, while a small, isolated occurrence
keeps its current weight.
The feedback/suppression layer (`feedback.py`) is not being restructured — it
is sound and stays as-is. It benefits automatically from cleaner input: fewer
and better-clustered findings to review, and less need for the bulk-clear and
rule-generalization UI to compensate for structural noise.
## Module Index
| Concern | File(s) |
|---|---|
| Log ingestion | `graylog_source.py`, `graylog_aggregate.py`, `graylog_mcp.py`, `syslog_server.py`, `logs.py` |
| Field name normalization | `normalization.py` |
| Value-shape classification | `field_shapes.py` |
| Field/stream discovery accumulation | `history.py` (`FieldDiscoveryStore`) |
| Profile suggestion & advisor merge | `profile_suggestions.py`, `llm.py` (`ollama_profile_advice`) |
| Stored profile schema | `stream_profiles.py` |
| Baselines | `baseline.py` |
| Detectors (burst/rare-value/sequence) | `detectors.py`, `sequences.py` |
| Per-source anomaly aggregation | `anomaly.py` |
| Incident grouping | `incidents.py` |
| Triage ranking | `triage.py` |
| Recommendations | `recommendations.py` |
| Feedback/suppression engine | `feedback.py` |
| Threat intel enrichment | `threat_intel.py` |
| Mitigation/blocking | `mitigation.py` |
| Metrics export | `metrics.py` |
| Orchestration (poll cycle, CLI monitor loop) | `monitor.py` |
| Dashboard (HTTP + UI) | `dashboard.py` |
| CLI entry points | `cli.py` |
## See Also
- [`README.md`](README.md) — day-to-day usage, dashboard workflow, settings reference.
- [`ROADMAP.md`](ROADMAP.md) — prioritized work, including section 0 for this rework.
- [`systemd/README.md`](systemd/README.md) — production deployment.