fgAI/ARCHITECTURE.md

SignalScope Architecture

This document explains how data moves through SignalScope today, the structural gaps found by auditing a real 25-stream production deployment, and the target architecture that ROADMAP.md section 0 is working toward. Read this before making changes to discovery, baselining, or detection — several of today's problems come from small local decisions that look correct in isolation but compound badly at real scale.

Principles

• Local-only, single-user. No data leaves the machine. No auth beyond binding to loopback (see the README's "Exposing the Dashboard Beyond Loopback" section before changing that).
• Never suppress something genuinely new. Every suppression mechanism in this codebase (the feedback engine's scope model, auto-apply's manual-field protection, drift flagging instead of silent re-correction) exists to uphold this. Any new feature that touches detection or suppression must be checked against it before it ships.
• Fix the source, not the symptom. A UI feature that makes it easier to clean up noise is worth less than a structural fix that stops the noise from being generated in a misleading shape in the first place. Section 0 of the roadmap exists because this session repeatedly reached for the former when the latter was available.

Data Flow (current)

Graylog MCP or local syslog
        |
        v
+-------------------+
| Ingestion         |  graylog_source.py / graylog_aggregate.py / syslog_server.py / logs.py
+-------------------+
        |
        v
+-------------------+
| Normalization &   |  normalization.py (name-alias canonical map)
| Discovery         |  field_shapes.py (value-shape classifier: ip/timestamp/port/
|                    |    numeric/categorical/free_text, confidence 0-100)
|                    |  history.py FieldDiscoveryStore (accumulates observed
|                    |    values across polls so discovery survives sampling)
+-------------------+
        |
        v
+-------------------+
| Profile Discovery  |  profile_suggestions.py: suggest_stream_profiles()
| (per stream)       |    - name-priority lists seed entity/categorical/numeric
|                    |      candidates (ENTITY_PRIORITY, CATEGORICAL_PRIORITY, ...)
|                    |    - field_shapes.py reinforces/extends selection for
|                    |      fields that did NOT match by name
|                    |    - _relationship_candidates() pairs entities against
|                    |      categorical/numeric fields, name-token driven
|                    |    - optional Ollama profile advisor refines only the
|                    |      fields the deterministic pass is unsure about
|                    |  stream_profiles.py: parsed StreamProfile, field_origin
|                    |    (auto/manual) per field, defaults missing to "manual"
+-------------------+
        |
        v
+-------------------+
| Baselining         |  baseline.py: BaselineStore, five-minute buckets per
| (per stream)       |    (stream, entity, field), SQLite, UTC weekday/hour
|                    |    aware comparison, retention/compaction
+-------------------+
        |
        v
+-------------------+
| Detection          |  baseline.py profile comparison -> field_deviations
|                    |  detectors.py: burst/rare-value/sequence detectors
|                    |  sequences.py: ordered cross-stream event sequences
+-------------------+
        |
        v
+-------------------+
| Aggregation        |  anomaly.py: detect_source_anomalies() sums per-source
|                    |    signals (event-rate z-scores, deny-rate, UTM count,
|                    |    new-destinations-vs-baseline, field_deviations, ...)
|                    |    into one score per source; feedback-aware (entity-
|                    |    scope suppression skips a source's finding entirely)
|                    |  incidents.py: build_incidents() groups anomalies +
|                    |    field_deviations by entity
|                    |  triage.py: build_triage_queue() ranks incidents +
|                    |    deviations by severity
|                    |  recommendations.py: iterates the same anomalies list
+-------------------+
        |
        v
+-------------------+
| Feedback /         |  feedback.py: FeedbackStore, scope model (value/field/
| Suppression        |    entity/stream), feedback_matches()/select_match()
|                    |    (most-specific-wins), suggest_generalizations()
|                    |    (suggest merging repeated narrow rules, never auto-
|                    |    applies), mandatory expiry on entity/stream scope
+-------------------+
        |
        v
+-------------------+
| Dashboard          |  dashboard.py: single-file HTTP server + polled JS UI
+-------------------+

monitor.py is the orchestrator that runs one full pass through this pipeline every poll cycle (build_status) and drives the continuous loop (monitor_loop), including auto-apply, drift flagging, and throttled Ollama calls.

Known Structural Gaps (found auditing a real deployment)

A live 25-stream deployment (~1M events/day) was audited directly while debugging why its triage queue felt unmanageable. Findings, in order of impact:

1. Name-matched fields never get shape-validated. _generic_entity_fields etc. in profile_suggestions.py only reinforce/extend selection with field_shapes.py for fields that failed to match a name-priority list. A field that already matches a trusted name — like context, which is in the common alias map as a generically useful correlation field — is accepted without ever checking whether its observed values actually look like the shape that role requires. In the audited deployment, context resolves to a full raw log line or URL (free-text) in most streams, not a low-cardinality categorical value, but it was never re-checked against that.

2. Relationship-candidate selection has no shape-awareness at all. _relationship_candidates pairs every entity field against whatever already landed in categorical_fields, purely via name-token matching (src/dst tokens, semantic group membership). It never asks "does this field's value shape make sense as a relationship target." Combined with gap 1, this let a *_ip -> context relationship (comparing a source IP against the full raw message) get selected in 12 of 25 audited stream profiles. Because a raw message is unique on essentially every event by construction, this relationship is guaranteed to look "new" every time — it produced 91 individual new_relationship field deviations across 5 streams in one poll, feeding 17 of 25 entries in the triage queue, all scored "critical."

3. Baselines are siloed per stream, even for entities shared-field discovery already recognizes as the same real-world thing. Shared-field discovery (profile_suggestions.py _shared_profile_fields, used for recommending profiles and for correlation) already knows that most of the 25 audited streams share the concept of "a source IP entity." But baselining (baseline.py) never uses that: each stream keeps a fully independent baseline per (stream, entity, field). A source IP behaving completely normally in one stream is treated as never-seen-before the first time a different stream happens to observe it, purely because of which stream polled it first — an artifact of how state is partitioned, not a real signal.

4. Findings are scored and surfaced individually, before any clustering. Nothing between "a field deviation was detected" and "it's a row in the triage queue" asks whether many of today's deviations are actually the same underlying pattern. Gap 2's 91 deviations are real per the letter of the detector logic — but they are one signal, not 91. Scoring and severity (anomaly.py, triage.py) currently have no notion of this, so a systemic misconfiguration and a genuinely isolated critical event can look identical in the UI: both max out at "critical."

These compound: gap 1 lets an unsuitable field into candidate pools, gap 2 lets it become a relationship target, gap 3 means there's no shared state to dampen the noise across related streams, and gap 4 means nothing catches the resulting flood before it reaches the human. Fixing gap 2 alone would have prevented the specific incident that triggered this audit; fixing all four is what makes the system hold up as more streams are added.

Target Architecture

• Shape-first validation everywhere a field is selected for a role, including name-matched fields, not only as a fallback for unmatched ones. Free-text-shaped fields are structurally excluded from relationship targets, and existing stored profiles are periodically re-validated against current observed shapes (flagged for review, never silently rewritten — consistent with the drift-flagging precedent already used for auto-applied field mappings).
• Shared, entity-scoped baselines for entities that shared-field discovery already recognizes as the same concept across multiple enabled streams, instead of one cold baseline per (stream, entity, field) tuple. A value known-normal in one contributing stream should not register as new in another.
• Cluster before score. Field deviations get grouped by (detector, field, value-pattern) signature before severity is assigned. A pattern recurring identically across many entities becomes one clustered triage entry with an occurrence count and entity breakdown, and — this is a deliberate inversion of today's behavior — a large, identical cluster is a signal to score lower, since it more often indicates a detector or profile problem than a security event, while a small, isolated occurrence keeps its current weight.

The feedback/suppression layer (feedback.py) is not being restructured — it is sound and stays as-is. It benefits automatically from cleaner input: fewer and better-clustered findings to review, and less need for the bulk-clear and rule-generalization UI to compensate for structural noise.

Module Index

Concern	File(s)
Log ingestion	graylog_source.py, graylog_aggregate.py, graylog_mcp.py, syslog_server.py, logs.py
Field name normalization	normalization.py
Value-shape classification	field_shapes.py
Field/stream discovery accumulation	history.py (FieldDiscoveryStore)
Profile suggestion & advisor merge	profile_suggestions.py, llm.py (ollama_profile_advice)
Stored profile schema	stream_profiles.py
Baselines	baseline.py
Detectors (burst/rare-value/sequence)	detectors.py, sequences.py
Per-source anomaly aggregation	anomaly.py
Incident grouping	incidents.py
Triage ranking	triage.py
Recommendations	recommendations.py
Feedback/suppression engine	feedback.py
Threat intel enrichment	threat_intel.py
Mitigation/blocking	mitigation.py
Metrics export	metrics.py
Orchestration (poll cycle, CLI monitor loop)	monitor.py
Dashboard (HTTP + UI)	dashboard.py
CLI entry points	cli.py

See Also

• README.md — day-to-day usage, dashboard workflow, settings reference.
• ROADMAP.md — prioritized work, including section 0 for this rework.
• systemd/README.md — production deployment.