fgAI/ARCHITECTURE.md

# SignalScope Architecture

This document explains how data moves through SignalScope today, the
structural gaps found by auditing a real 25-stream production deployment,
and the target architecture that [`ROADMAP.md`](ROADMAP.md) section 0 is
working toward. Read this before making changes to discovery, baselining, or
detection — several of today's problems come from small local decisions that
look correct in isolation but compound badly at real scale.

## Principles

- **Local-only, single-user.** No data leaves the machine. No auth beyond
  binding to loopback (see the README's "Exposing the Dashboard Beyond
  Loopback" section before changing that).
- **Never suppress something genuinely new.** Every suppression mechanism in
  this codebase (the feedback engine's scope model, auto-apply's manual-field
  protection, drift flagging instead of silent re-correction) exists to
  uphold this. Any new feature that touches detection or suppression must be
  checked against it before it ships.
- **Fix the source, not the symptom.** A UI feature that makes it easier to
  clean up noise is worth less than a structural fix that stops the noise
  from being generated in a misleading shape in the first place. Section 0 of
  the roadmap exists because this session repeatedly reached for the former
  when the latter was available.

## Data Flow (current)

```
Graylog MCP or local syslog
        |
        v
+-------------------+
| Ingestion         |  graylog_source.py / graylog_aggregate.py / syslog_server.py / logs.py
+-------------------+
        |
        v
+-------------------+
| Normalization &   |  normalization.py (name-alias canonical map)
| Discovery         |  field_shapes.py (value-shape classifier: ip/timestamp/port/
|                    |    numeric/categorical/free_text, confidence 0-100)
|                    |  history.py FieldDiscoveryStore (accumulates observed
|                    |    values across polls so discovery survives sampling)
+-------------------+
        |
        v
+-------------------+
| Profile Discovery  |  profile_suggestions.py: suggest_stream_profiles()
| (per stream)       |    - name-priority lists seed entity/categorical/numeric
|                    |      candidates (ENTITY_PRIORITY, CATEGORICAL_PRIORITY, ...)
|                    |    - field_shapes.py reinforces/extends selection for
|                    |      fields that did NOT match by name
|                    |    - _relationship_candidates() pairs entities against
|                    |      categorical/numeric fields, name-token driven
|                    |    - optional Ollama profile advisor refines only the
|                    |      fields the deterministic pass is unsure about
|                    |  stream_profiles.py: parsed StreamProfile, field_origin
|                    |    (auto/manual) per field, defaults missing to "manual"
+-------------------+
        |
        v
+-------------------+
| Baselining         |  baseline.py: BaselineStore, five-minute buckets per
| (per stream)       |    (stream, entity, field), SQLite, UTC weekday/hour
|                    |    aware comparison, retention/compaction
+-------------------+
        |
        v
+-------------------+
| Detection          |  baseline.py profile comparison -> field_deviations
|                    |  detectors.py: burst/rare-value/sequence detectors
|                    |  sequences.py: ordered cross-stream event sequences
+-------------------+
        |
        v
+-------------------+
| Aggregation        |  anomaly.py: detect_source_anomalies() sums per-source
|                    |    signals (event-rate z-scores, deny-rate, UTM count,
|                    |    new-destinations-vs-baseline, field_deviations, ...)
|                    |    into one score per source; feedback-aware (entity-
|                    |    scope suppression skips a source's finding entirely)
|                    |  incidents.py: build_incidents() groups anomalies +
|                    |    field_deviations by entity
|                    |  triage.py: build_triage_queue() ranks incidents +
|                    |    deviations by severity
|                    |  recommendations.py: iterates the same anomalies list
+-------------------+
        |
        v
+-------------------+
| Feedback /         |  feedback.py: FeedbackStore, scope model (value/field/
| Suppression        |    entity/stream), feedback_matches()/select_match()
|                    |    (most-specific-wins), suggest_generalizations()
|                    |    (suggest merging repeated narrow rules, never auto-
|                    |    applies), mandatory expiry on entity/stream scope
+-------------------+
        |
        v
+-------------------+
| Dashboard          |  dashboard.py: single-file HTTP server + polled JS UI
+-------------------+
```

`monitor.py` is the orchestrator that runs one full pass through this
pipeline every poll cycle (`build_status`) and drives the continuous loop
(`monitor_loop`), including auto-apply, drift flagging, and throttled Ollama
calls.

## Known Structural Gaps (found auditing a real deployment)

**Status: all four gaps below are now fixed and verified live** against the
same deployment that produced them (see the Target Architecture section for
what shipped, and `ROADMAP.md` section 0 for the checklist). This section is
kept as the historical record of what the audit actually found, since the
numbers below are what "verified" is measured against.

A live 25-stream deployment (~1M events/day) was audited directly while
debugging why its triage queue felt unmanageable. Findings, in order of
impact:

**1. Name-matched fields never get shape-validated.** `_generic_entity_fields`
etc. in `profile_suggestions.py` only reinforce/extend selection with
`field_shapes.py` for fields that *failed* to match a name-priority list. A
field that already matches a trusted name — like `context`, which is in the
common alias map as a generically useful correlation field — is accepted
without ever checking whether its *observed values* actually look like the
shape that role requires. In the audited deployment, `context` resolves to a
full raw log line or URL (free-text) in most streams, not a low-cardinality
categorical value, but it was never re-checked against that.

**2. Relationship-candidate selection has no shape-awareness at all.**
`_relationship_candidates` pairs every entity field against whatever already
landed in `categorical_fields`, purely via name-token matching (`src`/`dst`
tokens, semantic group membership). It never asks "does this field's value
shape make sense as a relationship target." Combined with gap 1, this let a
`*_ip -> context` relationship (comparing a source IP against the full raw
message) get selected in 12 of 25 audited stream profiles. Because a raw
message is unique on essentially every event by construction, this
relationship is *guaranteed* to look "new" every time — it produced 91
individual `new_relationship` field deviations across 5 streams in one poll,
feeding 17 of 25 entries in the triage queue, all scored "critical."

**3. Baselines are siloed per stream, even for entities shared-field
discovery already recognizes as the same real-world thing.** Shared-field
discovery (`profile_suggestions.py` `_shared_profile_fields`, used for
recommending profiles and for correlation) already knows that most of the 25
audited streams share the concept of "a source IP entity." But baselining
(`baseline.py`) never uses that: each stream keeps a fully independent
baseline per `(stream, entity, field)`. A source IP behaving completely
normally in one stream is treated as never-seen-before the first time a
*different* stream happens to observe it, purely because of which stream
polled it first — an artifact of how state is partitioned, not a real
signal.

**4. Findings are scored and surfaced individually, before any clustering.**
Nothing between "a field deviation was detected" and "it's a row in the
triage queue" asks whether many of today's deviations are actually the same
underlying pattern. Gap 2's 91 deviations are real per the letter of the
detector logic — but they are one signal, not 91. Scoring and severity
(`anomaly.py`, `triage.py`) currently have no notion of this, so a systemic
misconfiguration and a genuinely isolated critical event can look identical
in the UI: both max out at "critical."

These compound: gap 1 lets an unsuitable field into candidate pools, gap 2
lets it become a relationship target, gap 3 means there's no shared state to
dampen the noise across related streams, and gap 4 means nothing catches the
resulting flood before it reaches the human. Fixing gap 2 alone would have
prevented the specific incident that triggered this audit; fixing all four is
what makes the system hold up as more streams are added. Verified live after
all four shipped: the `context` relationship flood went from 91 deviations to
0, total field deviations from 117 to 17, and triage severity finally
differentiates (10 critical / 8 medium / 7 high, versus 25/25 critical
before).

## Target Architecture

This is now the shipped behavior, not just the target.

- **Shape-first validation everywhere a field is selected for a role**,
  including name-matched fields, not only as a fallback for unmatched ones.
  Free-text-shaped fields are structurally excluded from relationship
  targets, and existing stored profiles are periodically re-validated
  against current observed shapes (flagged for review, never silently
  rewritten — consistent with the drift-flagging precedent already used for
  auto-applied field mappings). Turning this on live surfaced a real,
  pre-existing calibration bug in `classify_field_shape()`: callers pass a
  *deduplicated* set of observed values (capped at 200 distinct entries),
  whose own length was being used as the cardinality denominator — since a
  deduplicated set is always 100% cardinality against itself, any field with
  more than ~16 distinct values looked "high cardinality" regardless of true
  traffic volume, misclassifying genuinely low-cardinality fields like
  `username` (82 distinct values out of 1,601 events) as free text. Fixed by
  passing the real event coverage (`coverage[field]`) as an explicit
  `total_events` parameter, so cardinality is judged against actual volume;
  omitting it preserves the old sample-as-population behavior for other
  callers.
- **Shared, entity-scoped baselines** for entities that shared-field
  discovery already recognizes as the same concept across multiple enabled
  streams, instead of one cold baseline per `(stream, entity, field)` tuple.
  A value known-normal in one contributing stream does not register as new
  in another. Implemented as an additive pooled scope in `baseline.py`
  (`shared_entity_pools()` in `profile_suggestions.py` computes which
  streams share an entity concept; pooled rows live under a synthetic
  `shared_entity:<group>` namespace in the *same* tables, so no schema
  migration was needed) covering entity presence/age and known
  relationship-target values specifically — the mechanism most directly
  responsible for the audited flood. Per-stream distribution baselines
  (event-rate/detector bursts, numeric baselines) stay siloed per stream, on
  purpose: those distributions genuinely differ by log source. Pooling
  arbitrary categorical `rare_value` newness is a deferred follow-up.
  Building this also surfaced and fixed a latent, previously-harmless bug in
  `ALIAS_CANONICAL_BY_FIELD`'s construction where a later, generic canonical
  group (`context`) could silently steal an alias (`hostname`, `dns_query`)
  from an earlier, more specific one.
- **Cluster before score.** Field deviations get grouped by
  `(detector, field)` signature before severity is assigned (value-pattern
  was dropped from the originally-planned signature — detector+field alone
  already matched the real incident's cross-stream pattern). A pattern
  recurring identically across many entities becomes one clustered triage
  entry with an occurrence count (`cluster_size`), and — a deliberate
  inversion of the old behavior — a cluster of 5 or more entities is capped
  at score 25, since it more often indicates a detector or profile problem
  than a security event, while a small, isolated occurrence keeps its full
  weight.

The feedback/suppression layer (`feedback.py`) is not being restructured — it
is sound and stays as-is. It benefits automatically from cleaner input: fewer
and better-clustered findings to review, and less need for the bulk-clear and
rule-generalization UI to compensate for structural noise.

## Module Index

| Concern | File(s) |
|---|---|
| Log ingestion | `graylog_source.py`, `graylog_aggregate.py`, `graylog_mcp.py`, `syslog_server.py`, `logs.py` |
| Field name normalization | `normalization.py` |
| Value-shape classification | `field_shapes.py` |
| Field/stream discovery accumulation | `history.py` (`FieldDiscoveryStore`) |
| Profile suggestion & advisor merge | `profile_suggestions.py`, `llm.py` (`ollama_profile_advice`) |
| Stored profile schema | `stream_profiles.py` |
| Baselines | `baseline.py` |
| Detectors (burst/rare-value/sequence) | `detectors.py`, `sequences.py` |
| Per-source anomaly aggregation | `anomaly.py` |
| Incident grouping | `incidents.py` |
| Triage ranking | `triage.py` |
| Recommendations | `recommendations.py` |
| Feedback/suppression engine | `feedback.py` |
| Threat intel enrichment | `threat_intel.py` |
| Mitigation/blocking | `mitigation.py` |
| Metrics export | `metrics.py` |
| Orchestration (poll cycle, CLI monitor loop) | `monitor.py` |
| Dashboard (HTTP + UI) | `dashboard.py` |
| CLI entry points | `cli.py` |

## See Also

- [`README.md`](README.md) — day-to-day usage, dashboard workflow, settings reference.
- [`ROADMAP.md`](ROADMAP.md) — prioritized work, including section 0 for this rework.
- [`systemd/README.md`](systemd/README.md) — production deployment.