fgAI/ROADMAP.md

# SignalScope Roadmap

This roadmap tracks the path from the current local Graylog analytics agent to a
production-ready, multi-source security analytics tool. Items are ordered by
operational value and dependency, not by UI appeal.

## Current Foundation

- [x] Graylog MCP connection, stream discovery, field discovery, and stream selection.
- [x] Per-stream profiles for entity, timestamp, categorical, and numeric fields.
- [x] Local five-minute field baselines with duplicate-event protection.
- [x] Time-aware baseline comparison using matching UTC weekday/hour when available.
- [x] Generic entity correlation for IP addresses, users, and hostnames.
- [x] Centralized common field alias normalization for firewall, DNS, proxy, endpoint, and web logs.
- [x] Cross-stream correlation timelines and investigation incident grouping.
- [x] Field-deviation review: expected, false positive, confirmed, note, and expiry.
- [x] Local Ollama analyst assessment with incident and feedback context.
- [x] Cached AbuseIPDB/VirusTotal enrichment with a daily request budget.
- [x] Prometheus-compatible low-cardinality metrics endpoint.
- [x] Local dashboard for status, findings, incidents, diagnostics, and configuration.

## 0. Structural Rework: Shape-Validated Discovery, Shared Baselines, Clustered Findings

Goal: fix root causes found by auditing a real 25-stream / ~1M-event/day
production deployment, instead of continuing to patch symptoms in the UI. See
[`ARCHITECTURE.md`](ARCHITECTURE.md) for the full data-flow diagram and the
detailed rationale behind each item below. This section blocks/supersedes
parts of sections 1 and 6 below (baseline confidence tooling and compact
incident cards both depend on findings being clustered correctly first) — do
this section before spending more effort on either.

- [x] Validate name-matched fields against observed value shape too, not just
  fields that had no name match. A field that matches a trusted name (e.g.
  `context`) but is actually free-text (a full raw log line, confirmed in 12
  of 25 audited stream profiles) must not be trusted just because its name
  matched a known-useful alias.
- [x] Make relationship-candidate selection shape-aware: never propose a
  free-text-shaped field as a relationship target, regardless of how it
  entered `categorical_fields`. This alone was responsible for 17 of 25
  entries in one real triage queue snapshot (91 individual `new_relationship`
  deviations from a single misconfigured `*->context` pairing, repeated
  across 5 streams).
- [x] Re-validate existing stored profiles' `relationship_fields` and
  `categorical_fields` against current observed shapes on an ongoing basis,
  not just at profile-creation time — flag (never silently rewrite) fields
  that no longer look fit for their assigned role, and surface a one-click
  "remove this field from relationships" action from the flag.
- [x] Pool baseline state by shared entity concept across streams that
  shared-field discovery already recognizes as the same real-world entity
  (for example a source IP seen in both a firewall stream and an IIS
  stream), instead of a cold, isolated baseline per stream per field. A value
  already known-normal in one contributing stream must not register as "new"
  the first time a different, related stream happens to see it. Implemented
  as an additive pooled scope (no schema migration — reuses the existing
  per-`stream_id`-keyed tables under a synthetic `shared_entity:<group>`
  namespace) covering entity presence/age and known relationship-target
  values; per-stream event-rate/detector-burst/numeric-distribution
  baselines are intentionally left untouched since those legitimately
  differ by log source. Pooling arbitrary categorical `rare_value` newness
  is a deferred follow-up, not part of this pass.
- [x] Cluster field deviations by `(detector, field)` signature *before*
  scoring, not after (value-pattern was dropped from the original signature —
  detector+field alone already matched the real incident's cross-stream
  pattern, and adding value-pattern would have under-clustered near-identical
  but not-byte-identical raw messages). A pattern recurring identically
  across many entities surfaces as one clustered item with an occurrence
  count (`cluster_size`) shown inline in the Evidence column, not N
  independently-scored "critical" rows that all have to be discovered,
  selected, and bulk-cleared by hand after the fact.
- [x] Recalibrate severity so cluster size is itself a signal: a pattern
  hitting five or more entities identically is capped at score 25 regardless
  of its uncapped score, so it never outranks an isolated, uncapped
  occurrence.
- [x] Update README usage documentation alongside each item above as it
  ships — a behavior change is not done until it is documented.

Acceptance: replaying the dataset that produced the original 25-entry,
all-"critical" triage queue (17 of them the same misconfigured relationship)
produces a small number of distinct, correctly-ranked items, with the
systemic pattern visible as one clustered entry showing "91 occurrences
across 5 streams," not 17 separate critical rows.

## 1. Detection Quality

Goal: make findings more accurate before adding more integrations.

- [x] Add baseline confidence based on sample count and time-bucket coverage.
- [x] Add generic event-rate burst detection per stream/entity.
- [x] Add rare-value detection with a minimum historical observation threshold.
- [x] Add detector-specific authentication failure, DNS volume, and denied-traffic burst thresholds.
- [x] Add configurable per-field detector weights.
- [x] Add sequence detection, for example DNS lookup -> outbound connection -> authentication event.
- [x] Add per-stream detector enablement and thresholds in the UI.
- [x] Add a dry-run replay command for historic JSONL or Graylog exports using temporary baselines.
- [x] Add direct Graylog MCP time-range replay and result comparison against saved detector configurations.
- [ ] Add dashboard controls for launching safe replay jobs and viewing detector deltas.
- [ ] Add baseline confidence tooling: per detector learning state, expected false-positive rate, and why a deviation crossed threshold.
- [ ] Add baseline maintenance tooling in the dashboard for retention, high-cardinality fields, and database compaction status.
- [x] Add observed-field profile recommendations so streams can get suggested entity, time, categorical, numeric, and detector settings.
- [x] Add optional Ollama profile advisor with validated JSON output for semantic field mapping.
- [x] Let the dashboard's Ollama analyst note reason about a small, bounded
  number of real log message excerpts from the top-scoring field deviations
  and incidents, not only pre-computed statistical scores — so it can flag
  suspicious *content* (an injection attempt in a URL, an unusual command
  line) that structural/statistical detectors can't see. Kept deliberately
  narrow (a handful of already-truncated excerpts from already-triaged top
  items, not the raw firehose) to stay within a local, GPU-constrained
  model's prompt budget, and purely qualitative — it changes the note's
  wording only, never scoring or suppression.

Acceptance: each finding shows its detector, confidence, baseline sample count, current value, expected value, and a bounded set of raw-event references.

## 2. Entity and Incident Investigation

Goal: make one incident answer what happened, to whom, and across which sources.

- [x] Allow multiple entity fields per stream, such as user plus source IP plus hostname.
- [ ] Add entity aliasing: map DHCP, VPN, DNS, and endpoint identities to the same host where evidence supports it.
- [x] Add incident lifecycle: open, acknowledged, resolved, reopened.
- [x] Persist incident state and analyst notes separately from transient detection output.
- [x] Add direct Graylog query links or query details for each timeline event.
- [x] Add investigation export as JSON and Markdown report.
- [ ] Add guided investigation tools that compare a selected incident against its baseline, related entities, and similar prior outcomes.

Acceptance: an analyst can open an incident, see an ordered multi-stream timeline, review evidence, and record an outcome without losing it after the next monitor poll.

## 3. Feedback and Learning Controls

Goal: reduce repeat noise without hiding genuinely new behavior.

- [x] Show active feedback rules, expiry, scope, author, and matching findings in the UI.
- [x] Support feedback scope choices: exact value, entity and field, entire entity, or stream-wide rule.
- [x] Require an expiry for broad suppressions and warn before a broad rule is saved.
- [x] Track detector precision using confirmed versus false-positive feedback.
- [x] Include feedback-derived precision and suppression counts in metrics and diagnostics.

Acceptance: a false-positive decision suppresses only the intended pattern and all suppressions are auditable and reversible.

## 4. Threat Intelligence and Enrichment

Goal: enrich public indicators predictably without exhausting provider quotas.

- [ ] Make cache TTL and daily budget configurable in the dashboard.
- [ ] Add provider health, cache hit ratio, and quota/budget visibility.
- [ ] Add domain and URL enrichment behind separate provider budgets.
- [ ] Keep enrichment asynchronous so a slow provider never delays monitoring.
- [ ] Store enrichment provenance and observation timestamps with incident evidence.

Acceptance: no refresh cycle causes uncontrolled external lookups, and every reputation result shows provider, timestamp, and cache state.

## 5. Production Operations

Goal: run reliably in a monitored environment.

- [ ] Add systemd unit files for monitor, dashboard, and optional local syslog listener.
- [ ] Add health and readiness endpoints with last successful Graylog fetch time.
- [x] Add configurable retention for baseline buckets and high-cardinality field values.
- [ ] Add structured application logs and configurable retention for status/history data.
- [ ] Add backup and migration procedure for SQLite state.
- [ ] Add Checkmk local-check output in addition to Prometheus metrics.
- [ ] Add authentication/reverse-proxy guidance before exposing the dashboard beyond loopback.

Acceptance: deployment, restart, upgrade, backup, and monitoring have documented, repeatable procedures.

## 6. Dashboard Usability

Goal: optimize the UI for security investigation rather than raw tables.

- [x] Separate stream enablement from one-profile-at-a-time profile editing.
- [x] Add stream coverage diagnostics for enabled/profiled/ready/no-event streams.
- [ ] Replace remaining long tables with compact incident and entity cards where appropriate.
- [ ] Add incident filters for stream, severity, entity type, review state, and time range.
- [ ] Add baseline versus current charts per selected entity and field.
- [ ] Add an interactive entity graph backed by actual correlations.
- [ ] Preserve user view state, filters, expanded evidence, and selected tab across refreshes.
- [ ] Add visual data-quality warnings for missing timestamps, entity fields, or truncated Graylog results.

Acceptance: common triage can be completed from the dashboard without manually parsing raw JSON or searching for stream IDs.

## 7. Integrations and Extensibility

Goal: add log sources and outputs without adding source-specific logic everywhere.

- [ ] Define versioned stream-profile templates for FortiGate, Windows, DNS/AdGuard, Nginx, Squid, VPN, and Proxmox.
- [x] Add inventory-style stream coverage to guide which streams need profiles before templates are added.
- [x] Add common alias normalizer so new firewall/proxy/DNS streams can populate source, destination, action, severity, service, and context without source-specific code.
- [x] Add generated stream-profile recommendations from observed common denominators.
- [ ] Add import/export for profile templates and detector settings.
- [ ] Separate source adapters, normalizers, detectors, enrichers, and output adapters into explicit extension interfaces.
- [ ] Add optional webhook/SIEM ticket output for confirmed high-severity incidents.
- [ ] Add role-aware configuration only if the dashboard is moved beyond single-user local operation.

Acceptance: adding a new Graylog stream normally requires a profile template and configuration, not a core-code change.

## Suggested Next Milestone

Section 0 (Structural Rework) is complete and verified against the real
25-stream production deployment that motivated it: the `context`
relationship flood dropped from 91 deviations to 0, total field deviations
from 117 to 17, and triage severity now differentiates (10 critical / 8
medium / 7 high, versus 25/25 critical before). See
[`ARCHITECTURE.md`](ARCHITECTURE.md) for the full before/after. Sections 1
(Detection Quality, in particular baseline confidence tooling) and 6
(Dashboard Usability, compact incident cards) were previously blocked on
this and are the natural next targets, now that findings are clustered and
scored correctly.