fgAI/ROADMAP.md

SignalScope Roadmap

This roadmap tracks the path from the current local Graylog analytics agent to a production-ready, multi-source security analytics tool. Items are ordered by operational value and dependency, not by UI appeal.

Current Foundation

• Graylog MCP connection, stream discovery, field discovery, and stream selection.
• Per-stream profiles for entity, timestamp, categorical, and numeric fields.
• Local five-minute field baselines with duplicate-event protection.
• Time-aware baseline comparison using matching UTC weekday/hour when available.
• Generic entity correlation for IP addresses, users, and hostnames.
• Centralized common field alias normalization for firewall, DNS, proxy, endpoint, and web logs.
• Cross-stream correlation timelines and investigation incident grouping.
• Field-deviation review: expected, false positive, confirmed, note, and expiry.
• Local Ollama analyst assessment with incident and feedback context.
• Cached AbuseIPDB/VirusTotal enrichment with a daily request budget.
• Prometheus-compatible low-cardinality metrics endpoint.
• Local dashboard for status, findings, incidents, diagnostics, and configuration.

0. Structural Rework: Shape-Validated Discovery, Shared Baselines, Clustered Findings

Goal: fix root causes found by auditing a real 25-stream / ~1M-event/day production deployment, instead of continuing to patch symptoms in the UI. See ARCHITECTURE.md for the full data-flow diagram and the detailed rationale behind each item below. This section blocks/supersedes parts of sections 1 and 6 below (baseline confidence tooling and compact incident cards both depend on findings being clustered correctly first) — do this section before spending more effort on either.

• Validate name-matched fields against observed value shape too, not just fields that had no name match. A field that matches a trusted name (e.g. context) but is actually free-text (a full raw log line, confirmed in 12 of 25 audited stream profiles) must not be trusted just because its name matched a known-useful alias.
• Make relationship-candidate selection shape-aware: never propose a free-text-shaped field as a relationship target, regardless of how it entered categorical_fields. This alone was responsible for 17 of 25 entries in one real triage queue snapshot (91 individual new_relationship deviations from a single misconfigured *->context pairing, repeated across 5 streams).
• Re-validate existing stored profiles' relationship_fields and categorical_fields against current observed shapes on an ongoing basis, not just at profile-creation time — flag (never silently rewrite) fields that no longer look fit for their assigned role, and surface a one-click "remove this field from relationships" action from the flag.
• Pool baseline state by shared entity concept across streams that shared-field discovery already recognizes as the same real-world entity (for example a source IP seen in both a firewall stream and an IIS stream), instead of a cold, isolated baseline per stream per field. A value already known-normal in one contributing stream must not register as "new" the first time a different, related stream happens to see it. Implemented as an additive pooled scope (no schema migration — reuses the existing per-stream_id-keyed tables under a synthetic shared_entity:<group> namespace) covering entity presence/age and known relationship-target values; per-stream event-rate/detector-burst/numeric-distribution baselines are intentionally left untouched since those legitimately differ by log source. Pooling arbitrary categorical rare_value newness is a deferred follow-up, not part of this pass.
• Cluster field deviations by (detector, field) signature before scoring, not after (value-pattern was dropped from the original signature — detector+field alone already matched the real incident's cross-stream pattern, and adding value-pattern would have under-clustered near-identical but not-byte-identical raw messages). A pattern recurring identically across many entities surfaces as one clustered item with an occurrence count (cluster_size) shown inline in the Evidence column, not N independently-scored "critical" rows that all have to be discovered, selected, and bulk-cleared by hand after the fact.
• Recalibrate severity so cluster size is itself a signal: a pattern hitting five or more entities identically is capped at score 25 regardless of its uncapped score, so it never outranks an isolated, uncapped occurrence.
• Update README usage documentation alongside each item above as it ships — a behavior change is not done until it is documented.

Acceptance: replaying the dataset that produced the original 25-entry, all-"critical" triage queue (17 of them the same misconfigured relationship) produces a small number of distinct, correctly-ranked items, with the systemic pattern visible as one clustered entry showing "91 occurrences across 5 streams," not 17 separate critical rows.

1. Detection Quality

Goal: make findings more accurate before adding more integrations.

• Add baseline confidence based on sample count and time-bucket coverage.
• Add generic event-rate burst detection per stream/entity.
• Add rare-value detection with a minimum historical observation threshold.
• Add detector-specific authentication failure, DNS volume, and denied-traffic burst thresholds.
• Add configurable per-field detector weights.
• Add sequence detection, for example DNS lookup -> outbound connection -> authentication event.
• Add per-stream detector enablement and thresholds in the UI.
• Add a dry-run replay command for historic JSONL or Graylog exports using temporary baselines.
• Add direct Graylog MCP time-range replay and result comparison against saved detector configurations.
• Add dashboard controls for launching safe replay jobs and viewing detector deltas.
• Add baseline confidence tooling: per detector learning state, expected false-positive rate, and why a deviation crossed threshold. The prior "detector precision" metric only existed in the Prometheus endpoint and recomputed from whatever was active in the current poll cycle — a revoked or expired suppression rule made the signal vanish. Replaced with a permanent, append-only review-outcome log (DetectorPrecisionStore, separate from the mutable FeedbackStore) recorded the moment a human clicks Expected/False positive/Confirmed, plus per-detector learning state (BaselineStore.detector_readiness(), mirroring the existing per-field profile_readiness()) and a plain-language "what triggers this" reference — all surfaced in a new Detector Health dashboard panel.
• Add baseline maintenance tooling in the dashboard for retention, high-cardinality fields, and database compaction status.
• Add observed-field profile recommendations so streams can get suggested entity, time, categorical, numeric, and detector settings.
• Add optional Ollama profile advisor with validated JSON output for semantic field mapping.
• Let the dashboard's Ollama analyst note reason about a small, bounded number of real log message excerpts from the top-scoring field deviations and incidents, not only pre-computed statistical scores — so it can flag suspicious content (an injection attempt in a URL, an unusual command line) that structural/statistical detectors can't see. Kept deliberately narrow (a handful of already-truncated excerpts from already-triaged top items, not the raw firehose) to stay within a local, GPU-constrained model's prompt budget, and purely qualitative — it changes the note's wording only, never scoring or suppression.

Acceptance: each finding shows its detector, confidence, baseline sample count, current value, expected value, and a bounded set of raw-event references.

2. Entity and Incident Investigation

Goal: make one incident answer what happened, to whom, and across which sources.

• Allow multiple entity fields per stream, such as user plus source IP plus hostname.
• Add entity aliasing: map DHCP, VPN, DNS, and endpoint identities to the same host where evidence supports it.
• Add incident lifecycle: open, acknowledged, resolved, reopened.
• Persist incident state and analyst notes separately from transient detection output.
• Add direct Graylog query links or query details for each timeline event.
• Add investigation export as JSON and Markdown report.
• Add guided investigation tools that compare a selected incident against its baseline, related entities, and similar prior outcomes.

Acceptance: an analyst can open an incident, see an ordered multi-stream timeline, review evidence, and record an outcome without losing it after the next monitor poll.

3. Feedback and Learning Controls

Goal: reduce repeat noise without hiding genuinely new behavior.

• Show active feedback rules, expiry, scope, author, and matching findings in the UI.
• Support feedback scope choices: exact value, entity and field, entire entity, or stream-wide rule.
• Require an expiry for broad suppressions and warn before a broad rule is saved.
• Track detector precision using confirmed versus false-positive feedback.
• Include feedback-derived precision and suppression counts in metrics and diagnostics.

Acceptance: a false-positive decision suppresses only the intended pattern and all suppressions are auditable and reversible.

4. Threat Intelligence and Enrichment

Goal: enrich public indicators predictably without exhausting provider quotas.

• Make cache TTL and daily budget configurable in the dashboard.
• Add provider health, cache hit ratio, and quota/budget visibility.
• Add domain and URL enrichment behind separate provider budgets.
• Keep enrichment asynchronous so a slow provider never delays monitoring.
• Store enrichment provenance and observation timestamps with incident evidence.

Acceptance: no refresh cycle causes uncontrolled external lookups, and every reputation result shows provider, timestamp, and cache state.

5. Production Operations

Goal: run reliably in a monitored environment.

• Add systemd unit files for monitor, dashboard, and optional local syslog listener.
• Add health and readiness endpoints with last successful Graylog fetch time.
• Add configurable retention for baseline buckets and high-cardinality field values.
• Add structured application logs and configurable retention for status/history data.
• Add backup and migration procedure for SQLite state.
• Add Checkmk local-check output in addition to Prometheus metrics.
• Add authentication/reverse-proxy guidance before exposing the dashboard beyond loopback.

Acceptance: deployment, restart, upgrade, backup, and monitoring have documented, repeatable procedures.

6. Dashboard Usability

Goal: optimize the UI for security investigation rather than raw tables.

• Separate stream enablement from one-profile-at-a-time profile editing.
• Add stream coverage diagnostics for enabled/profiled/ready/no-event streams.
• Replace remaining long tables with compact incident and entity cards where appropriate.
• Add incident filters for stream, severity, entity type, review state, and time range.
• Add baseline versus current charts per selected entity and field.
• Add an interactive entity graph backed by actual correlations.
• Preserve user view state, filters, expanded evidence, and selected tab across refreshes.
• Add visual data-quality warnings for missing timestamps, entity fields, or truncated Graylog results.

Acceptance: common triage can be completed from the dashboard without manually parsing raw JSON or searching for stream IDs.

7. Integrations and Extensibility

Goal: add log sources and outputs without adding source-specific logic everywhere.

• Define versioned stream-profile templates for FortiGate, Windows, DNS/AdGuard, Nginx, Squid, VPN, and Proxmox.
• Add inventory-style stream coverage to guide which streams need profiles before templates are added.
• Add common alias normalizer so new firewall/proxy/DNS streams can populate source, destination, action, severity, service, and context without source-specific code.
• Add generated stream-profile recommendations from observed common denominators.
• Add import/export for profile templates and detector settings.
• Separate source adapters, normalizers, detectors, enrichers, and output adapters into explicit extension interfaces.
• Add optional webhook/SIEM ticket output for confirmed high-severity incidents.
• Add role-aware configuration only if the dashboard is moved beyond single-user local operation.

Acceptance: adding a new Graylog stream normally requires a profile template and configuration, not a core-code change.

Suggested Next Milestone

Section 0 (Structural Rework) is complete and verified against the real 25-stream production deployment that motivated it: the context relationship flood dropped from 91 deviations to 0, total field deviations from 117 to 17, and triage severity now differentiates (10 critical / 8 medium / 7 high, versus 25/25 critical before). See ARCHITECTURE.md for the full before/after. Sections 1 (Detection Quality, in particular baseline confidence tooling) and 6 (Dashboard Usability, compact incident cards) were previously blocked on this and are the natural next targets, now that findings are clustered and scored correctly.