diff --git a/README.md b/README.md index 9d1ae56..24c12ae 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,8 @@ FortiGate is one supported example. The same workflow applies to DNS/AdGuard, Wi The Python module and legacy `fgai` command remain available for compatibility. New installations can use `signalscope`. +The prioritized implementation plan is tracked in [ROADMAP.md](ROADMAP.md). + Autoblocking is dry-run by default. The tool will not block RFC1918, loopback, multicast, link-local, reserved, or allowlisted addresses unless you change the code. ## Screenshots diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..40edaea --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,113 @@ +# SignalScope Roadmap + +This roadmap tracks the path from the current local Graylog analytics agent to a +production-ready, multi-source security analytics tool. Items are ordered by +operational value and dependency, not by UI appeal. + +## Current Foundation + +- [x] Graylog MCP connection, stream discovery, field discovery, and stream selection. +- [x] Per-stream profiles for entity, timestamp, categorical, and numeric fields. +- [x] Local five-minute field baselines with duplicate-event protection. +- [x] Time-aware baseline comparison using matching UTC weekday/hour when available. +- [x] Generic entity correlation for IP addresses, users, and hostnames. +- [x] Cross-stream correlation timelines and investigation incident grouping. +- [x] Field-deviation review: expected, false positive, confirmed, note, and expiry. +- [x] Local Ollama analyst assessment with incident and feedback context. +- [x] Cached AbuseIPDB/VirusTotal enrichment with a daily request budget. +- [x] Prometheus-compatible low-cardinality metrics endpoint. +- [x] Local dashboard for status, findings, incidents, diagnostics, and configuration. + +## 1. Detection Quality + +Goal: make findings more accurate before adding more integrations. + +- [ ] Add baseline confidence based on sample count, profile age, and time-bucket coverage. +- [ ] Add burst detection for event rate, authentication failures, DNS volume, and denied traffic. +- [ ] Add rare-value detection with frequency thresholds and configurable field weights. +- [ ] Add sequence detection, for example DNS lookup -> outbound connection -> authentication event. +- [ ] Add per-stream detector enablement and thresholds in the UI. +- [ ] Add a dry-run replay command to evaluate detector changes against a selected historic Graylog time range. + +Acceptance: each finding shows its detector, confidence, baseline sample count, current value, expected value, and a bounded set of raw-event references. + +## 2. Entity and Incident Investigation + +Goal: make one incident answer what happened, to whom, and across which sources. + +- [ ] Allow multiple entity fields per stream, such as user plus source IP plus hostname. +- [ ] Add entity aliasing: map DHCP, VPN, DNS, and endpoint identities to the same host where evidence supports it. +- [ ] Add configurable incident grouping windows and incident lifecycle: open, acknowledged, resolved, reopened. +- [ ] Persist incident state and analyst notes separately from transient detection output. +- [ ] Add direct Graylog query links or query details for each timeline event. +- [ ] Add investigation export as JSON and Markdown report. + +Acceptance: an analyst can open an incident, see an ordered multi-stream timeline, review evidence, and record an outcome without losing it after the next monitor poll. + +## 3. Feedback and Learning Controls + +Goal: reduce repeat noise without hiding genuinely new behavior. + +- [ ] Show active feedback rules, expiry, scope, author, and matching findings in the UI. +- [ ] Support feedback scope choices: exact value, entity and field, entire entity, or stream-wide rule. +- [ ] Require an expiry for broad suppressions and warn before a broad rule is saved. +- [ ] Track detector precision using confirmed versus false-positive feedback. +- [ ] Include feedback-derived precision and suppression counts in metrics and diagnostics. + +Acceptance: a false-positive decision suppresses only the intended pattern and all suppressions are auditable and reversible. + +## 4. Threat Intelligence and Enrichment + +Goal: enrich public indicators predictably without exhausting provider quotas. + +- [ ] Make cache TTL and daily budget configurable in the dashboard. +- [ ] Add provider health, cache hit ratio, and quota/budget visibility. +- [ ] Add domain and URL enrichment behind separate provider budgets. +- [ ] Keep enrichment asynchronous so a slow provider never delays monitoring. +- [ ] Store enrichment provenance and observation timestamps with incident evidence. + +Acceptance: no refresh cycle causes uncontrolled external lookups, and every reputation result shows provider, timestamp, and cache state. + +## 5. Production Operations + +Goal: run reliably in a monitored environment. + +- [ ] Add systemd unit files for monitor, dashboard, and optional local syslog listener. +- [ ] Add health and readiness endpoints with last successful Graylog fetch time. +- [ ] Add structured application logs and configurable retention for status/history/baseline data. +- [ ] Add backup and migration procedure for SQLite state. +- [ ] Add Checkmk local-check output in addition to Prometheus metrics. +- [ ] Add authentication/reverse-proxy guidance before exposing the dashboard beyond loopback. + +Acceptance: deployment, restart, upgrade, backup, and monitoring have documented, repeatable procedures. + +## 6. Dashboard Usability + +Goal: optimize the UI for security investigation rather than raw tables. + +- [ ] Replace remaining long tables with compact incident and entity cards where appropriate. +- [ ] Add incident filters for stream, severity, entity type, review state, and time range. +- [ ] Add baseline versus current charts per selected entity and field. +- [ ] Add an interactive entity graph backed by actual correlations. +- [ ] Preserve user view state, filters, expanded evidence, and selected tab across refreshes. +- [ ] Add visual data-quality warnings for missing timestamps, entity fields, or truncated Graylog results. + +Acceptance: common triage can be completed from the dashboard without manually parsing raw JSON or searching for stream IDs. + +## 7. Integrations and Extensibility + +Goal: add log sources and outputs without adding source-specific logic everywhere. + +- [ ] Define versioned stream-profile templates for FortiGate, Windows, DNS/AdGuard, Nginx, Squid, VPN, and Proxmox. +- [ ] Add import/export for profile templates and detector settings. +- [ ] Separate source adapters, normalizers, detectors, enrichers, and output adapters into explicit extension interfaces. +- [ ] Add optional webhook/SIEM ticket output for confirmed high-severity incidents. +- [ ] Add role-aware configuration only if the dashboard is moved beyond single-user local operation. + +Acceptance: adding a new Graylog stream normally requires a profile template and configuration, not a core-code change. + +## Suggested Next Milestone + +Complete Detection Quality first: confidence scoring, burst/rare-value detectors, +per-stream thresholds, and historical replay. These features determine whether +the incident and dashboard work remains trustworthy as more log sources are added. diff --git a/src/fgai/dashboard.py b/src/fgai/dashboard.py index 9b54f65..bfde00f 100644 --- a/src/fgai/dashboard.py +++ b/src/fgai/dashboard.py @@ -60,6 +60,9 @@ HTML = """ .review-actions button[data-status="false_positive"] { border-color: #b7823a; color: #ffd36e; } .review-actions button[data-status="confirmed"] { border-color: #2a9b6e; color: #7be3ae; } .chart { width: 100%; height: 220px; background: #04182d; border: 1px solid #163b59; } + .graph { width: 100%; height: 300px; background: #04182d; border: 1px solid #163b59; } + .sort-button { border: 0; background: transparent; color: #83bce9; cursor: pointer; font: inherit; font-weight: 600; padding: 0; } + .sort-button:hover { color: #d9e8f7; } @media (max-width: 860px) { .hero, .split { grid-template-columns: 1fr; } .hero img { display: none; } } @@ -74,10 +77,10 @@ HTML = """ -

Events and Anomalies

Baseline and Stream Health

AI Assessment

LLM assessment disabled.

Investigation Incidents

Anomalies

Recommendations

+

Events and Anomalies

Baseline and Stream Health

Correlation Map

AI Assessment

LLM assessment disabled.

Investigation Incidents

Anomalies

Recommendations

Field Baseline Deviations

Related Activity Across Sources

Block Candidates

Threat Intelligence

Policy Findings

Diagnostics

-

Runtime Configuration

+

Runtime Configuration