diff --git a/README.md b/README.md index 9d1ae56..24c12ae 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,8 @@ FortiGate is one supported example. The same workflow applies to DNS/AdGuard, Wi The Python module and legacy `fgai` command remain available for compatibility. New installations can use `signalscope`. +The prioritized implementation plan is tracked in [ROADMAP.md](ROADMAP.md). + Autoblocking is dry-run by default. The tool will not block RFC1918, loopback, multicast, link-local, reserved, or allowlisted addresses unless you change the code. ## Screenshots diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..40edaea --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,113 @@ +# SignalScope Roadmap + +This roadmap tracks the path from the current local Graylog analytics agent to a +production-ready, multi-source security analytics tool. Items are ordered by +operational value and dependency, not by UI appeal. + +## Current Foundation + +- [x] Graylog MCP connection, stream discovery, field discovery, and stream selection. +- [x] Per-stream profiles for entity, timestamp, categorical, and numeric fields. +- [x] Local five-minute field baselines with duplicate-event protection. +- [x] Time-aware baseline comparison using matching UTC weekday/hour when available. +- [x] Generic entity correlation for IP addresses, users, and hostnames. +- [x] Cross-stream correlation timelines and investigation incident grouping. +- [x] Field-deviation review: expected, false positive, confirmed, note, and expiry. +- [x] Local Ollama analyst assessment with incident and feedback context. +- [x] Cached AbuseIPDB/VirusTotal enrichment with a daily request budget. +- [x] Prometheus-compatible low-cardinality metrics endpoint. +- [x] Local dashboard for status, findings, incidents, diagnostics, and configuration. + +## 1. Detection Quality + +Goal: make findings more accurate before adding more integrations. + +- [ ] Add baseline confidence based on sample count, profile age, and time-bucket coverage. +- [ ] Add burst detection for event rate, authentication failures, DNS volume, and denied traffic. +- [ ] Add rare-value detection with frequency thresholds and configurable field weights. +- [ ] Add sequence detection, for example DNS lookup -> outbound connection -> authentication event. +- [ ] Add per-stream detector enablement and thresholds in the UI. +- [ ] Add a dry-run replay command to evaluate detector changes against a selected historic Graylog time range. + +Acceptance: each finding shows its detector, confidence, baseline sample count, current value, expected value, and a bounded set of raw-event references. + +## 2. Entity and Incident Investigation + +Goal: make one incident answer what happened, to whom, and across which sources. + +- [ ] Allow multiple entity fields per stream, such as user plus source IP plus hostname. +- [ ] Add entity aliasing: map DHCP, VPN, DNS, and endpoint identities to the same host where evidence supports it. +- [ ] Add configurable incident grouping windows and incident lifecycle: open, acknowledged, resolved, reopened. +- [ ] Persist incident state and analyst notes separately from transient detection output. +- [ ] Add direct Graylog query links or query details for each timeline event. +- [ ] Add investigation export as JSON and Markdown report. + +Acceptance: an analyst can open an incident, see an ordered multi-stream timeline, review evidence, and record an outcome without losing it after the next monitor poll. + +## 3. Feedback and Learning Controls + +Goal: reduce repeat noise without hiding genuinely new behavior. + +- [ ] Show active feedback rules, expiry, scope, author, and matching findings in the UI. +- [ ] Support feedback scope choices: exact value, entity and field, entire entity, or stream-wide rule. +- [ ] Require an expiry for broad suppressions and warn before a broad rule is saved. +- [ ] Track detector precision using confirmed versus false-positive feedback. +- [ ] Include feedback-derived precision and suppression counts in metrics and diagnostics. + +Acceptance: a false-positive decision suppresses only the intended pattern and all suppressions are auditable and reversible. + +## 4. Threat Intelligence and Enrichment + +Goal: enrich public indicators predictably without exhausting provider quotas. + +- [ ] Make cache TTL and daily budget configurable in the dashboard. +- [ ] Add provider health, cache hit ratio, and quota/budget visibility. +- [ ] Add domain and URL enrichment behind separate provider budgets. +- [ ] Keep enrichment asynchronous so a slow provider never delays monitoring. +- [ ] Store enrichment provenance and observation timestamps with incident evidence. + +Acceptance: no refresh cycle causes uncontrolled external lookups, and every reputation result shows provider, timestamp, and cache state. + +## 5. Production Operations + +Goal: run reliably in a monitored environment. + +- [ ] Add systemd unit files for monitor, dashboard, and optional local syslog listener. +- [ ] Add health and readiness endpoints with last successful Graylog fetch time. +- [ ] Add structured application logs and configurable retention for status/history/baseline data. +- [ ] Add backup and migration procedure for SQLite state. +- [ ] Add Checkmk local-check output in addition to Prometheus metrics. +- [ ] Add authentication/reverse-proxy guidance before exposing the dashboard beyond loopback. + +Acceptance: deployment, restart, upgrade, backup, and monitoring have documented, repeatable procedures. + +## 6. Dashboard Usability + +Goal: optimize the UI for security investigation rather than raw tables. + +- [ ] Replace remaining long tables with compact incident and entity cards where appropriate. +- [ ] Add incident filters for stream, severity, entity type, review state, and time range. +- [ ] Add baseline versus current charts per selected entity and field. +- [ ] Add an interactive entity graph backed by actual correlations. +- [ ] Preserve user view state, filters, expanded evidence, and selected tab across refreshes. +- [ ] Add visual data-quality warnings for missing timestamps, entity fields, or truncated Graylog results. + +Acceptance: common triage can be completed from the dashboard without manually parsing raw JSON or searching for stream IDs. + +## 7. Integrations and Extensibility + +Goal: add log sources and outputs without adding source-specific logic everywhere. + +- [ ] Define versioned stream-profile templates for FortiGate, Windows, DNS/AdGuard, Nginx, Squid, VPN, and Proxmox. +- [ ] Add import/export for profile templates and detector settings. +- [ ] Separate source adapters, normalizers, detectors, enrichers, and output adapters into explicit extension interfaces. +- [ ] Add optional webhook/SIEM ticket output for confirmed high-severity incidents. +- [ ] Add role-aware configuration only if the dashboard is moved beyond single-user local operation. + +Acceptance: adding a new Graylog stream normally requires a profile template and configuration, not a core-code change. + +## Suggested Next Milestone + +Complete Detection Quality first: confidence scoring, burst/rare-value detectors, +per-stream thresholds, and historical replay. These features determine whether +the incident and dashboard work remains trustworthy as more log sources are added. diff --git a/src/fgai/dashboard.py b/src/fgai/dashboard.py index 9b54f65..bfde00f 100644 --- a/src/fgai/dashboard.py +++ b/src/fgai/dashboard.py @@ -60,6 +60,9 @@ HTML = """ .review-actions button[data-status="false_positive"] { border-color: #b7823a; color: #ffd36e; } .review-actions button[data-status="confirmed"] { border-color: #2a9b6e; color: #7be3ae; } .chart { width: 100%; height: 220px; background: #04182d; border: 1px solid #163b59; } + .graph { width: 100%; height: 300px; background: #04182d; border: 1px solid #163b59; } + .sort-button { border: 0; background: transparent; color: #83bce9; cursor: pointer; font: inherit; font-weight: 600; padding: 0; } + .sort-button:hover { color: #d9e8f7; } @media (max-width: 860px) { .hero, .split { grid-template-columns: 1fr; } .hero img { display: none; } } @@ -74,10 +77,10 @@ HTML = """ -