Add dashboard-triggered replay jobs and detector deltas
Closes the last ROADMAP §1 item. The replay/replay-graylog CLI backend already existed and was already safe (a temporary, auto-cleaned SQLite baseline, never touching live state) — the gap was purely that a human had to SSH in and run it by hand. Extracts the core logic from cli.py into run_local_replay()/run_graylog_replay() in replay.py, reused by both the CLI (now thin wrappers) and a new dashboard panel (Diagnostics -> Replay & Detector Deltas). Runs as a background job via an in-memory job store, since the dashboard already runs ThreadingHTTPServer and this session directly observed live MCP polling take anywhere from 9 minutes to over an hour — a synchronous request/response was never viable. Deliberately serialized to one job at a time given this box's GPU/CPU constraints. Also removed a redundant "Detector Precision" panel that duplicated (with a weaker, current-poll-cycle-only computation) what the new Detector Health panel already does correctly with real historical data. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
31
ROADMAP.md
31
ROADMAP.md
@@ -93,7 +93,17 @@ Goal: make findings more accurate before adding more integrations.
|
||||
- [x] Add per-stream detector enablement and thresholds in the UI.
|
||||
- [x] Add a dry-run replay command for historic JSONL or Graylog exports using temporary baselines.
|
||||
- [x] Add direct Graylog MCP time-range replay and result comparison against saved detector configurations.
|
||||
- [ ] Add dashboard controls for launching safe replay jobs and viewing detector deltas.
|
||||
- [x] Add dashboard controls for launching safe replay jobs and viewing
|
||||
detector deltas. The backend already existed (`replay`/`replay-graylog`
|
||||
CLI commands, both against a temporary, auto-cleaned SQLite baseline that
|
||||
never touches live state) — the gap was that a human had to SSH in and
|
||||
run it by hand. Now runs from `Diagnostics -> Replay & Detector Deltas` as
|
||||
a background job (the dashboard already runs `ThreadingHTTPServer`, so a
|
||||
replay job doesn't block normal status polling), deliberately serialized
|
||||
to one job at a time since this box is already resource-constrained
|
||||
during a live poll. Along the way, removed a redundant, misleading old
|
||||
"Detector Precision" panel that duplicated (with a weaker, current-cycle-only
|
||||
computation) what the new Detector Health panel already does correctly.
|
||||
- [x] Add baseline confidence tooling: per detector learning state, expected
|
||||
false-positive rate, and why a deviation crossed threshold. The prior
|
||||
"detector precision" metric only existed in the Prometheus endpoint and
|
||||
@@ -215,13 +225,12 @@ Acceptance: adding a new Graylog stream normally requires a profile template and
|
||||
|
||||
## Suggested Next Milestone
|
||||
|
||||
Section 0 (Structural Rework) is complete and verified against the real
|
||||
25-stream production deployment that motivated it: the `context`
|
||||
relationship flood dropped from 91 deviations to 0, total field deviations
|
||||
from 117 to 17, and triage severity now differentiates (10 critical / 8
|
||||
medium / 7 high, versus 25/25 critical before). See
|
||||
[`ARCHITECTURE.md`](ARCHITECTURE.md) for the full before/after. Sections 1
|
||||
(Detection Quality, in particular baseline confidence tooling) and 6
|
||||
(Dashboard Usability, compact incident cards) were previously blocked on
|
||||
this and are the natural next targets, now that findings are clustered and
|
||||
scored correctly.
|
||||
Sections 0 (Structural Rework) and 1 (Detection Quality) are both complete.
|
||||
Section 0 is verified against the real 25-stream production deployment
|
||||
that motivated it: the `context` relationship flood dropped from 91
|
||||
deviations to 0, total field deviations from 117 to 17, and triage
|
||||
severity now differentiates (10 critical / 8 medium / 7 high, versus
|
||||
25/25 critical before) — see [`ARCHITECTURE.md`](ARCHITECTURE.md) for the
|
||||
full before/after. Section 6 (Dashboard Usability, in particular compact
|
||||
incident cards) was previously blocked on section 0 and is the natural
|
||||
next target, now that findings are clustered and scored correctly.
|
||||
|
||||
Reference in New Issue
Block a user