Add dashboard-triggered replay jobs and detector deltas

Closes the last ROADMAP §1 item. The replay/replay-graylog CLI backend
already existed and was already safe (a temporary, auto-cleaned SQLite
baseline, never touching live state) — the gap was purely that a human had
to SSH in and run it by hand. Extracts the core logic from cli.py into
run_local_replay()/run_graylog_replay() in replay.py, reused by both the
CLI (now thin wrappers) and a new dashboard panel (Diagnostics -> Replay &
Detector Deltas). Runs as a background job via an in-memory job store,
since the dashboard already runs ThreadingHTTPServer and this session
directly observed live MCP polling take anywhere from 9 minutes to over an
hour — a synchronous request/response was never viable. Deliberately
serialized to one job at a time given this box's GPU/CPU constraints.

Also removed a redundant "Detector Precision" panel that duplicated (with
a weaker, current-poll-cycle-only computation) what the new Detector
Health panel already does correctly with real historical data.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
larssand
2026-09-24 18:39:54 +02:00
parent bc4d2f465f
commit 3e01ac977d
8 changed files with 474 additions and 156 deletions

View File

@@ -93,7 +93,17 @@ Goal: make findings more accurate before adding more integrations.
- [x] Add per-stream detector enablement and thresholds in the UI.
- [x] Add a dry-run replay command for historic JSONL or Graylog exports using temporary baselines.
- [x] Add direct Graylog MCP time-range replay and result comparison against saved detector configurations.
- [ ] Add dashboard controls for launching safe replay jobs and viewing detector deltas.
- [x] Add dashboard controls for launching safe replay jobs and viewing
detector deltas. The backend already existed (`replay`/`replay-graylog`
CLI commands, both against a temporary, auto-cleaned SQLite baseline that
never touches live state) — the gap was that a human had to SSH in and
run it by hand. Now runs from `Diagnostics -> Replay & Detector Deltas` as
a background job (the dashboard already runs `ThreadingHTTPServer`, so a
replay job doesn't block normal status polling), deliberately serialized
to one job at a time since this box is already resource-constrained
during a live poll. Along the way, removed a redundant, misleading old
"Detector Precision" panel that duplicated (with a weaker, current-cycle-only
computation) what the new Detector Health panel already does correctly.
- [x] Add baseline confidence tooling: per detector learning state, expected
false-positive rate, and why a deviation crossed threshold. The prior
"detector precision" metric only existed in the Prometheus endpoint and
@@ -215,13 +225,12 @@ Acceptance: adding a new Graylog stream normally requires a profile template and
## Suggested Next Milestone
Section 0 (Structural Rework) is complete and verified against the real
25-stream production deployment that motivated it: the `context`
relationship flood dropped from 91 deviations to 0, total field deviations
from 117 to 17, and triage severity now differentiates (10 critical / 8
medium / 7 high, versus 25/25 critical before). See
[`ARCHITECTURE.md`](ARCHITECTURE.md) for the full before/after. Sections 1
(Detection Quality, in particular baseline confidence tooling) and 6
(Dashboard Usability, compact incident cards) were previously blocked on
this and are the natural next targets, now that findings are clustered and
scored correctly.
Sections 0 (Structural Rework) and 1 (Detection Quality) are both complete.
Section 0 is verified against the real 25-stream production deployment
that motivated it: the `context` relationship flood dropped from 91
deviations to 0, total field deviations from 117 to 17, and triage
severity now differentiates (10 critical / 8 medium / 7 high, versus
25/25 critical before) — see [`ARCHITECTURE.md`](ARCHITECTURE.md) for the
full before/after. Section 6 (Dashboard Usability, in particular compact
incident cards) was previously blocked on section 0 and is the natural
next target, now that findings are clustered and scored correctly.