Public/fgAI
SHA256
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add real detector health tooling: learning state and historical precision

Browse Source 
Replaces the old Prometheus-only "detector precision" metric, which
recomputed from whatever field_deviations happened to be active in the
current poll cycle cross-referenced against live FeedbackStore rules — a
revoked or expired suppression rule made the signal vanish, and nothing was
tracked between polls. Adds DetectorPrecisionStore, a permanent append-only
log of review outcomes (kept deliberately separate from FeedbackStore's
mutable suppression rules) recorded the moment a human reviews a finding,
plus BaselineStore.detector_readiness() mirroring the existing per-field
profile_readiness() for per-detector learning state, and a plain-language
detector reference. Surfaced in a new Detector Health dashboard panel.
Observability only — nothing here changes scoring or suppression.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
larssand
2026-09-24 16:22:02 +02:00
parent b0cc9dbdfe
commit 36516faf9a
11 changed files with 315 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
ROADMAP.md
View File

@@ -94,7 +94,17 @@ Goal: make findings more accurate before adding more integrations.
- [x] Add a dry-run replay command for historic JSONL or Graylog exports using temporary baselines.
- [x] Add direct Graylog MCP time-range replay and result comparison against saved detector configurations.
- [ ] Add dashboard controls for launching safe replay jobs and viewing detector deltas.
- [ ] Add baseline confidence tooling: per detector learning state, expected false-positive rate, and why a deviation crossed threshold.
- [x] Add baseline confidence tooling: per detector learning state, expected
false-positive rate, and why a deviation crossed threshold. The prior
"detector precision" metric only existed in the Prometheus endpoint and
recomputed from whatever was active in the current poll cycle — a
revoked or expired suppression rule made the signal vanish. Replaced with
a permanent, append-only review-outcome log (`DetectorPrecisionStore`,
separate from the mutable `FeedbackStore`) recorded the moment a human
clicks Expected/False positive/Confirmed, plus per-detector learning
state (`BaselineStore.detector_readiness()`, mirroring the existing
per-field `profile_readiness()`) and a plain-language "what triggers
this" reference — all surfaced in a new Detector Health dashboard panel.
- [ ] Add baseline maintenance tooling in the dashboard for retention, high-cardinality fields, and database compaction status.
- [x] Add observed-field profile recommendations so streams can get suggested entity, time, categorical, numeric, and detector settings.
- [x] Add optional Ollama profile advisor with validated JSON output for semantic field mapping.