Replaces the old Prometheus-only "detector precision" metric, which
recomputed from whatever field_deviations happened to be active in the
current poll cycle cross-referenced against live FeedbackStore rules — a
revoked or expired suppression rule made the signal vanish, and nothing was
tracked between polls. Adds DetectorPrecisionStore, a permanent append-only
log of review outcomes (kept deliberately separate from FeedbackStore's
mutable suppression rules) recorded the moment a human reviews a finding,
plus BaselineStore.detector_readiness() mirroring the existing per-field
profile_readiness() for per-detector learning state, and a plain-language
detector reference. Surfaced in a new Detector Health dashboard panel.
Observability only — nothing here changes scoring or suppression.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Ollama dashboard assessment was built entirely from pre-computed
statistical scores/reasons/counts — a deliberately tested compaction step
stripped all raw message content before it reached the model, presumably to
keep the prompt small for a local, GPU-constrained instance. That raw
content (sample_events/timeline, already truncated to short messages)
already flows through the pipeline; it was just filtered out at the last
step. Now the top-scoring field deviations and incidents each carry a
small, bounded number of real message excerpts, and the prompt explicitly
asks the model to flag suspicious content (injection attempts, unusual
command lines) that pure statistical scoring can't see — purely a
qualitative addition to the narrative note, never feeding back into scoring
or suppression.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
classify_field_shape() received a deduplicated set of observed values with
no real event-volume context, so cardinality was always measured against
itself — trivially 100%, making any field with more than ~16 distinct
values look "high cardinality" no matter how large the real traffic volume
was. This misclassified genuinely low-cardinality fields (e.g. username: 82
distinct values across 1,601 events) as free text, which the new shape-veto
work then surfaced as false-positive "remove this field" warnings in
production. Fixed by passing real event coverage as an explicit
total_events parameter; omitting it keeps prior behavior for other callers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Completes the last deferred item from ROADMAP §0: entities recognized as
the same real-world concept across enabled streams (e.g. srcip in both a
firewall and an IIS stream) now share baseline knowledge of entity
presence/age and known relationship-target values, instead of each stream
learning it cold. Implemented additively (synthetic shared_entity:<group>
namespace in the existing baseline.py tables, no schema migration), scoped
to entity presence + relationship known-values only — per-stream
distribution baselines stay untouched. Also fixes a latent alias-map
collision bug found while building this (ALIAS_CANONICAL_BY_FIELD let
"context" silently steal "hostname"/"dns_query" from their own canonical
groups).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Feedback rules can now be suggested for merging when a human has already
reviewed the same pattern multiple times (never guesses about new values).
LLM defaults sized for constrained VRAM instead of llama3.3/llama3.1.
Adds systemd units, a state backup script, and reverse-proxy auth guidance
for production deployment. Findings view is collapsible and sorts
suggested-review rows to the top for a quicker overview.
Active Feedback Rules gets a status filter, and Field Baseline
Deviations gets multi-select with a bulk apply bar. Bulk scope is
restricted to exact-value/field only, so a batch action can never
create a broad rule that would suppress a genuinely new deviation.
Field roles are now inferred from observed values instead of names,
with src/dst disambiguation, value-pattern detector hints, an Ollama
advisor scoped to ambiguous fields only, confidence-gated auto-apply
with manual-field protection, and drift flagging for review
Feedback rules now support value/field/entity/stream scope, require
expiry for broad suppressions, and can be revoked individually.
Adds an active-rules panel, detector precision metrics, and diagnostics.
