Pool baseline state across streams sharing an entity concept
Completes the last deferred item from ROADMAP §0: entities recognized as the same real-world concept across enabled streams (e.g. srcip in both a firewall and an IIS stream) now share baseline knowledge of entity presence/age and known relationship-target values, instead of each stream learning it cold. Implemented additively (synthetic shared_entity:<group> namespace in the existing baseline.py tables, no schema migration), scoped to entity presence + relationship known-values only — per-stream distribution baselines stay untouched. Also fixes a latent alias-map collision bug found while building this (ALIAS_CANONICAL_BY_FIELD let "context" silently steal "hostname"/"dns_query" from their own canonical groups). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
29
ROADMAP.md
29
ROADMAP.md
@@ -45,13 +45,19 @@ this section before spending more effort on either.
|
||||
not just at profile-creation time — flag (never silently rewrite) fields
|
||||
that no longer look fit for their assigned role, and surface a one-click
|
||||
"remove this field from relationships" action from the flag.
|
||||
- [ ] Pool baseline state by shared entity concept across streams that
|
||||
- [x] Pool baseline state by shared entity concept across streams that
|
||||
shared-field discovery already recognizes as the same real-world entity
|
||||
(for example a source IP seen in both a firewall stream and an IIS
|
||||
stream), instead of a cold, isolated baseline per stream per field. A value
|
||||
already known-normal in one contributing stream must not register as "new"
|
||||
the first time a different, related stream happens to see it. Deferred:
|
||||
needs a `baseline.py` schema change and its own dedicated planning pass.
|
||||
the first time a different, related stream happens to see it. Implemented
|
||||
as an additive pooled scope (no schema migration — reuses the existing
|
||||
per-`stream_id`-keyed tables under a synthetic `shared_entity:<group>`
|
||||
namespace) covering entity presence/age and known relationship-target
|
||||
values; per-stream event-rate/detector-burst/numeric-distribution
|
||||
baselines are intentionally left untouched since those legitimately
|
||||
differ by log source. Pooling arbitrary categorical `rare_value` newness
|
||||
is a deferred follow-up, not part of this pass.
|
||||
- [x] Cluster field deviations by `(detector, field)` signature *before*
|
||||
scoring, not after (value-pattern was dropped from the original signature —
|
||||
detector+field alone already matched the real incident's cross-stream
|
||||
@@ -179,10 +185,13 @@ Acceptance: adding a new Graylog stream normally requires a profile template and
|
||||
|
||||
## Suggested Next Milestone
|
||||
|
||||
Complete section 0 (Structural Rework) first. It was added after auditing a
|
||||
real production deployment and finding that the existing Detection Quality
|
||||
and Dashboard Usability work, while individually correct, sits on top of a
|
||||
discovery/scoring pipeline that lets one misconfigured field flood the triage
|
||||
queue with dozens of look-alike "critical" entries. Fixing the pipeline is
|
||||
higher leverage than any further UI work until it's done — see
|
||||
[`ARCHITECTURE.md`](ARCHITECTURE.md).
|
||||
Section 0 (Structural Rework) is complete and verified against the real
|
||||
25-stream production deployment that motivated it: the `context`
|
||||
relationship flood dropped from 91 deviations to 0, total field deviations
|
||||
from 117 to 17, and triage severity now differentiates (10 critical / 8
|
||||
medium / 7 high, versus 25/25 critical before). See
|
||||
[`ARCHITECTURE.md`](ARCHITECTURE.md) for the full before/after. Sections 1
|
||||
(Detection Quality, in particular baseline confidence tooling) and 6
|
||||
(Dashboard Usability, compact incident cards) were previously blocked on
|
||||
this and are the natural next targets, now that findings are clustered and
|
||||
scored correctly.
|
||||
|
||||
Reference in New Issue
Block a user