Pool baseline state across streams sharing an entity concept

Completes the last deferred item from ROADMAP §0: entities recognized as
the same real-world concept across enabled streams (e.g. srcip in both a
firewall and an IIS stream) now share baseline knowledge of entity
presence/age and known relationship-target values, instead of each stream
learning it cold. Implemented additively (synthetic shared_entity:<group>
namespace in the existing baseline.py tables, no schema migration), scoped
to entity presence + relationship known-values only — per-stream
distribution baselines stay untouched. Also fixes a latent alias-map
collision bug found while building this (ALIAS_CANONICAL_BY_FIELD let
"context" silently steal "hostname"/"dns_query" from their own canonical
groups).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
larssand
2026-09-23 18:50:14 +02:00
parent 8c155ec9cf
commit 83ffef3570
12 changed files with 305 additions and 41 deletions

View File

@@ -45,13 +45,19 @@ this section before spending more effort on either.
not just at profile-creation time — flag (never silently rewrite) fields
that no longer look fit for their assigned role, and surface a one-click
"remove this field from relationships" action from the flag.
- [ ] Pool baseline state by shared entity concept across streams that
- [x] Pool baseline state by shared entity concept across streams that
shared-field discovery already recognizes as the same real-world entity
(for example a source IP seen in both a firewall stream and an IIS
stream), instead of a cold, isolated baseline per stream per field. A value
already known-normal in one contributing stream must not register as "new"
the first time a different, related stream happens to see it. Deferred:
needs a `baseline.py` schema change and its own dedicated planning pass.
the first time a different, related stream happens to see it. Implemented
as an additive pooled scope (no schema migration — reuses the existing
per-`stream_id`-keyed tables under a synthetic `shared_entity:<group>`
namespace) covering entity presence/age and known relationship-target
values; per-stream event-rate/detector-burst/numeric-distribution
baselines are intentionally left untouched since those legitimately
differ by log source. Pooling arbitrary categorical `rare_value` newness
is a deferred follow-up, not part of this pass.
- [x] Cluster field deviations by `(detector, field)` signature *before*
scoring, not after (value-pattern was dropped from the original signature —
detector+field alone already matched the real incident's cross-stream
@@ -179,10 +185,13 @@ Acceptance: adding a new Graylog stream normally requires a profile template and
## Suggested Next Milestone
Complete section 0 (Structural Rework) first. It was added after auditing a
real production deployment and finding that the existing Detection Quality
and Dashboard Usability work, while individually correct, sits on top of a
discovery/scoring pipeline that lets one misconfigured field flood the triage
queue with dozens of look-alike "critical" entries. Fixing the pipeline is
higher leverage than any further UI work until it's done — see
[`ARCHITECTURE.md`](ARCHITECTURE.md).
Section 0 (Structural Rework) is complete and verified against the real
25-stream production deployment that motivated it: the `context`
relationship flood dropped from 91 deviations to 0, total field deviations
from 117 to 17, and triage severity now differentiates (10 critical / 8
medium / 7 high, versus 25/25 critical before). See
[`ARCHITECTURE.md`](ARCHITECTURE.md) for the full before/after. Sections 1
(Detection Quality, in particular baseline confidence tooling) and 6
(Dashboard Usability, compact incident cards) were previously blocked on
this and are the natural next targets, now that findings are clustered and
scored correctly.