Pool baseline state across streams sharing an entity concept

Completes the last deferred item from ROADMAP §0: entities recognized as
the same real-world concept across enabled streams (e.g. srcip in both a
firewall and an IIS stream) now share baseline knowledge of entity
presence/age and known relationship-target values, instead of each stream
learning it cold. Implemented additively (synthetic shared_entity:<group>
namespace in the existing baseline.py tables, no schema migration), scoped
to entity presence + relationship known-values only — per-stream
distribution baselines stay untouched. Also fixes a latent alias-map
collision bug found while building this (ALIAS_CANONICAL_BY_FIELD let
"context" silently steal "hostname"/"dns_query" from their own canonical
groups).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
larssand
2026-09-23 18:50:14 +02:00
parent 8c155ec9cf
commit 83ffef3570
12 changed files with 305 additions and 41 deletions

View File

@@ -107,6 +107,12 @@ calls.
## Known Structural Gaps (found auditing a real deployment)
**Status: all four gaps below are now fixed and verified live** against the
same deployment that produced them (see the Target Architecture section for
what shipped, and `ROADMAP.md` section 0 for the checklist). This section is
kept as the historical record of what the audit actually found, since the
numbers below are what "verified" is measured against.
A live 25-stream deployment (~1M events/day) was audited directly while
debugging why its triage queue felt unmanageable. Findings, in order of
impact:
@@ -159,10 +165,16 @@ lets it become a relationship target, gap 3 means there's no shared state to
dampen the noise across related streams, and gap 4 means nothing catches the
resulting flood before it reaches the human. Fixing gap 2 alone would have
prevented the specific incident that triggered this audit; fixing all four is
what makes the system hold up as more streams are added.
what makes the system hold up as more streams are added. Verified live after
all four shipped: the `context` relationship flood went from 91 deviations to
0, total field deviations from 117 to 17, and triage severity finally
differentiates (10 critical / 8 medium / 7 high, versus 25/25 critical
before).
## Target Architecture
This is now the shipped behavior, not just the target.
- **Shape-first validation everywhere a field is selected for a role**,
including name-matched fields, not only as a fallback for unmatched ones.
Free-text-shaped fields are structurally excluded from relationship
@@ -173,16 +185,31 @@ what makes the system hold up as more streams are added.
- **Shared, entity-scoped baselines** for entities that shared-field
discovery already recognizes as the same concept across multiple enabled
streams, instead of one cold baseline per `(stream, entity, field)` tuple.
A value known-normal in one contributing stream should not register as new
in another.
A value known-normal in one contributing stream does not register as new
in another. Implemented as an additive pooled scope in `baseline.py`
(`shared_entity_pools()` in `profile_suggestions.py` computes which
streams share an entity concept; pooled rows live under a synthetic
`shared_entity:<group>` namespace in the *same* tables, so no schema
migration was needed) covering entity presence/age and known
relationship-target values specifically — the mechanism most directly
responsible for the audited flood. Per-stream distribution baselines
(event-rate/detector bursts, numeric baselines) stay siloed per stream, on
purpose: those distributions genuinely differ by log source. Pooling
arbitrary categorical `rare_value` newness is a deferred follow-up.
Building this also surfaced and fixed a latent, previously-harmless bug in
`ALIAS_CANONICAL_BY_FIELD`'s construction where a later, generic canonical
group (`context`) could silently steal an alias (`hostname`, `dns_query`)
from an earlier, more specific one.
- **Cluster before score.** Field deviations get grouped by
`(detector, field, value-pattern)` signature before severity is assigned.
A pattern recurring identically across many entities becomes one clustered
triage entry with an occurrence count and entity breakdown, and — this is a
deliberate inversion of today's behavior — a large, identical cluster is a
signal to score *lower*, since it more often indicates a detector or
profile problem than a security event, while a small, isolated occurrence
keeps its current weight.
`(detector, field)` signature before severity is assigned (value-pattern
was dropped from the originally-planned signature — detector+field alone
already matched the real incident's cross-stream pattern). A pattern
recurring identically across many entities becomes one clustered triage
entry with an occurrence count (`cluster_size`), and — a deliberate
inversion of the old behavior — a cluster of 5 or more entities is capped
at score 25, since it more often indicates a detector or profile problem
than a security event, while a small, isolated occurrence keeps its full
weight.
The feedback/suppression layer (`feedback.py`) is not being restructured — it
is sound and stays as-is. It benefits automatically from cleaner input: fewer