Pool baseline state across streams sharing an entity concept
Completes the last deferred item from ROADMAP §0: entities recognized as the same real-world concept across enabled streams (e.g. srcip in both a firewall and an IIS stream) now share baseline knowledge of entity presence/age and known relationship-target values, instead of each stream learning it cold. Implemented additively (synthetic shared_entity:<group> namespace in the existing baseline.py tables, no schema migration), scoped to entity presence + relationship known-values only — per-stream distribution baselines stay untouched. Also fixes a latent alias-map collision bug found while building this (ALIAS_CANONICAL_BY_FIELD let "context" silently steal "hostname"/"dns_query" from their own canonical groups). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -107,6 +107,12 @@ calls.
|
||||
|
||||
## Known Structural Gaps (found auditing a real deployment)
|
||||
|
||||
**Status: all four gaps below are now fixed and verified live** against the
|
||||
same deployment that produced them (see the Target Architecture section for
|
||||
what shipped, and `ROADMAP.md` section 0 for the checklist). This section is
|
||||
kept as the historical record of what the audit actually found, since the
|
||||
numbers below are what "verified" is measured against.
|
||||
|
||||
A live 25-stream deployment (~1M events/day) was audited directly while
|
||||
debugging why its triage queue felt unmanageable. Findings, in order of
|
||||
impact:
|
||||
@@ -159,10 +165,16 @@ lets it become a relationship target, gap 3 means there's no shared state to
|
||||
dampen the noise across related streams, and gap 4 means nothing catches the
|
||||
resulting flood before it reaches the human. Fixing gap 2 alone would have
|
||||
prevented the specific incident that triggered this audit; fixing all four is
|
||||
what makes the system hold up as more streams are added.
|
||||
what makes the system hold up as more streams are added. Verified live after
|
||||
all four shipped: the `context` relationship flood went from 91 deviations to
|
||||
0, total field deviations from 117 to 17, and triage severity finally
|
||||
differentiates (10 critical / 8 medium / 7 high, versus 25/25 critical
|
||||
before).
|
||||
|
||||
## Target Architecture
|
||||
|
||||
This is now the shipped behavior, not just the target.
|
||||
|
||||
- **Shape-first validation everywhere a field is selected for a role**,
|
||||
including name-matched fields, not only as a fallback for unmatched ones.
|
||||
Free-text-shaped fields are structurally excluded from relationship
|
||||
@@ -173,16 +185,31 @@ what makes the system hold up as more streams are added.
|
||||
- **Shared, entity-scoped baselines** for entities that shared-field
|
||||
discovery already recognizes as the same concept across multiple enabled
|
||||
streams, instead of one cold baseline per `(stream, entity, field)` tuple.
|
||||
A value known-normal in one contributing stream should not register as new
|
||||
in another.
|
||||
A value known-normal in one contributing stream does not register as new
|
||||
in another. Implemented as an additive pooled scope in `baseline.py`
|
||||
(`shared_entity_pools()` in `profile_suggestions.py` computes which
|
||||
streams share an entity concept; pooled rows live under a synthetic
|
||||
`shared_entity:<group>` namespace in the *same* tables, so no schema
|
||||
migration was needed) covering entity presence/age and known
|
||||
relationship-target values specifically — the mechanism most directly
|
||||
responsible for the audited flood. Per-stream distribution baselines
|
||||
(event-rate/detector bursts, numeric baselines) stay siloed per stream, on
|
||||
purpose: those distributions genuinely differ by log source. Pooling
|
||||
arbitrary categorical `rare_value` newness is a deferred follow-up.
|
||||
Building this also surfaced and fixed a latent, previously-harmless bug in
|
||||
`ALIAS_CANONICAL_BY_FIELD`'s construction where a later, generic canonical
|
||||
group (`context`) could silently steal an alias (`hostname`, `dns_query`)
|
||||
from an earlier, more specific one.
|
||||
- **Cluster before score.** Field deviations get grouped by
|
||||
`(detector, field, value-pattern)` signature before severity is assigned.
|
||||
A pattern recurring identically across many entities becomes one clustered
|
||||
triage entry with an occurrence count and entity breakdown, and — this is a
|
||||
deliberate inversion of today's behavior — a large, identical cluster is a
|
||||
signal to score *lower*, since it more often indicates a detector or
|
||||
profile problem than a security event, while a small, isolated occurrence
|
||||
keeps its current weight.
|
||||
`(detector, field)` signature before severity is assigned (value-pattern
|
||||
was dropped from the originally-planned signature — detector+field alone
|
||||
already matched the real incident's cross-stream pattern). A pattern
|
||||
recurring identically across many entities becomes one clustered triage
|
||||
entry with an occurrence count (`cluster_size`), and — a deliberate
|
||||
inversion of the old behavior — a cluster of 5 or more entities is capped
|
||||
at score 25, since it more often indicates a detector or profile problem
|
||||
than a security event, while a small, isolated occurrence keeps its full
|
||||
weight.
|
||||
|
||||
The feedback/suppression layer (`feedback.py`) is not being restructured — it
|
||||
is sound and stays as-is. It benefits automatically from cleaner input: fewer
|
||||
|
||||
Reference in New Issue
Block a user