ny roadmap och arch

ROADMAP.md

@@ -19,6 +19,57 @@ operational value and dependency, not by UI appeal.
 - [x] Prometheus-compatible low-cardinality metrics endpoint.
 - [x] Local dashboard for status, findings, incidents, diagnostics, and configuration.
 
+## 0. Structural Rework: Shape-Validated Discovery, Shared Baselines, Clustered Findings
+
+Goal: fix root causes found by auditing a real 25-stream / ~1M-event/day
+production deployment, instead of continuing to patch symptoms in the UI. See
+[`ARCHITECTURE.md`](ARCHITECTURE.md) for the full data-flow diagram and the
+detailed rationale behind each item below. This section blocks/supersedes
+parts of sections 1 and 6 below (baseline confidence tooling and compact
+incident cards both depend on findings being clustered correctly first) — do
+this section before spending more effort on either.
+
+- [ ] Validate name-matched fields against observed value shape too, not just
+  fields that had no name match. A field that matches a trusted name (e.g.
+  `context`) but is actually free-text (a full raw log line, confirmed in 12
+  of 25 audited stream profiles) must not be trusted just because its name
+  matched a known-useful alias.
+- [ ] Make relationship-candidate selection shape-aware: never propose a
+  free-text-shaped field as a relationship target, regardless of how it
+  entered `categorical_fields`. This alone was responsible for 17 of 25
+  entries in one real triage queue snapshot (91 individual `new_relationship`
+  deviations from a single misconfigured `*->context` pairing, repeated
+  across 5 streams).
+- [ ] Re-validate existing stored profiles' `relationship_fields` and
+  `categorical_fields` against current observed shapes on an ongoing basis,
+  not just at profile-creation time — flag (never silently rewrite) fields
+  that no longer look fit for their assigned role, and surface a one-click
+  "remove this field from relationships" action from the flag.
+- [ ] Pool baseline state by shared entity concept across streams that
+  shared-field discovery already recognizes as the same real-world entity
+  (for example a source IP seen in both a firewall stream and an IIS
+  stream), instead of a cold, isolated baseline per stream per field. A value
+  already known-normal in one contributing stream must not register as "new"
+  the first time a different, related stream happens to see it.
+- [ ] Cluster field deviations by `(detector, field, value-pattern)`
+  signature *before* scoring, not after. A pattern recurring identically
+  across many entities must surface as one clustered item with an occurrence
+  count and a "how many entities" breakdown, not N independently-scored
+  "critical" rows that all have to be discovered, selected, and bulk-cleared
+  by hand after the fact.
+- [ ] Recalibrate severity so cluster size is itself a signal: a pattern
+  hitting many unrelated entities identically is usually a
+  configuration/detector problem, not N separate critical incidents, and
+  should score at or below an isolated occurrence, never above it.
+- [ ] Update README usage documentation alongside each item above as it
+  ships — a behavior change is not done until it is documented.
+
+Acceptance: replaying the dataset that produced the original 25-entry,
+all-"critical" triage queue (17 of them the same misconfigured relationship)
+produces a small number of distinct, correctly-ranked items, with the
+systemic pattern visible as one clustered entry showing "91 occurrences
+across 5 streams," not 17 separate critical rows.
+
 ## 1. Detection Quality
 
 Goal: make findings more accurate before adding more integrations.
@@ -124,6 +175,10 @@ Acceptance: adding a new Graylog stream normally requires a profile template and
 
 ## Suggested Next Milestone
 
-Complete Detection Quality first: confidence scoring, burst/rare-value detectors,
-per-stream thresholds, and historical replay. These features determine whether
-the incident and dashboard work remains trustworthy as more log sources are added.
+Complete section 0 (Structural Rework) first. It was added after auditing a
+real production deployment and finding that the existing Detection Quality
+and Dashboard Usability work, while individually correct, sits on top of a
+discovery/scoring pipeline that lets one misconfigured field flood the triage
+queue with dozens of look-alike "critical" entries. Fixing the pipeline is
+higher leverage than any further UI work until it's done — see
+[`ARCHITECTURE.md`](ARCHITECTURE.md).