diff --git a/README.md b/README.md index f1f3c32..78ecf87 100644 --- a/README.md +++ b/README.md @@ -68,6 +68,9 @@ It is organized around the operational workflow: readiness, data quality, and normalized top fields. 11. Mark findings as `Expected`, `False positive`, or `Confirmed` so repeated known behavior is labeled and lower priority in later refreshes. +12. Watch `Baseline DB size` in `Overview`. If it keeps growing quickly, lower + baseline retention or max values in `Settings`, then run baseline + maintenance during a planned stop. The UI also includes a `How To` tab with the same operational checklist. Use it when adding new streams or when the dashboard has data but it is unclear what @@ -97,6 +100,8 @@ The top status badges are health indicators: `N` source entities. Profile readiness is checked separately per stream field. - `Ollama: cached` or `ok` means local LLM output is available. It is supporting evidence, not the source of truth. +- `Baseline DB size` is the local SQLite baseline on disk. Retention deletes old + rows, but SQLite only returns disk space after a manual `VACUUM`. Use `Diagnostics -> Stream Coverage` to decide whether the monitor is healthy: @@ -113,6 +118,29 @@ Use `Diagnostics -> Stream Coverage` to decide whether the monitor is healthy: - `sample capped`: aggregate mode counted the full window, but raw events were intentionally capped to keep context queries manageable. +Baseline storage settings are in `Settings`: + +- `Baseline training days`: how old a field baseline must be before deviations + are promoted into triage. +- `Baseline bucket retention days`: how long five-minute rate/count buckets are + kept. +- `Baseline value retention days`: how long stale one-off categorical values are + kept. +- `Max values per entity field`: cap for distinct values per + stream/entity/field. Lower this if `profile_values` grows too fast. + +For a large existing baseline, stop the service before compacting: + +```bash +./start.sh stop +signalscope baseline-maintenance --baseline-db state/fgai-baseline.sqlite3 --retention-days 7 --value-retention-days 3 --max-values-per-field 500 --vacuum +./start.sh start +``` + +`--vacuum` can take time and needs free disk space close to the current DB size. +Run maintenance without `--vacuum` first if you only want to inspect row counts, +deleted rows, and reclaimable bytes. + `Ready Fields` is shown as `ready/tracked`, for example `0/8`. The stream profile tracks 8 fields, but none of those fields are mature yet. A field needs at least 12 five-minute buckets and the configured `Baseline training days` before it is diff --git a/src/fgai/dashboard.py b/src/fgai/dashboard.py index ffa0a8d..84e9e3f 100644 --- a/src/fgai/dashboard.py +++ b/src/fgai/dashboard.py @@ -185,7 +185,7 @@ HTML = """

Triage Queue

Field Baseline Deviations

Related Activity Across Sources

Block Candidates

Threat Intelligence

Policy Findings

Diagnostics

Recommended Stream Profiles

Waiting for observed stream data.

Installed Ollama Models

Loading local Ollama models.

Runtime Configuration

-

How To Use SignalScope

1. Normal workflow

  1. Settings: connect Graylog MCP and enable streams.
  2. Settings: apply recommended profiles for missing streams.
  3. Diagnostics: confirm raw samples, aggregate counts, and profile readiness.
  4. Overview: use Operator Guidance, incidents, trends, and correlation map.
  5. Findings: review only high-signal deviations first, then mark decisions.

2. Stream health

  • ready: profile exists, events are arriving, and baseline fields are ready.
  • learning: profile exists and events arrive, but baseline age or buckets are still too low.
  • missing_profile: stream is enabled but no profile exists. Apply or edit one.
  • no_events: stream is enabled but the current poll has no raw sample events.
  • partial_fetch: Graylog returned only part of the requested raw sample.

3. Ready fields

0/8 means 8 profile fields are tracked but none are mature enough yet. A field needs at least 12 baseline buckets and the configured Baseline training days before it is ready.

During learning, treat findings as signals to tune profiles, not as final alerts.

4. Profiles

  • Entity fields define who or what behavior is tracked, such as user, host, source IP, or application actor.
  • Baseline fields define the changing behavior to learn, such as action, event ID, service, URL, status, or counters.
  • Relationships learn pairs such as username to srcip or host to process.
  • Detectors add burst checks for auth failures, DNS queries, and deny actions.

5. Findings

  • Start with Triage Queue and incidents, not raw long tables.
  • Open evidence details before confirming a finding.
  • Use Expected for known behavior, False positive for bad signal, Confirmed for real investigation items.
  • Use expiry when a behavior is expected only temporarily.

6. High EPS / MCP

  • Use aggregate or auto fetch mode for high EPS streams.
  • Keep raw samples small enough for context; aggregate counts represent the full window.
  • Sample capped is normal in aggregate mode. Truncated raw mode means you may miss context.
  • If MCP is stale, the UI shows cached status so you can still inspect previous findings.

7. Ollama

  • Dashboard assessment summarizes current evidence.
  • Profile advisor maps unknown/custom fields and suggests relationships.
  • Ollama advice is constrained to fields discovered from Graylog; unknown fields are rejected.

8. What to fix first

  1. No streams enabled.
  2. Enabled streams with missing profiles.
  3. Enabled streams with zero raw events.
  4. Profiles stuck at 0 ready fields after the training window.
  5. Too many repeated findings without review feedback.
+

How To Use SignalScope

1. Normal workflow

  1. Settings: connect Graylog MCP and enable streams.
  2. Settings: apply recommended profiles for missing streams.
  3. Diagnostics: confirm raw samples, aggregate counts, and profile readiness.
  4. Overview: use Operator Guidance, incidents, trends, and correlation map.
  5. Findings: review only high-signal deviations first, then mark decisions.

2. Stream health

  • ready: profile exists, events are arriving, and baseline fields are ready.
  • learning: profile exists and events arrive, but baseline age or buckets are still too low.
  • missing_profile: stream is enabled but no profile exists. Apply or edit one.
  • no_events: stream is enabled but the current poll has no raw sample events.
  • partial_fetch: Graylog returned only part of the requested raw sample.

3. Ready fields

0/8 means 8 profile fields are tracked but none are mature enough yet. A field needs at least 12 baseline buckets and the configured Baseline training days before it is ready.

During learning, treat findings as signals to tune profiles, not as final alerts.

4. Profiles

  • Entity fields define who or what behavior is tracked, such as user, host, source IP, or application actor.
  • Baseline fields define the changing behavior to learn, such as action, event ID, service, URL, status, or counters.
  • Relationships learn pairs such as username to srcip or host to process.
  • Detectors add burst checks for auth failures, DNS queries, and deny actions.

5. Findings

  • Start with Triage Queue and incidents, not raw long tables.
  • Open evidence details before confirming a finding.
  • Use Expected for known behavior, False positive for bad signal, Confirmed for real investigation items.
  • Use expiry when a behavior is expected only temporarily.

6. High EPS / MCP

  • Use aggregate or auto fetch mode for high EPS streams.
  • Keep raw samples small enough for context; aggregate counts represent the full window.
  • Sample capped is normal in aggregate mode. Truncated raw mode means you may miss context.
  • If MCP is stale, the UI shows cached status so you can still inspect previous findings.

7. Ollama

  • Dashboard assessment summarizes current evidence.
  • Profile advisor maps unknown/custom fields and suggests relationships.
  • Ollama advice is constrained to fields discovered from Graylog; unknown fields are rejected.

8. Baseline DB size

  • Baseline DB size is the local SQLite behavior baseline.
  • Retention deletes old rows, but SQLite only returns disk space after manual VACUUM.
  • If the DB grows fast, lower bucket retention, value retention, or max values per entity field in Settings.
  • Stop the service before running signalscope baseline-maintenance --vacuum.

9. What to fix first

  1. No streams enabled.
  2. Enabled streams with missing profiles.
  3. Enabled streams with zero raw events.
  4. Profiles stuck at 0 ready fields after the training window.
  5. Too many repeated findings without review feedback.
  6. Baseline DB growing without maintenance.