fgAI/README.md

# SignalScope

SignalScope is a local multi-source security analytics agent. Its primary mode connects to Graylog through MCP, discovers the streams and fields already available in your environment, and uses stream profiles to normalize events, build baselines, correlate entities, and explain anomalies with a local LLM.

Its running only locally and if using LLM it's running also locally so no data is sent or exposed outside.

FortiGate is one supported example. The same workflow applies to DNS/AdGuard, Windows Event Logs, Sysmon, Nginx, Squid, VPN, Proxmox, Filebeat-collected logs, and future Graylog streams.

The Python module and legacy `fgai` command remain available for compatibility. New installations can use `signalscope`.

The prioritized implementation plan is tracked in [ROADMAP.md](ROADMAP.md). For
how data actually flows through the system and the structural rework
currently in progress, see [ARCHITECTURE.md](ARCHITECTURE.md).

Autoblocking is dry-run by default. The tool will not block RFC1918, loopback, multicast, link-local, reserved, or allowlisted addresses unless you change the code.

## Screenshots

### Settings
![Settings](images/settings.jpg)

### Findings
![Findings](images/findings.jpg)

### Correlation
![Correlation](images/correlation.jpg)

## Quick Start

```bash
python -m venv .venv
source .venv/bin/activate
pip install -e .
```

Or use the helper script, which creates/uses `.venv` automatically and runs `pip install -e .`:

```bash
./start.sh
./start.sh status
./start.sh analyze
./start.sh stop
```

`./start.sh` starts three local background processes:

- Optional UDP syslog listener writing `logs/fg_syslog.jsonl`
- Continuous monitor writing `state/fgai-status.json`
- Local dashboard at `http://127.0.0.1:8088`

## Dashboard How To

The dashboard is the normal way to run SignalScope once the service is started.
It is organized around the operational workflow:

1. Open `Settings`.
2. Select `Graylog MCP`.
3. Enter the Graylog MCP URL and token, then save.
4. Click `Load streams`.
5. Enable the streams you want SignalScope to monitor, then save again.
6. Apply missing recommended profiles, or click `Edit profile` on a stream to
   choose its entity, time, baseline, detector, and weight fields manually.
7. Let the baseline learn for the configured `Baseline training days` before
   treating every deviation as actionable.
8. Use `Overview` for incidents, trends, stream health, AI assessment, and the
   correlation map.
9. Use `Findings` for the triage queue, field baseline deviations, related
   activity across sources, threat intelligence, and policy findings.
10. Use `Diagnostics` to confirm stream coverage, MCP fetch health, profile
    readiness, data quality, and normalized top fields.
11. Mark findings as `Expected`, `False positive`, or `Confirmed` so repeated
    known behavior is labeled and lower priority in later refreshes.
12. Watch `Baseline DB size` in `Overview`. If it keeps growing quickly, lower
    baseline retention or max values in `Settings`, then run baseline
    maintenance during a planned stop.

The UI also includes a `How To` tab with the same operational checklist. Use it
when adding new streams or when the dashboard has data but it is unclear what
needs attention next.

## Operating The Dashboard

Start in `Overview`, not in the long diagnostics tables. The intended daily flow
is:

1. Read `Operator Guidance` for the next action SignalScope thinks is most useful.
2. Check `Investigation Incidents` for grouped entity-level problems.
3. Use the correlation graph to see whether the same entity appears across
   several streams.
4. Open `Findings` only after you have an entity or incident to inspect.
5. Expand evidence rows and use the Graylog query link to inspect the raw events.
6. Mark the finding as `Expected`, `False positive`, or `Confirmed`.

The top status badges are health indicators:

- `Graylog MCP: connected` means the last completed MCP query worked.
- `Graylog MCP: refreshing` means a poll is in progress and the UI is showing
  previous counters until the poll completes.
- `Graylog MCP: error` means the UI may be showing cached data. Check the MCP
  URL, token, DNS/TLS, and stream permissions.
- `Baseline: N sources ready` means historical source-IP baselines exist for
  `N` source entities. Profile readiness is checked separately per stream field.
- `Ollama: cached` or `ok` means local LLM output is available. It is supporting
  evidence, not the source of truth.
- `Baseline DB size` is the local SQLite baseline on disk. Retention deletes old
  rows, but SQLite only returns disk space after a manual `VACUUM`.

Use `Diagnostics -> Stream Coverage` to decide whether the monitor is healthy:

- `ready`: stream is enabled, profile exists, events are arriving, and profile
  fields have enough baseline history.
- `learning`: stream is enabled and profiled, but baseline age or bucket count is
  still too low.
- `missing_profile`: stream is enabled but no profile exists. Apply a
  recommended profile or edit one manually.
- `no_events`: stream is enabled but no raw sample was returned in the last poll.
  Check stream activity, query, range, or permissions.
- `partial_fetch`: Graylog returned only part of the raw sample. Use aggregate or
  auto mode with a smaller raw sample on high-EPS streams.
- `sample capped`: aggregate mode counted the full window, but raw events were
  intentionally capped to keep context queries manageable.

Baseline storage settings are in `Settings`:

- `Baseline training days`: how old a field baseline must be before deviations
  are promoted into triage.
- `Baseline bucket retention days`: how long five-minute rate/count buckets are
  kept.
- `Baseline value retention days`: how long stale one-off categorical values are
  kept.
- `Max values per entity field`: cap for distinct values per
  stream/entity/field. Lower this if `profile_values` grows too fast.

Retention and per-field value capping already run automatically every poll
cycle — nothing above needs a manual trigger. `Diagnostics -> Baseline
Maintenance` shows the result: current DB size, reclaimable bytes, rows
deleted last cycle, and specifically which `(stream, entity, field)` rows
got trimmed for exceeding the max-values-per-field cap, so you can tell
whether a field is worth excluding from its profile instead of just raising
the cap. Only disk-space reclamation (`VACUUM`) needs a manual, offline step:

For a large existing baseline, stop the service before compacting:

```bash
./start.sh stop
signalscope baseline-maintenance --baseline-db state/fgai-baseline.sqlite3 --retention-days 7 --value-retention-days 3 --max-values-per-field 500 --vacuum
./start.sh start
```

`--vacuum` can take time and needs free disk space close to the current DB size.
Run maintenance without `--vacuum` first if you only want to inspect row counts,
deleted rows, and reclaimable bytes.

`Ready Fields` is shown as `ready/tracked`, for example `0/8`. The stream profile
tracks 8 fields, but none of those fields are mature yet. A field needs at least
12 five-minute buckets and the configured `Baseline training days` before it is
ready. With the default seven-day training window, new profiles can show `0/X`
for days even while data is being learned.

Interpret findings by maturity:

- During `learning`, findings are mostly profile-tuning signals.
- When fields are `ready`, high-score deviations are more meaningful.
- `new_relationship` means a new field pair appeared, such as
  `username -> srcip` or `host -> process.name`.
- `rare_value` means a new value appeared for a profiled entity and field.
- `event_rate_burst`, `auth_failure_burst`, `dns_query_burst`, and
  `deny_action_burst` compare the current window to historical buckets.

Field deviations that share the same detector and field are clustered before
scoring, not after. If the same `(detector, field)` signature appears across
five or more entities in one poll (`false_positive`/`expected`-suppressed
deviations don't count toward that total), every deviation in the cluster is
tagged with a `cluster_size` and its score is capped at 25, and the Evidence
column shows `[part of a N-entity pattern]` ahead of the reason. This is a
deliberate inversion of the usual "more occurrences = more severe" rule: a
pattern hitting many unrelated entities identically almost always means a
detector or profile problem (a field that shouldn't have been picked as a
relationship target, for example) rather than N separate critical security
events, so it is scored at or below an isolated occurrence instead of above
it. A small, isolated deviation keeps its full score. See
[`ARCHITECTURE.md`](ARCHITECTURE.md) for the incident that motivated this.

The review buttons are part of the detection loop:

- `Expected`: known behavior that should stay visible but not keep creating
  noise for the same scope.
- `False positive`: weak or bad signal for this scoped pattern.
- `Confirmed`: real investigation item.
- Expiry should be used for temporary expected changes, such as maintenance or a
  migration window.

Every review click also permanently records which detector produced the
finding and the verdict, independent of the suppression rule it creates —
so revoking or letting a rule expire later never erases that history. Check
`Diagnostics -> Detector Health` to see, per detector: whether it has
learned enough baseline history to be trusted yet, its confirmed-vs-false-positive
precision over the last 30 days, and a one-line description of what
actually triggers it. A detector with no reviews yet shows `n/a` for
precision rather than a misleading 0% or 100% — the number only becomes
meaningful once there's real history behind it.

## Primary Workflow: Graylog MCP

Graylog 7.1 MCP is the primary log-source integration. In the dashboard, open
`Settings`, select `Graylog MCP`, provide the MCP URL and a read-only API token,
then load and enable the streams to analyze. SignalScope uses MCP `list_streams`,
`list_fields`, `search_messages`, and `aggregate_messages` to work with existing
log sources rather than requiring every source to be forwarded locally.

The token field accepts a raw Graylog API token, the Base64 value after `Basic `,
or a complete `Basic <value>` header. Tokens are stored only in the local runtime
configuration and are never returned by the dashboard API.

Use `Edit profile` on a stream to load its fields. The field table shows Graylog
datatype/capability metadata and lets you select one or more entity fields, a
time field, and categorical/numeric fields for the stream profile. Profiles are
stored under `graylog_stream_profiles` in `state/fgai-config.json`.

The Settings page also shows recommended stream profiles built from observed
field coverage and cardinality. These recommendations use deterministic
discovery first, then can optionally be refined by a local Ollama profile advisor
model such as `qwen3:8b` (default) or a smaller model on constrained
hardware. Enable `Ollama profile advisor` and set
`Profile advisor model` in Settings. Advisor output must be valid JSON and is
validated against fields actually seen in the stream before it can be applied.
Unknown fields, raw message fields, internal `fgai_*` fields, and unknown
detectors are rejected. The advisor is only called for fields the deterministic
pass is genuinely unsure about (see below), not the whole profile every poll, and
is throttled by `Profile advisor interval seconds` so it does not run more often
than necessary on a memory-constrained local GPU.

Field discovery does not rely only on field names. Every observed field is also
classified from its sample values into a shape (IP address, timestamp, port,
numeric, low-cardinality categorical, or free text) with a 0-100 confidence
score. A field name match is still used, but only as a small, capped nudge on
top of that shape evidence, so a vendor-specific field name SignalScope has
never seen before can still be recognized as an entity, categorical, or numeric
field from its values alone. Two IP-shaped fields with ambiguous names are
told apart by which one leans more private (source-like) versus more public
(destination-like). Detectors can also be suggested from observed value
patterns, for example a low-cardinality field containing values such as
`denied` or `blocked` suggesting the deny-action detector, independent of
what the field or stream happens to be named. `Field confidence threshold` in
Settings controls both how confident a field must be before it counts as
unambiguous (for advisor scoping) and, when auto-apply is enabled, before its
stream profile is applied automatically. Each recommended profile in the
dashboard status payload carries the underlying per-field shape and confidence
values, so a `field_confidence`/`field_shapes` map is available for any field
even when it was not selected into the applied profile.

A name match never overrides value shape, even for names on the trusted
priority lists such as `context`. If the observed values for a name-matched
field confidently look like free text (a raw log line or URL, not a
low-cardinality category) or like a timestamp, it is excluded from entity,
categorical, and relationship-target selection regardless of its name, both
for a fresh recommendation and as a second, independent check when a field
reaches relationship-candidate selection through cross-stream field sharing.
The timestamp case matters for the same reason free text does: a timestamp
is near-unique on essentially every event, so tracking it as a categorical
or relationship field is guaranteed to produce a "never seen before" finding
every single time, exactly the failure mode that made `context` a problem.
A field with too few samples to classify is never excluded on that basis
alone; the veto only fires on a confident, contradicting shape.

Windows-like streams are recognized from stream names such as `Windows`,
`Winlog`, `Security Event Log`, `Sysmon`, and `Powershell`, or from Windows event
fields. The MCP search asks for common Winlogbeat/ECS names such as
`winlog.event_id`, `event.code`, `user.name`, `host.name`, `source.ip`,
`winlog.channel`, and `process.name` before a profile exists. Their default
recommendation favors normalized `username`, `hostname`, and `srcip` as
entities, then uses the observed Windows fields such as event ID,
action/outcome, channel/provider, `logon_type`, process/service fields, and
event category/type as categorical baseline fields. The authentication-failure
detector is enabled by default. You can still edit the applied profile per
stream when your Windows parser uses different field names or when a stream
contains a narrower log type.

Settings also lists locally installed Ollama models from `http://127.0.0.1:11434/api/tags`.
Click a model name to fill both the dashboard analyst model and profile advisor
model fields.

The settings page treats stream enablement and profile editing separately. The
checkboxes decide which streams are monitored. Click `Edit profile` on one stream
to load its fields and edit only that stream's profile; saving with no active
profile editor leaves existing profiles unchanged.

When many Graylog streams are available, use `Diagnostics -> Stream Coverage` to
see which streams are enabled, which have profiles, how many profile fields are
baseline-ready, how many events were fetched, and whether a stream is `ready`,
`learning`, `missing_profile`, `no_events`, or `not_enabled`.

Recommended stream profiles are an onboarding helper, not a fixed FortiGate
parser. SignalScope inspects the fields observed from each Graylog stream and
looks for common denominator fields such as entities, timestamps, actions,
severities, categories, ports, DNS names, URLs, process fields, Windows event
IDs, and numeric counters. Fields that appear across multiple enabled streams
are preferred when they are useful for correlation or baselining. The local
Ollama profile advisor can refine those recommendations, but the deterministic
profile discovery remains the fallback when Ollama is disabled, missing, slow, or
returns invalid JSON.

Cross-stream discovery is semantic, not just exact-name matching. Source IP
fields such as `srcip`, `source.ip`, `source_ip`, and `client_ip` are grouped as
the same shared entity field. The same approach is used for destination IPs,
ports, timestamps, actions, severities, usernames, hosts, event IDs, DNS names,
services, URLs, and context/message fields. Recommended profiles show both the
selected per-stream fields and the shared alias groups so you can see why a field
is useful for correlation even when different products use different schemas.
If the Ollama advisor returns no usable profile for a stream, the row stays on
the deterministic profile and is labeled as a heuristic fallback instead of
pretending the whole recommendation failed.

When two or more enabled streams recognize the same real-world entity
concept through this alias grouping (for example `srcip` used as the entity
field in both a firewall stream and an IIS stream), SignalScope
automatically pools part of their baseline state instead of learning it
independently per stream: an entity's presence/age history and its known
relationship-target values are shared across the pool, so a value already
established as normal in one contributing stream does not get reported as
"new" the first time a different, related stream happens to observe it.
This is automatic — there is nothing to configure — and it only ever
*reduces* false "already new" flags for values genuinely seen elsewhere in
the pool; a value that is actually new everywhere is still reported exactly
as before. Per-stream statistical baselines (event-rate bursts, detector
bursts, numeric-field distributions) are not pooled and stay independent per
stream, since those distributions legitimately differ by log source.
Active pools are listed under `Diagnostics -> Shared Entity Baselines`,
showing which entity concept is shared and which streams contribute to it.

Recommended profiles can be re-applied to existing profiles. Enable `show
existing profiles` and click `Update profile` to append newly discovered entity,
categorical, numeric, and detector fields. Existing profile names, field weights,
and detector threshold settings are preserved, so this is the fast path after
field-alias matching improves or after Graylog starts parsing additional fields.

Enable `Auto-apply high-confidence stream profiles` in Settings to skip the
`Apply`/`Update profile` click for streams where every selected field already
clears `Field confidence threshold`. A profile is only ever auto-applied when it
has no manually-set field: if even one field in an existing profile was set by
hand, that whole stream is left alone by auto-apply and stays on the normal
review flow. Fields written by auto-apply are marked internally so a later
manual edit through the Profile editor is never silently overwritten by a
future auto-apply cycle. If an auto-applied field's classification drifts
later, for example because a vendor changed their log format, SignalScope does
not silently re-apply a new mapping; it only records the drift in the status
payload (`profile_drift`) for now, with no dashboard panel yet, so the live
profile stays exactly as it was applied until someone reviews it. The auto-apply
and advisor-scoping thresholds share the same `Field confidence threshold`
setting, and drift-history retention is controlled separately by `Field
confidence history retention days`.

Profile discovery is accumulated over monitor cycles. This matters in high-EPS
environments where each poll only fetches a raw sample for context while
aggregate queries count the full window. Fields seen in earlier samples are kept
in the local history database and continue to participate in recommended
profiles and shared-field matching even if the current raw sample does not
contain them. This lets late-arriving or less frequent fields such as custom
`lcs_*` application fields stay visible long enough to be reviewed and appended
to an existing profile.

Shared-field discovery is also used as the base for cross-source correlation.
SignalScope groups exact aliases and broader semantic families such as source IP,
user, host, ID, status/result, action, type/category, domain, URL, and custom
namespaces such as `lcs_*`. These shared groups are the foundation for a global
correlation profile and future flow graphs that show how users, hosts, IPs,
applications, IDs, statuses, and destinations relate across streams.

Enabled streams are normalized through the same event model. Stream profiles
define the entity, timestamp, categorical, and numeric fields used for baselines.
The dashboard and Ollama then correlate behavior across sources, for example a
client IP appearing in FortiGate, AdGuard/DNS, Windows Security, Nginx, Squid,
VPN, or Proxmox.

A stream profile can track multiple entities from the same event, such as
`username`, `srcip`, and `hostname`. SignalScope stores profile baselines for
each selected entity value, which makes cross-source investigation work even when
one source is user-centric and another is IP- or host-centric.

Profiles can also track field relationships as behavior patterns. This is useful
when the suspicious signal is not a single new value, but a new combination such
as a known user logging in successfully from a source IP that has never been seen
for that user before. Add `relationship_fields` to a stream profile, for example:

```json
[
  {
    "stream_id": "windows-security",
    "entity_field": "username",
    "categorical_fields": ["action", "eventid"],
    "relationship_fields": [
      {"left": "username", "right": "srcip", "name": "user source IP"},
      {"left": "username", "right": "hostname", "name": "user host"}
    ]
  }
]
```

After the baseline has learned those relationships, a new `username -> srcip` or
`username -> hostname` pair is reported as `new_relationship` with sample events.
The dashboard profile editor exposes this as `Behavior relationships (JSON)`.

An existing stored profile's `categorical_fields` and `relationship_fields` are
re-checked against current observed shapes on every poll, not just when the
profile was first created — a field can look fine at profile-creation time and
drift into free text later, or the classifier's read of it can simply have been
wrong initially. A mismatch never rewrites the stored profile automatically;
it only shows up under `Recommended Stream Profiles` as a warning naming the
field, the stream, and the confidence behind the mismatch, with a button that
opens that stream's profile editor so a person decides whether to remove it.

SignalScope keeps a common alias map for fields such as source IP, destination
IP, ports, action, severity, service/protocol, DNS query, URL, message, and event
type. This lets Related Activity and correlations work with firewall/proxy/DNS
streams that use names like `src_addr`, `destination.ip`, `dest_port`,
`fw_action`, `priority`, `proto`, or `full_message` without adding a new parser
for every product.

Correlation is entity-aware rather than FortiGate-specific. SignalScope recognizes
common IP fields such as `srcip`, `source_ip`, `remote_addr`, and Windows event
IP fields; account fields such as `username`, `user`, and `TargetUserName`; and
host fields such as `hostname`, `computer`, and `winlog_computer_name`. Configure
the exact entity field per stream in the profile when your Graylog schema differs.

Each profile baseline is stored per stream, entity, selected field, and five-minute
bucket. Once enough history exists, SignalScope compares the current rate or
numeric value to the same UTC weekday/hour where possible, then falls back to the
stream's overall history. Repeated MCP pages are fingerprinted so the same
Graylog event is not learned repeatedly. Related anomalies, profile deviations,
and multi-stream correlations are grouped into investigation incidents with a
compact evidence timeline.

Timeline and related-activity rows include copyable Graylog query details built
from normalized source, destination, action, and DNS fields. These are query
details rather than hard-coded web links, so they work with MCP and with Graylog
deployments behind different URLs or reverse proxies.

Incident lifecycle state is stored locally in `state/signalscope-incidents.json`.
Use the dashboard incident actions to acknowledge, resolve, or reopen an incident
and attach a note. The state is keyed to a stable incident fingerprint so it can
survive monitor refreshes even when the current detection window changes.

Field deviation review state is stored locally as feedback. Mark a deviation as
`Expected`, `False positive`, or `Confirmed` from the Findings page. The decision
is scoped to the stream, entity, field or detector pattern, optional value, note,
and expiry time. Expected and false-positive feedback does not erase the finding;
it keeps the row reviewable while reducing repeat noise for the same scoped
pattern and giving Ollama context that the behavior is already known.

Export the current investigation view when you need to share or archive an
incident outside the dashboard:

```bash
signalscope export-investigation --format markdown --output exports/investigation.md
signalscope export-investigation --incident-id <incident-id> --format json
```

The report is built from `state/fgai-status.json` by default and includes
summary counters, stream coverage, incident state, analyst notes, evidence,
timeline rows, and Graylog query details. The dashboard exposes the same data at
`/api/export/incidents?format=markdown` or `format=json`.

With a stream profile in place, SignalScope also builds independent burst
baselines for authentication failures, DNS queries, and deny/block actions when
those events are present. These are evaluated per configured entity, so a Windows
account, DNS client, or firewall source is compared to its own history.

SignalScope also detects ordered behavior sequences across any streams that share
an entity. The built-in sequence is category-based, not source-specific:
`dns_query -> network_connection -> auth_failure`. Those categories can come from
AdGuard, Windows DNS, a proxy, firewall, VPN, endpoint, or any other Graylog
stream as long as the fields normalize into the same generic event model.

Stream profiles can also carry `field_weights` to tune scoring without changing
the baseline itself. Weights are multipliers from `0` to `5` and can target a
field, a detector, or a field+detector pair:

```json
{
  "url": 1.5,
  "auth_failure_burst": 2,
  "query_domain": {
    "rare_value": 1.8
  }
}
```

Use replay or `replay-graylog --compare-config-file` to test score changes before
applying them to the live profile.

Replay a historic JSONL or Graylog export without changing the live baseline:

```bash
signalscope replay --logs exports/windows-history.jsonl --stream-id <configured-stream-id>
```

Replay uses a temporary SQLite baseline and evaluates events in timestamp order.
It reports detector counts and the findings that would have been generated. Use
the configured stream ID so the export is evaluated with that stream's profile.

Replay directly from Graylog MCP without touching the live baseline:

```bash
signalscope replay-graylog --range-seconds 86400
```

By default this uses the enabled streams from the dashboard configuration. Limit
the run to one or more streams with repeated `--stream-id` flags. To test a
candidate detector/profile configuration before applying it, compare it against
the current runtime config:

```bash
signalscope replay-graylog --range-seconds 86400 --compare-config-file exports/candidate-config.json
```

The comparison reports detector-count, field-finding, and source-anomaly deltas
using the same fetched event window.

The current MCP endpoint is `http://<graylog-host>:9000/api/mcp`. Enable it in
Graylog under `System -> Configurations -> MCP` and use stream IDs internally;
the fgAI stream picker resolves titles in the UI.

For live monitoring, `graylog_range_seconds` controls how far back each MCP poll
searches. The default is 300 seconds, so each poll re-checks the last five
minutes. `graylog_max_events_per_stream` caps raw events fetched from each
stream during that window. If a stream hits the cap, the dashboard marks the
window as truncated because high EPS means SignalScope sampled only part of the
Graylog result set. For very high-volume streams, prefer aggregate baselines and
targeted drill-down queries over trying to pull every raw event through MCP.
Large values such as 100000 can require hundreds of paged MCP searches across
enabled streams. If Graylog times out or rejects the query, SignalScope keeps the
events already fetched, marks the stream as a partial fetch, and shows the MCP
error in Diagnostics instead of failing the whole dashboard update.
The monitor also stores the last successful dashboard status in
`state/signalscope-status-cache.sqlite3`. If a later live MCP fetch fails before
usable data is available, the dashboard keeps showing the last good findings,
graphs, incidents, and correlations with a stale-data warning instead of going
blank.

Graylog fetch mode controls how high-volume streams are read:

- `raw`: fetch raw events up to `graylog_max_events_per_stream`.
- `aggregate`: use Graylog MCP `aggregate_messages` for total event volume, then
  fetch only `graylog_raw_sample_events` raw events per stream for findings and
  drill-down context.
- `auto`: use aggregate mode automatically when `graylog_max_events_per_stream`
  is larger than `graylog_raw_sample_events`.

For high EPS environments, keep `graylog_range_seconds` at 300, set
`graylog_fetch_mode` to `auto` or `aggregate`, and use a modest raw sample such
as 5000. The dashboard then shows aggregate event volume without forcing every
raw log line through MCP each poll. In aggregate mode the raw sample is capped at
10000 events per stream to stay within Graylog's default result-window limit;
aggregate counts are used for volume above that.

## Monitoring Export

The dashboard also exposes Prometheus text metrics at:

```text
http://127.0.0.1:8088/metrics
```

This endpoint is passive and has no Prometheus or Grafana dependency. It reports
low-cardinality event counts, anomaly severities, incident counts, baseline readiness, and Graylog
MCP health. Use it later as a Prometheus scrape target or as input for a Checkmk
local check. Do not use source IPs, domains, or raw event IDs as metric labels.

Enable cached Ollama analyst notes in the dashboard:

```bash
FGAI_LLM=1 OLLAMA_MODEL=llama3.1 ./start.sh restart
```

The monitor refreshes deterministic detections every `FGAI_MONITOR_INTERVAL` seconds and refreshes the LLM note every `FGAI_LLM_INTERVAL` seconds, default `300`.
The dashboard also exposes `Ollama assessment timeout seconds`; raise this when
Ollama is running but large multi-stream summaries still time out.

The assessment is mostly built from pre-computed statistical scores and reasons
(deviation scores, burst z-scores, relationship names), not raw log content —
deliberately compacted this way to keep the prompt small for a local, often
GPU-constrained model even when the underlying deployment has hundreds of
streams or millions of daily events. On top of that, the top-scoring field
deviations and incidents each carry a small, bounded number of real message
excerpts (already truncated short), so the note can call out content a
statistical score alone can't surface, such as a suspicious value in a URL or
command-line field. This is intentionally a qualitative addition layered on
top of the statistical findings, not a replacement for them, and it never
feeds back into scoring or suppression — it only changes the wording of the
narrative note.

The script activates `.venv` inside the script process. If you also want your current shell prompt to show the venv, run:

```bash
source .venv/bin/activate
```

For UDP `514`, the script starts only the listener command with `sudo`:

```bash
FGAI_SYSLOG_PORT=514 ./start.sh
```

The syslog receiver rotates the active JSONL input at 25 MB by default. Rotated
files are gzip-compressed and 14 archives are retained. Override this when needed:

```bash
FGAI_LOG_ROTATE_BYTES=$((100 * 1024 * 1024)) FGAI_LOG_ROTATE_COUNT=30 ./start.sh restart
```

The continuous monitor also stores a local SQLite behavior baseline at
`state/fgai-baseline.sqlite3`. A source becomes baseline-ready after 12 completed
five-minute windows. Historical rate and hitcount-rate deviations then contribute
to its anomaly score. Set `FGAI_BASELINE_DB` to use another location.

SignalScope prunes old baseline buckets during each monitor cycle. The defaults
keep 7 days of buckets and dedupe history, prune stale one-off categorical
values after 3 days, and cap high-cardinality values per stream/entity/field.
Very noisy fields such as raw messages, URLs, payloads, request/response bodies,
tokens, sessions, hashes and long values are still counted in bucket baselines
but are not stored as distinct rare-value candidates.
Tune these in the dashboard or in `state/fgai-config.json`:

```json
{
  "baseline_training_days": 7,
  "baseline_retention_days": 7,
  "baseline_value_retention_days": 3,
  "baseline_max_values_per_field": 500
}
```

`baseline_training_days` is the minimum baseline age before profile deviations
are promoted into live triage. For production data, set this to the amount of
history you trust, commonly `7` to `14` days. The dashboard shows fields as
`learning` until both bucket count and baseline age are sufficient.

If an existing baseline database has already grown large, stop the monitor and
run a manual prune plus SQLite compaction:

```bash
signalscope baseline-maintenance --baseline-db state/fgai-baseline.sqlite3 --retention-days 7 --value-retention-days 3 --max-values-per-field 500 --vacuum
```

`VACUUM` can take time on a large database, needs free disk space roughly equal
to the database size, and should not be run while the monitor is actively
writing. Without `--vacuum`, SQLite may delete rows but keep the file size.

Analyze local logs:

```bash
fgai analyze-logs --logs logs/fg_syslog.jsonl
```

Open the live UI after `./start.sh`:

```bash
xdg-open http://127.0.0.1:8088
```

Score likely traffic anomalies:

```bash
fgai detect-anomalies --logs logs/fg_syslog.jsonl --min-score 35
fgai detect-anomalies --logs logs/fg_syslog.jsonl --min-score 35 --llm --llm-timeout 300
```

Generate response and policy recommendations:

```bash
fgai recommend --logs logs/fg_syslog.jsonl --min-score 35
```

Optional external reputation enrichment is disabled by default. To use VirusTotal for public source/destination IP reputation:

```bash
export FGAI_THREAT_INTEL=1
export ABUSEIPDB_API_KEY='...'
fgai recommend --logs logs/fg_syslog.jsonl --min-score 35 --threat-intel
```

VirusTotal is also supported:

```bash
export FGAI_THREAT_INTEL=1
export FGAI_THREAT_INTEL_PROVIDER=virustotal
export VIRUSTOTAL_API_KEY='...'
fgai recommend --logs logs/fg_syslog.jsonl --min-score 35 --threat-intel
```

Threat intelligence can also be configured in the dashboard settings. Choose
`auto`, `abuseipdb`, or `virustotal`, paste the provider API key, and set the
daily lookup budget and cache TTLs. API keys are stored only in the local config
file and are not returned back to the browser after saving.

Threat intelligence responses are cached locally in `state/threat-intel-cache.json`. Successful results are reused for seven days by default, failures for one hour, and SignalScope permits at most 100 new provider lookups per UTC day. Cached responses are returned even after that budget is reached. Tune these safeguards in the dashboard or with `FGAI_THREAT_INTEL_TTL_SECONDS`, `FGAI_THREAT_INTEL_ERROR_TTL_SECONDS`, and `FGAI_THREAT_INTEL_DAILY_LIMIT`.

Listen for FortiGate syslog locally:

```bash
fgai listen-syslog --port 5514 --output logs/fg_syslog.jsonl
```

Run the listener quietly in the background:

```bash
./start.sh
```

Stop the background listener:

```bash
./start.sh stop
```

UDP port `514` normally needs root privileges on Linux:

```bash
sudo .venv/bin/fgai listen-syslog --port 514 --output logs/fg_syslog.jsonl
```

Test FortiGate API access:

```bash
export FORTIGATE_HOST=192.0.2.10
export FORTIGATE_API_TOKEN='...'
export FORTIGATE_VERIFY_TLS=false
fgai test-connection
fgai fetch-policies --output exports/policies.json
```

Audit a FortiGate policy export:

```bash
fgai audit-policies --config exports/fortigate.conf
```

Or fetch policies through the FortiGate API and audit that JSON:

```bash
fgai fetch-policies --output exports/policies.json
fgai audit-policies --config exports/policies.json --llm --llm-timeout 300
```

Find block candidates without changing the firewall:

```bash
fgai suggest-blocks --logs logs/fg_syslog.jsonl
```

Execute guarded quarantine actions:

```bash
export FORTIGATE_HOST=192.0.2.10
export FORTIGATE_API_TOKEN='...'
fgai suggest-blocks --logs logs/fg_syslog.jsonl --execute --expiry-minutes 60
```

Optional local LLM summary through Ollama:

```bash
ollama pull llama3.1
fgai analyze-logs --logs logs/fg_syslog.jsonl --llm --llm-timeout 300
```

For slower machines or a memory-constrained GPU, pick a smaller model instead
of just raising the timeout:

```bash
ollama pull qwen2.5:3b-instruct
OLLAMA_MODEL=qwen2.5:3b-instruct OLLAMA_TIMEOUT=300 fgai analyze-logs --logs logs/fg_syslog.jsonl --llm
```

**Sizing the model to your GPU**: the defaults (`llama3.1`, 8B, for the
dashboard assessment; `qwen3:8b` for the profile advisor) are a reasonable
general-purpose choice, but on a memory-constrained card (6 GB VRAM or less,
e.g. a mobile/low-power GPU) they can leave little headroom for context,
which shows up as slow or timed-out assessments rather than an obvious
out-of-memory error. If that happens, switch to a smaller model such as
`qwen2.5:3b-instruct` for both settings. On a workstation-class GPU with
16 GB+ VRAM, a larger model such as `qwen3:14b` for the dashboard assessment
will give noticeably better analysis quality. The profile advisor only ever
classifies a handful of ambiguous fields per call (see the Graylog MCP
section below), so it can stay small even on constrained hardware without
much quality loss.

**Whichever model you choose, `ollama pull` it first** — Settings just holds
a model name string, and the dashboard doesn't pull anything on your behalf.
Pointing `Ollama model`/`Profile advisor model` at a tag you haven't pulled
locally makes every assessment fail silently rather than an obvious error.
Check `ollama list` (or the dashboard's own "Installed Ollama Models" panel
in Settings) before changing either setting.

## Optional FortiGate Input

## Synthetic Windows Test Input

For testing a Graylog Beats input without a Windows host, generate Windows
Security-style JSONL events locally, then use Filebeat to ship them over TCP:

```bash
python scripts/generate_windows_events.py --interval 0.5
filebeat -e -c examples/filebeat-windows-synthetic.yml
```

Update the absolute JSONL path and Graylog host in the Filebeat template first.
Route `stream_hint: Windows` to a dedicated Graylog stream, then enable that
stream in SignalScope. The Settings page should recommend a Windows profile once
sample events have been fetched; apply it and adjust the entity/categorical
fields if your parser uses different names. Filebeat uses its Logstash output to communicate with Graylog's
Beats input on TCP `5044`. [Graylog Beats input documentation](https://go2docs.graylog.org/current/getting_in_log_data/beats_input.html)

For logs, configure FortiGate syslog to write into a local file such as `logs/fg_syslog.jsonl`. The parser supports common key/value syslog lines and JSONL.

For policies, export a FortiOS config backup and pass it to `audit-policies`.

Example FortiGate syslog target, run on the FortiGate CLI and replace the server IP with this machine:

```text
config log syslogd setting
    set status enable
    set server "192.0.2.50"
    set port 5514
    set mode udp
    set format default
end
```

## Environment

- `FORTIGATE_HOST`: firewall hostname or IP.
- `FORTIGATE_API_TOKEN`: REST API token.
- `FORTIGATE_VERIFY_TLS`: `true` or `false`, defaults to `true`.
- `FGAI_ALLOWLIST`: comma-separated IPs/CIDRs never to block.
- `OLLAMA_HOST`: defaults to `http://127.0.0.1:11434`.
- `OLLAMA_MODEL`: defaults to `llama3.1`. Size this to your GPU's VRAM — see the sizing note above.
- `OLLAMA_TIMEOUT`: Ollama request timeout in seconds, defaults to `180`.
- `FGAI_LLM`: set to `1` to enable dashboard Ollama analyst notes.
- `FGAI_LLM_INTERVAL`: seconds between dashboard LLM notes, defaults to `300`.
- `llm_timeout` in the dashboard config controls the dashboard assessment
  timeout after startup; it defaults to `180`.
- `FGAI_THREAT_INTEL`: set to `1` to enable external threat intelligence lookups.
- `ABUSEIPDB_API_KEY`: AbuseIPDB API key for public IP reputation enrichment.
- `ABUSEIPDB_MAX_AGE_DAYS`: report age window for AbuseIPDB, defaults to `90`.
- `FGAI_THREAT_INTEL_PROVIDER`: `auto`, `abuseipdb`, or `virustotal`.
- `VIRUSTOTAL_API_KEY`: VirusTotal API key for public IP reputation enrichment.

## Safety Model

The agent separates detection from enforcement:

- UTM events are scored from FortiGate logs (`ips`, `virus`, `anomaly`, `ddos`, `webfilter`, `app-ctrl`, `waf`, `dns`).
- Source IPs must be globally routable and outside the allowlist.
- Blocking requires `--execute`.
- The FortiGate API call is limited to the quarantine/banned user monitor endpoint.

## Production Deployment

`start.sh` is convenient for a single machine you're logged into, but has no
supervision (a crashed process just stays dead) and no boot-time start.

### Running under systemd

Unit files for the monitor, dashboard, and optional local syslog listener are
in `systemd/`, with full setup and update instructions in
[`systemd/README.md`](systemd/README.md). They mirror `start.sh`'s exact
process arguments, add `Restart=on-failure`, and run under a dedicated
`signalscope` system user with `ProtectSystem=strict`/`ProtectHome=true`
sandboxing rather than as root.

### Backup and Restore

Everything SignalScope needs to restore — runtime configuration, feedback
rules, behavior baselines, incident state, and caches — lives under `state/`.
Nothing else needs backing up; logs and exports are reproducible/disposable.

```bash
scripts/backup_state.sh                    # writes to ./backups/signalscope-state-<timestamp>.tar.gz
scripts/backup_state.sh /path/to/backups   # or a custom destination
```

SQLite databases (`fgai-baseline.sqlite3`, `signalscope-history.sqlite3`,
`signalscope-status-cache.sqlite3`) are backed up with `sqlite3 .backup`,
which takes a consistent snapshot even while the monitor is actively writing
to them, rather than a plain file copy that can capture a torn/inconsistent
write mid-transaction. JSON files (`fgai-config.json`,
`signalscope-feedback.json`, `signalscope-incidents.json`,
`threat-intel-cache.json`) are already written atomically by SignalScope
(temp file + rename) and are copied directly.

To restore: stop the monitor and dashboard, extract the archive's contents
over `state/`, then restart.

### Exposing the Dashboard Beyond Loopback

The dashboard has **no built-in authentication** — its entire access-control
model is "only accept connections from `127.0.0.1`" (`start.sh`'s and the
systemd unit's default `--host 127.0.0.1`). API routes that change state
(`POST /api/config`, `/api/feedback`, `/api/feedback/bulk`, incident actions,
and more) additionally check that the client's socket address is loopback,
regardless of what `Host` header a request carries.

If you need to reach the dashboard from another machine, do not change
`--host` to `0.0.0.0` directly — put an authenticating reverse proxy in front
of it instead and keep SignalScope itself bound to loopback. For example,
with Caddy:

```
signalscope.internal.example.com {
    basicauth {
        analyst $2a$14$...   # generate with: caddy hash-password
    }
    reverse_proxy 127.0.0.1:8088
}
```

Or nginx with `auth_basic` / an `auth_request` subrequest to whatever SSO the
rest of your infrastructure already uses. Either way, terminate TLS at the
proxy and keep the token/credential prompts outside SignalScope's own process
— since state-changing routes only trust the *proxy's* loopback connection,
not the original client, the proxy is the only thing standing between a
public request and dashboard write access once `--host` is opened up.