fgAI/README.md

# Fortigate AI ML Inspection Agent

Local FortiGate log and policy inspection agent. It parses FortiGate syslog/JSONL logs, audits FortiOS policy exports, highlights UTM events, and can quarantine malicious source IPs through the FortiGate API when explicitly enabled.

Autoblocking is dry-run by default. The tool will not block RFC1918, loopback, multicast, link-local, reserved, or allowlisted addresses unless you change the code.

## Quick Start

```bash
python -m venv .venv
source .venv/bin/activate
pip install -e .
```

Or use the helper script, which creates/uses `.venv` automatically and runs `pip install -e .`:

```bash
./start.sh
./start.sh status
./start.sh analyze
./start.sh stop
```

`./start.sh` starts three local background processes:

- UDP syslog listener writing `logs/fg_syslog.jsonl`
- Continuous monitor writing `state/fgai-status.json`
- Local dashboard at `http://127.0.0.1:8088`

## Graylog MCP

Graylog 7.1 MCP can be used as the active log source instead of the local JSONL
listener. In the dashboard, open `Settings`, select `Graylog MCP`, provide the
MCP URL and a read-only API token, then load and enable the streams to analyze.

The token field accepts a raw Graylog API token, the Base64 value after `Basic `,
or a complete `Basic <value>` header. Tokens are stored only in the local runtime
configuration and are never returned by the dashboard API.

Use `Load selected stream fields` after choosing a stream. The field table shows
Graylog datatype/capability metadata and lets you select an entity field, a time
field, and categorical/numeric fields for the stream profile. Profiles are stored
under `graylog_stream_profiles` in `state/fgai-config.json`.

Enabled streams are normalized through the same event model. The dashboard and
Ollama correlate source IPs that occur across two or more streams, for example
FortiGate, AdGuard/DNS, Windows Security, Nginx, Squid, VPN, or Proxmox.

The current MCP endpoint is `http://<graylog-host>:9000/api/mcp`. Enable it in
Graylog under `System -> Configurations -> MCP` and use stream IDs internally;
the fgAI stream picker resolves titles in the UI.

## Monitoring Export

The dashboard also exposes Prometheus text metrics at:

```text
http://127.0.0.1:8088/metrics
```

This endpoint is passive and has no Prometheus or Grafana dependency. It reports
low-cardinality event counts, anomaly severities, baseline readiness, and Graylog
MCP health. Use it later as a Prometheus scrape target or as input for a Checkmk
local check. Do not use source IPs, domains, or raw event IDs as metric labels.

Enable cached Ollama analyst notes in the dashboard:

```bash
FGAI_LLM=1 OLLAMA_MODEL=llama3.1 ./start.sh restart
```

The monitor refreshes deterministic detections every `FGAI_MONITOR_INTERVAL` seconds and refreshes the LLM note every `FGAI_LLM_INTERVAL` seconds, default `300`.

The script activates `.venv` inside the script process. If you also want your current shell prompt to show the venv, run:

```bash
source .venv/bin/activate
```

For UDP `514`, the script starts only the listener command with `sudo`:

```bash
FGAI_SYSLOG_PORT=514 ./start.sh
```

The syslog receiver rotates the active JSONL input at 25 MB by default. Rotated
files are gzip-compressed and 14 archives are retained. Override this when needed:

```bash
FGAI_LOG_ROTATE_BYTES=$((100 * 1024 * 1024)) FGAI_LOG_ROTATE_COUNT=30 ./start.sh restart
```

The continuous monitor also stores a local SQLite behavior baseline at
`state/fgai-baseline.sqlite3`. A source becomes baseline-ready after 12 completed
five-minute windows. Historical rate and hitcount-rate deviations then contribute
to its anomaly score. Set `FGAI_BASELINE_DB` to use another location.

Analyze local logs:

```bash
fgai analyze-logs --logs logs/fg_syslog.jsonl
```

Open the live UI after `./start.sh`:

```bash
xdg-open http://127.0.0.1:8088
```

Score likely traffic anomalies:

```bash
fgai detect-anomalies --logs logs/fg_syslog.jsonl --min-score 35
fgai detect-anomalies --logs logs/fg_syslog.jsonl --min-score 35 --llm --llm-timeout 300
```

Generate response and policy recommendations:

```bash
fgai recommend --logs logs/fg_syslog.jsonl --min-score 35
```

Optional external reputation enrichment is disabled by default. To use VirusTotal for public source/destination IP reputation:

```bash
export FGAI_THREAT_INTEL=1
export ABUSEIPDB_API_KEY='...'
fgai recommend --logs logs/fg_syslog.jsonl --min-score 35 --threat-intel
```

VirusTotal is also supported:

```bash
export FGAI_THREAT_INTEL=1
export FGAI_THREAT_INTEL_PROVIDER=virustotal
export VIRUSTOTAL_API_KEY='...'
fgai recommend --logs logs/fg_syslog.jsonl --min-score 35 --threat-intel
```

Listen for FortiGate syslog locally:

```bash
fgai listen-syslog --port 5514 --output logs/fg_syslog.jsonl
```

Run the listener quietly in the background:

```bash
./start.sh
```

Stop the background listener:

```bash
./start.sh stop
```

UDP port `514` normally needs root privileges on Linux:

```bash
sudo .venv/bin/fgai listen-syslog --port 514 --output logs/fg_syslog.jsonl
```

Test FortiGate API access:

```bash
export FORTIGATE_HOST=192.0.2.10
export FORTIGATE_API_TOKEN='...'
export FORTIGATE_VERIFY_TLS=false
fgai test-connection
fgai fetch-policies --output exports/policies.json
```

Audit a FortiGate policy export:

```bash
fgai audit-policies --config exports/fortigate.conf
```

Or fetch policies through the FortiGate API and audit that JSON:

```bash
fgai fetch-policies --output exports/policies.json
fgai audit-policies --config exports/policies.json --llm --llm-timeout 300
```

Find block candidates without changing the firewall:

```bash
fgai suggest-blocks --logs logs/fg_syslog.jsonl
```

Execute guarded quarantine actions:

```bash
export FORTIGATE_HOST=192.0.2.10
export FORTIGATE_API_TOKEN='...'
fgai suggest-blocks --logs logs/fg_syslog.jsonl --execute --expiry-minutes 60
```

Optional local LLM summary through Ollama:

```bash
ollama pull llama3.3
fgai analyze-logs --logs logs/fg_syslog.jsonl --llm --llm-timeout 300
```

For slower machines or large models:

```bash
OLLAMA_MODEL=llama3.1 OLLAMA_TIMEOUT=300 fgai analyze-logs --logs logs/fg_syslog.jsonl --llm
```

## FortiGate Inputs

For logs, configure FortiGate syslog to write into a local file such as `logs/fg_syslog.jsonl`. The parser supports common key/value syslog lines and JSONL.

For policies, export a FortiOS config backup and pass it to `audit-policies`.

Example FortiGate syslog target, run on the FortiGate CLI and replace the server IP with this machine:

```text
config log syslogd setting
    set status enable
    set server "192.0.2.50"
    set port 5514
    set mode udp
    set format default
end
```

## Environment

- `FORTIGATE_HOST`: firewall hostname or IP.
- `FORTIGATE_API_TOKEN`: REST API token.
- `FORTIGATE_VERIFY_TLS`: `true` or `false`, defaults to `true`.
- `FGAI_ALLOWLIST`: comma-separated IPs/CIDRs never to block.
- `OLLAMA_HOST`: defaults to `http://127.0.0.1:11434`.
- `OLLAMA_MODEL`: defaults to `llama3.1`.
- `OLLAMA_TIMEOUT`: Ollama request timeout in seconds, defaults to `180`.
- `FGAI_LLM`: set to `1` to enable dashboard Ollama analyst notes.
- `FGAI_LLM_INTERVAL`: seconds between dashboard LLM notes, defaults to `300`.
- `FGAI_THREAT_INTEL`: set to `1` to enable external threat intelligence lookups.
- `ABUSEIPDB_API_KEY`: AbuseIPDB API key for public IP reputation enrichment.
- `ABUSEIPDB_MAX_AGE_DAYS`: report age window for AbuseIPDB, defaults to `90`.
- `FGAI_THREAT_INTEL_PROVIDER`: `auto`, `abuseipdb`, or `virustotal`.
- `VIRUSTOTAL_API_KEY`: VirusTotal API key for public IP reputation enrichment.

## Safety Model

The agent separates detection from enforcement:

- UTM events are scored from FortiGate logs (`ips`, `virus`, `anomaly`, `ddos`, `webfilter`, `app-ctrl`, `waf`, `dns`).
- Source IPs must be globally routable and outside the allowlist.
- Blocking requires `--execute`.
- The FortiGate API call is limited to the quarantine/banned user monitor endpoint.