Public/fgAI
SHA256
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Field baseline deviations now show distinct Mark expected, Mark false positive, and Mark confirmed

Browse Source
This commit is contained in:
larssand
2026-06-24 18:54:37 +02:00
parent 63449c9e80
commit f6bee0438c
6 changed files with 110 additions and 290 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
.project Normal file
View File

@@ -0,0 +1,11 @@
<?xml version="1.0" encoding="UTF-8"?>
<projectDescription>
<name>fgAI</name>
<comment></comment>
<projects>
</projects>
<buildSpec>
</buildSpec>
<natures>
</natures>
</projectDescription>

2
README.md
View File

@@ -155,6 +155,8 @@ export VIRUSTOTAL_API_KEY='...'
fgai recommend --logs logs/fg_syslog.jsonl --min-score 35 --threat-intel fgai recommend --logs logs/fg_syslog.jsonl --min-score 35 --threat-intel
``` ```
Threat intelligence responses are cached locally in `state/threat-intel-cache.json`. Successful results are reused for seven days by default, failures for one hour, and SignalScope permits at most 100 new provider lookups per UTC day. Cached responses are returned even after that budget is reached. Tune these safeguards with `FGAI_THREAT_INTEL_TTL_SECONDS`, `FGAI_THREAT_INTEL_ERROR_TTL_SECONDS`, and `FGAI_THREAT_INTEL_DAILY_LIMIT`.
Listen for FortiGate syslog locally: Listen for FortiGate syslog locally:
```bash ```bash

283
README.md~
View File

@@ -1,283 +0,0 @@
# SignalScope
SignalScope is a local multi-source security analytics agent. Its primary mode connects to Graylog through MCP, discovers the streams and fields already available in your environment, and uses stream profiles to normalize events, build baselines, correlate entities, and explain anomalies with a local LLM.
Its running only locally and if using LLM it's running also locally so no data is sent or exposed outside.
FortiGate is one supported example. The same workflow applies to DNS/AdGuard, Windows Event Logs, Sysmon, Nginx, Squid, VPN, Proxmox, Filebeat-collected logs, and future Graylog streams.
The Python module and legacy `fgai` command remain available for compatibility. New installations can use `signalscope`.
Autoblocking is dry-run by default. The tool will not block RFC1918, loopback, multicast, link-local, reserved, or allowlisted addresses unless you change the code.
## Quick Start
```bash
python -m venv .venv
source .venv/bin/activate
pip install -e .
```
Or use the helper script, which creates/uses `.venv` automatically and runs `pip install -e .`:
```bash
./start.sh
./start.sh status
./start.sh analyze
./start.sh stop
```
`./start.sh` starts three local background processes:
- Optional UDP syslog listener writing `logs/fg_syslog.jsonl`
- Continuous monitor writing `state/fgai-status.json`
- Local dashboard at `http://127.0.0.1:8088`
## Primary Workflow: Graylog MCP
Graylog 7.1 MCP is the primary log-source integration. In the dashboard, open
`Settings`, select `Graylog MCP`, provide the MCP URL and a read-only API token,
then load and enable the streams to analyze. SignalScope uses MCP `list_streams`,
`list_fields`, `search_messages`, and `aggregate_messages` to work with existing
log sources rather than requiring every source to be forwarded locally.
The token field accepts a raw Graylog API token, the Base64 value after `Basic `,
or a complete `Basic <value>` header. Tokens are stored only in the local runtime
configuration and are never returned by the dashboard API.
Use `Load selected stream fields` after choosing a stream. The field table shows
Graylog datatype/capability metadata and lets you select an entity field, a time
field, and categorical/numeric fields for the stream profile. Profiles are stored
under `graylog_stream_profiles` in `state/fgai-config.json`.
Enabled streams are normalized through the same event model. Stream profiles
define the entity, timestamp, categorical, and numeric fields used for baselines.
The dashboard and Ollama then correlate behavior across sources, for example a
client IP appearing in FortiGate, AdGuard/DNS, Windows Security, Nginx, Squid,
VPN, or Proxmox.
The current MCP endpoint is `http://<graylog-host>:9000/api/mcp`. Enable it in
Graylog under `System -> Configurations -> MCP` and use stream IDs internally;
the fgAI stream picker resolves titles in the UI.
## Monitoring Export
The dashboard also exposes Prometheus text metrics at:
```text
http://127.0.0.1:8088/metrics
```
This endpoint is passive and has no Prometheus or Grafana dependency. It reports
low-cardinality event counts, anomaly severities, baseline readiness, and Graylog
MCP health. Use it later as a Prometheus scrape target or as input for a Checkmk
local check. Do not use source IPs, domains, or raw event IDs as metric labels.
Enable cached Ollama analyst notes in the dashboard:
```bash
FGAI_LLM=1 OLLAMA_MODEL=llama3.1 ./start.sh restart
```
The monitor refreshes deterministic detections every `FGAI_MONITOR_INTERVAL` seconds and refreshes the LLM note every `FGAI_LLM_INTERVAL` seconds, default `300`.
The script activates `.venv` inside the script process. If you also want your current shell prompt to show the venv, run:
```bash
source .venv/bin/activate
```
For UDP `514`, the script starts only the listener command with `sudo`:
```bash
FGAI_SYSLOG_PORT=514 ./start.sh
```
The syslog receiver rotates the active JSONL input at 25 MB by default. Rotated
files are gzip-compressed and 14 archives are retained. Override this when needed:
```bash
FGAI_LOG_ROTATE_BYTES=$((100 * 1024 * 1024)) FGAI_LOG_ROTATE_COUNT=30 ./start.sh restart
```
The continuous monitor also stores a local SQLite behavior baseline at
`state/fgai-baseline.sqlite3`. A source becomes baseline-ready after 12 completed
five-minute windows. Historical rate and hitcount-rate deviations then contribute
to its anomaly score. Set `FGAI_BASELINE_DB` to use another location.
Analyze local logs:
```bash
fgai analyze-logs --logs logs/fg_syslog.jsonl
```
Open the live UI after `./start.sh`:
```bash
xdg-open http://127.0.0.1:8088
```
Score likely traffic anomalies:
```bash
fgai detect-anomalies --logs logs/fg_syslog.jsonl --min-score 35
fgai detect-anomalies --logs logs/fg_syslog.jsonl --min-score 35 --llm --llm-timeout 300
```
Generate response and policy recommendations:
```bash
fgai recommend --logs logs/fg_syslog.jsonl --min-score 35
```
Optional external reputation enrichment is disabled by default. To use VirusTotal for public source/destination IP reputation:
```bash
export FGAI_THREAT_INTEL=1
export ABUSEIPDB_API_KEY='...'
fgai recommend --logs logs/fg_syslog.jsonl --min-score 35 --threat-intel
```
VirusTotal is also supported:
```bash
export FGAI_THREAT_INTEL=1
export FGAI_THREAT_INTEL_PROVIDER=virustotal
export VIRUSTOTAL_API_KEY='...'
fgai recommend --logs logs/fg_syslog.jsonl --min-score 35 --threat-intel
```
Listen for FortiGate syslog locally:
```bash
fgai listen-syslog --port 5514 --output logs/fg_syslog.jsonl
```
Run the listener quietly in the background:
```bash
./start.sh
```
Stop the background listener:
```bash
./start.sh stop
```
UDP port `514` normally needs root privileges on Linux:
```bash
sudo .venv/bin/fgai listen-syslog --port 514 --output logs/fg_syslog.jsonl
```
Test FortiGate API access:
```bash
export FORTIGATE_HOST=192.0.2.10
export FORTIGATE_API_TOKEN='...'
export FORTIGATE_VERIFY_TLS=false
fgai test-connection
fgai fetch-policies --output exports/policies.json
```
Audit a FortiGate policy export:
```bash
fgai audit-policies --config exports/fortigate.conf
```
Or fetch policies through the FortiGate API and audit that JSON:
```bash
fgai fetch-policies --output exports/policies.json
fgai audit-policies --config exports/policies.json --llm --llm-timeout 300
```
Find block candidates without changing the firewall:
```bash
fgai suggest-blocks --logs logs/fg_syslog.jsonl
```
Execute guarded quarantine actions:
```bash
export FORTIGATE_HOST=192.0.2.10
export FORTIGATE_API_TOKEN='...'
fgai suggest-blocks --logs logs/fg_syslog.jsonl --execute --expiry-minutes 60
```
Optional local LLM summary through Ollama:
```bash
ollama pull llama3.3
fgai analyze-logs --logs logs/fg_syslog.jsonl --llm --llm-timeout 300
```
For slower machines or large models:
```bash
OLLAMA_MODEL=llama3.1 OLLAMA_TIMEOUT=300 fgai analyze-logs --logs logs/fg_syslog.jsonl --llm
```
## Optional FortiGate Input
## Synthetic Windows Test Input
For testing a Graylog Beats input without a Windows host, generate Windows
Security-style JSONL events locally, then use Filebeat to ship them over TCP:
```bash
python scripts/generate_windows_events.py --interval 0.5
filebeat -e -c examples/filebeat-windows-synthetic.yml
```
Update the absolute JSONL path and Graylog host in the Filebeat template first.
Route `stream_hint: Windows` to a dedicated Graylog stream, then enable that
stream in SignalScope and configure a profile such as entity `user` or
`source_ip`, categorical `event_id`, `status`, `logon_type`, and numeric fields
when present. Filebeat uses its Logstash output to communicate with Graylog's
Beats input on TCP `5044`. [Graylog Beats input documentation](https://go2docs.graylog.org/current/getting_in_log_data/beats_input.html)
For logs, configure FortiGate syslog to write into a local file such as `logs/fg_syslog.jsonl`. The parser supports common key/value syslog lines and JSONL.
For policies, export a FortiOS config backup and pass it to `audit-policies`.
Example FortiGate syslog target, run on the FortiGate CLI and replace the server IP with this machine:
```text
config log syslogd setting
set status enable
set server "192.0.2.50"
set port 5514
set mode udp
set format default
end
```
## Environment
- `FORTIGATE_HOST`: firewall hostname or IP.
- `FORTIGATE_API_TOKEN`: REST API token.
- `FORTIGATE_VERIFY_TLS`: `true` or `false`, defaults to `true`.
- `FGAI_ALLOWLIST`: comma-separated IPs/CIDRs never to block.
- `OLLAMA_HOST`: defaults to `http://127.0.0.1:11434`.
- `OLLAMA_MODEL`: defaults to `llama3.1`.
- `OLLAMA_TIMEOUT`: Ollama request timeout in seconds, defaults to `180`.
- `FGAI_LLM`: set to `1` to enable dashboard Ollama analyst notes.
- `FGAI_LLM_INTERVAL`: seconds between dashboard LLM notes, defaults to `300`.
- `FGAI_THREAT_INTEL`: set to `1` to enable external threat intelligence lookups.
- `ABUSEIPDB_API_KEY`: AbuseIPDB API key for public IP reputation enrichment.
- `ABUSEIPDB_MAX_AGE_DAYS`: report age window for AbuseIPDB, defaults to `90`.
- `FGAI_THREAT_INTEL_PROVIDER`: `auto`, `abuseipdb`, or `virustotal`.
- `VIRUSTOTAL_API_KEY`: VirusTotal API key for public IP reputation enrichment.
## Safety Model
The agent separates detection from enforcement:
- UTM events are scored from FortiGate logs (`ips`, `virus`, `anomaly`, `ddos`, `webfilter`, `app-ctrl`, `waf`, `dns`).
- Source IPs must be globally routable and outside the allowlist.
- Blocking requires `--execute`.
- The FortiGate API call is limited to the quarantine/banned user monitor endpoint.

11
src/fgai/dashboard.py
View File

@@ -55,6 +55,10 @@ HTML = """<!doctype html>
.field-row code { width: fit-content; } .field-row code { width: fit-content; }
.field-controls { display: flex; flex-wrap: wrap; gap: 10px; } .field-controls { display: flex; flex-wrap: wrap; gap: 10px; }
.field-controls label { white-space: nowrap; } .field-controls label { white-space: nowrap; }
.review-actions { display: flex; flex-wrap: wrap; gap: 6px; min-width: 250px; }
.review-actions button { border: 1px solid #39709a; background: #0b2944; color: #d9e8f7; padding: 6px 8px; cursor: pointer; }
.review-actions button[data-status="false_positive"] { border-color: #b7823a; color: #ffd36e; }
.review-actions button[data-status="confirmed"] { border-color: #2a9b6e; color: #7be3ae; }
.chart { width: 100%; height: 220px; background: #04182d; border: 1px solid #163b59; } .chart { width: 100%; height: 220px; background: #04182d; border: 1px solid #163b59; }
@media (max-width: 860px) { .hero, .split { grid-template-columns: 1fr; } .hero img { display: none; } } @media (max-width: 860px) { .hero, .split { grid-template-columns: 1fr; } .hero img { display: none; } }
</style> </style>
@@ -71,7 +75,7 @@ HTML = """<!doctype html>
</section> </section>
<nav class="tabs" aria-label="Dashboard views"><button class="tab active" data-tab="overview">Overview</button><button class="tab" data-tab="findings">Findings</button><button class="tab" data-tab="diagnostics">Diagnostics</button><button class="tab" data-tab="settings">Settings</button></nav> <nav class="tabs" aria-label="Dashboard views"><button class="tab active" data-tab="overview">Overview</button><button class="tab" data-tab="findings">Findings</button><button class="tab" data-tab="diagnostics">Diagnostics</button><button class="tab" data-tab="settings">Settings</button></nav>
<div data-view="overview" class="active"><section class="split"><div class="panel"><h2>Events and Anomalies</h2><canvas id="trendChart" class="chart"></canvas></div><div class="panel"><h2>Baseline and Stream Health</h2><div id="health"></div></div></section><section class="panel"><h2>AI Assessment</h2><div id="llmAssessment" class="muted">LLM assessment disabled.</div></section><section class="split"><div class="panel"><h2>Anomalies</h2><div id="anomalies"></div></div><div class="panel"><h2>Recommendations</h2><div id="recommendations"></div></div></section></div> <div data-view="overview" class="active"><section class="split"><div class="panel"><h2>Events and Anomalies</h2><canvas id="trendChart" class="chart"></canvas></div><div class="panel"><h2>Baseline and Stream Health</h2><div id="health"></div></div></section><section class="panel"><h2>AI Assessment</h2><div id="llmAssessment" class="muted">LLM assessment disabled.</div></section><section class="split"><div class="panel"><h2>Anomalies</h2><div id="anomalies"></div></div><div class="panel"><h2>Recommendations</h2><div id="recommendations"></div></div></section></div>
<div data-view="findings"><section class="panel"><h2>Field Baseline Deviations</h2><div id="fieldDeviations"></div></section><section class="panel"><h2>Related Activity Across Sources</h2><div id="relatedActivity"></div></section><section class="split"><div class="panel"><h2>Block Candidates</h2><div id="blocks"></div></div><div class="panel"><h2>Threat Intelligence</h2><div id="reputation"></div></div></section><section class="panel"><h2>Policy Findings</h2><div id="policies"></div></section></div> <div data-view="findings"><section class="panel"><h2>Field Baseline Deviations</h2><div id="feedbackNotice" class="muted" role="status"></div><div id="fieldDeviations"></div></section><section class="panel"><h2>Related Activity Across Sources</h2><div id="relatedActivity"></div></section><section class="split"><div class="panel"><h2>Block Candidates</h2><div id="blocks"></div></div><div class="panel"><h2>Threat Intelligence</h2><div id="reputation"></div></div></section><section class="panel"><h2>Policy Findings</h2><div id="policies"></div></section></div>
<div data-view="diagnostics"><section class="panel"><h2>Diagnostics</h2><div id="diagnostics"></div></section></div> <div data-view="diagnostics"><section class="panel"><h2>Diagnostics</h2><div id="diagnostics"></div></section></div>
<div data-view="settings"><section class="panel"><h2>Runtime Configuration</h2><form id="settingsForm"><div class="grid"><label>Log source<br><select name="log_source"><option value="local_syslog">Local syslog</option><option value="graylog_mcp">Graylog MCP</option></select></label><label>Graylog MCP URL<br><input name="graylog_mcp_url" type="url" placeholder="https://graylog.example/api/mcp"></label><label>Graylog streams<br><button type="button" id="loadStreams">Load streams</button><div id="streamPicker" class="muted">Load streams after URL and token are saved.</div></label><label>Discover fields<br><button type="button" id="loadFields">Load selected stream fields</button><div id="fieldPicker" class="muted">Select a stream first.</div></label><label>Graylog query<br><input name="graylog_query" placeholder="*"></label><label>Graylog field mapping (JSON)<br><textarea name="graylog_field_mapping" placeholder='{"srcip":"client_ip","dstip":"server_ip","action":"event_action"}'></textarea></label><label>Graylog MCP token<br><input name="graylog_mcp_token" type="password" placeholder="Leave blank to keep current token"></label><label>Ollama model<br><input name="llm_model" placeholder="llama3.1"></label><label><input name="llm_enabled" type="checkbox"> Enable Ollama analysis</label><label><input name="threat_intel_enabled" type="checkbox"> Enable threat intelligence</label></div><p><button type="submit">Save configuration</button> <span id="settingsResult" class="muted"></span></p></form></section></div> <div data-view="settings"><section class="panel"><h2>Runtime Configuration</h2><form id="settingsForm"><div class="grid"><label>Log source<br><select name="log_source"><option value="local_syslog">Local syslog</option><option value="graylog_mcp">Graylog MCP</option></select></label><label>Graylog MCP URL<br><input name="graylog_mcp_url" type="url" placeholder="https://graylog.example/api/mcp"></label><label>Graylog streams<br><button type="button" id="loadStreams">Load streams</button><div id="streamPicker" class="muted">Load streams after URL and token are saved.</div></label><label>Discover fields<br><button type="button" id="loadFields">Load selected stream fields</button><div id="fieldPicker" class="muted">Select a stream first.</div></label><label>Graylog query<br><input name="graylog_query" placeholder="*"></label><label>Graylog field mapping (JSON)<br><textarea name="graylog_field_mapping" placeholder='{"srcip":"client_ip","dstip":"server_ip","action":"event_action"}'></textarea></label><label>Graylog MCP token<br><input name="graylog_mcp_token" type="password" placeholder="Leave blank to keep current token"></label><label>Ollama model<br><input name="llm_model" placeholder="llama3.1"></label><label><input name="llm_enabled" type="checkbox"> Enable Ollama analysis</label><label><input name="threat_intel_enabled" type="checkbox"> Enable threat intelligence</label></div><p><button type="submit">Save configuration</button> <span id="settingsResult" class="muted"></span></p></form></section></div>
</main> </main>
@@ -157,12 +161,13 @@ async function refresh() {
const streamTitles = Object.fromEntries((configuration.graylog_streams || []).map(item => [item.id, item.title || item.id])); const streamTitles = Object.fromEntries((configuration.graylog_streams || []).map(item => [item.id, item.title || item.id]));
const fieldRows = Object.entries(data.field_deviations || {}).flatMap(([entity, deviations]) => (deviations || []).map(item => ({entity, stream_title: streamTitles[item.stream_id] || item.stream_id, ...item}))); const fieldRows = Object.entries(data.field_deviations || {}).flatMap(([entity, deviations]) => (deviations || []).map(item => ({entity, stream_title: streamTitles[item.stream_id] || item.stream_id, ...item})));
document.getElementById('fieldDeviations').innerHTML = table(fieldRows, [ document.getElementById('fieldDeviations').innerHTML = table(fieldRows, [
{label:'Entity', key:'entity'}, {label:'Stream', key:'stream_title'}, {label:'Field', key:'field'}, {label:'Score', key:'score'}, {label:'Review', render:r => esc(r.feedback || 'unreviewed')}, {label:'Evidence', render:r => { const summary=esc(`${r.reason}; current ${r.current ?? '-'} vs baseline ${r.baseline ?? '-'}; values: ${(r.sample_values || []).join(', ') || '-'}`); const events=(r.sample_events || []).map(item => esc(`${item.timestamp} | ${item.source} -> ${item.destination} | ${item.action} ${item.service} | ${item.value} | ${item.message}`)).join('<br>'); return events ? `<details><summary>${summary}</summary><p>${events}</p></details>` : summary; }}, {label:'Action', render:r => `<button class="feedback" data-status="expected" data-entity="${esc(r.entity)}" data-stream="${esc(r.stream_id)}" data-field="${esc(r.field)}">Expected</button> <button class="feedback" data-status="false_positive" data-entity="${esc(r.entity)}" data-stream="${esc(r.stream_id)}" data-field="${esc(r.field)}">False positive</button> <button class="feedback" data-status="confirmed" data-entity="${esc(r.entity)}" data-stream="${esc(r.stream_id)}" data-field="${esc(r.field)}">Confirm</button>`} {label:'Entity', key:'entity'}, {label:'Stream', key:'stream_title'}, {label:'Field', key:'field'}, {label:'Score', key:'score'}, {label:'Review', render:r => esc(r.feedback || 'unreviewed')}, {label:'Evidence', render:r => { const summary=esc(`${r.reason}; current ${r.current ?? '-'} vs baseline ${r.baseline ?? '-'}; values: ${(r.sample_values || []).join(', ') || '-'}`); const events=(r.sample_events || []).map(item => esc(`${item.timestamp} | ${item.source} -> ${item.destination} | ${item.action} ${item.service} | ${item.value} | ${item.message}`)).join('<br>'); return events ? `<details><summary>${summary}</summary><p>${events}</p></details>` : summary; }}, {label:'Review action', render:r => `<div class="review-actions"><button class="feedback" data-status="expected" data-entity="${esc(r.entity)}" data-stream="${esc(r.stream_id)}" data-field="${esc(r.field)}">Mark expected</button><button class="feedback" data-status="false_positive" data-entity="${esc(r.entity)}" data-stream="${esc(r.stream_id)}" data-field="${esc(r.field)}">Mark false positive</button><button class="feedback" data-status="confirmed" data-entity="${esc(r.entity)}" data-stream="${esc(r.stream_id)}" data-field="${esc(r.field)}">Mark confirmed</button></div>`}
]); ]);
document.querySelectorAll('.feedback').forEach(button => button.addEventListener('click', async () => { document.querySelectorAll('.feedback').forEach(button => button.addEventListener('click', async () => {
const note = prompt('Review note (optional):') || ''; const note = prompt('Review note (optional):') || '';
const days = prompt('Expiry in days (0 = no expiry):', '0') || '0'; const days = prompt('Expiry in days (0 = no expiry):', '0') || '0';
await fetch('/api/feedback', {method:'POST', headers:{'Content-Type':'application/json'}, body:JSON.stringify({status:button.dataset.status, entity:button.dataset.entity, stream_id:button.dataset.stream, field:button.dataset.field, note, expires_at: Number(days) > 0 ? Math.floor(Date.now()/1000) + Number(days) * 86400 : 0})}); const response = await fetch('/api/feedback', {method:'POST', headers:{'Content-Type':'application/json'}, body:JSON.stringify({status:button.dataset.status, entity:button.dataset.entity, stream_id:button.dataset.stream, field:button.dataset.field, note, expires_at: Number(days) > 0 ? Math.floor(Date.now()/1000) + Number(days) * 86400 : 0})});
document.getElementById('feedbackNotice').textContent = response.ok ? 'Review saved. The matching pattern will be labeled on the next refresh.' : 'Could not save review.';
refresh(); refresh();
})); }));
document.getElementById('relatedActivity').innerHTML = table(relatedRows, [ document.getElementById('relatedActivity').innerHTML = table(relatedRows, [

59
src/fgai/threat_intel.py
View File

@@ -19,20 +19,29 @@ def is_public_ip(value: str | None) -> bool:
class ThreatIntelClient: class ThreatIntelClient:
def __init__(self, *, cache_file: str = "state/threat-intel-cache.json", ttl_seconds: int = 86400, enabled: bool | None = None) -> None: def __init__(self, *, cache_file: str = "state/threat-intel-cache.json", ttl_seconds: int | None = None, enabled: bool | None = None) -> None:
self.enabled = os.getenv("FGAI_THREAT_INTEL", "").lower() in {"1", "true", "yes", "on"} if enabled is None else enabled self.enabled = os.getenv("FGAI_THREAT_INTEL", "").lower() in {"1", "true", "yes", "on"} if enabled is None else enabled
self.abuseipdb_key = os.getenv("ABUSEIPDB_API_KEY") self.abuseipdb_key = os.getenv("ABUSEIPDB_API_KEY")
self.virustotal_key = os.getenv("VIRUSTOTAL_API_KEY") self.virustotal_key = os.getenv("VIRUSTOTAL_API_KEY")
self.provider = os.getenv("FGAI_THREAT_INTEL_PROVIDER", "auto").lower() self.provider = os.getenv("FGAI_THREAT_INTEL_PROVIDER", "auto").lower()
self.max_age_days = int(os.getenv("ABUSEIPDB_MAX_AGE_DAYS", "90")) self.max_age_days = int(os.getenv("ABUSEIPDB_MAX_AGE_DAYS", "90"))
self.cache_path = Path(cache_file) self.cache_path = Path(cache_file)
self.ttl_seconds = ttl_seconds self.ttl_seconds = ttl_seconds if ttl_seconds is not None else int(os.getenv("FGAI_THREAT_INTEL_TTL_SECONDS", "604800"))
self.error_ttl_seconds = int(os.getenv("FGAI_THREAT_INTEL_ERROR_TTL_SECONDS", "3600"))
self.daily_limit = int(os.getenv("FGAI_THREAT_INTEL_DAILY_LIMIT", "100"))
self.cache = self._read_cache() self.cache = self._read_cache()
def status(self) -> dict[str, object]: def status(self) -> dict[str, object]:
provider = self._select_provider() provider = self._select_provider()
has_key = bool(self.abuseipdb_key if provider == "abuseipdb" else self.virustotal_key) has_key = bool(self.abuseipdb_key if provider == "abuseipdb" else self.virustotal_key)
return {"enabled": self.enabled, "provider": provider, "configured": has_key} return {
"enabled": self.enabled,
"provider": provider,
"configured": has_key,
"cache_ttl_seconds": self.ttl_seconds,
"daily_limit": self.daily_limit,
"requests_today": self._requests_today(provider),
}
def _read_cache(self) -> dict[str, dict[str, object]]: def _read_cache(self) -> dict[str, dict[str, object]]:
if not self.cache_path.exists(): if not self.cache_path.exists():
@@ -46,6 +55,39 @@ class ThreatIntelClient:
self.cache_path.parent.mkdir(parents=True, exist_ok=True) self.cache_path.parent.mkdir(parents=True, exist_ok=True)
self.cache_path.write_text(json.dumps(self.cache, indent=2, sort_keys=True), encoding="utf-8") self.cache_path.write_text(json.dumps(self.cache, indent=2, sort_keys=True), encoding="utf-8")
def _cache_fresh(self, cached: dict[str, object], now: int) -> bool:
ttl = self.ttl_seconds if cached.get("status") == "ok" else self.error_ttl_seconds
return now - int(cached.get("cached_at", 0)) < ttl
def _today(self) -> str:
return time.strftime("%Y-%m-%d", time.gmtime())
def _requests_today(self, provider: str) -> int:
meta = self.cache.get("_meta", {})
if not isinstance(meta, dict):
return 0
requests_by_day = meta.get("requests_by_day", {})
if not isinstance(requests_by_day, dict):
return 0
provider_counts = requests_by_day.get(provider, {})
return int(provider_counts.get(self._today(), 0)) if isinstance(provider_counts, dict) else 0
def _record_request(self, provider: str) -> None:
meta = self.cache.setdefault("_meta", {})
if not isinstance(meta, dict):
meta = {}
self.cache["_meta"] = meta
requests_by_day = meta.setdefault("requests_by_day", {})
if not isinstance(requests_by_day, dict):
requests_by_day = {}
meta["requests_by_day"] = requests_by_day
provider_counts = requests_by_day.setdefault(provider, {})
if not isinstance(provider_counts, dict):
provider_counts = {}
requests_by_day[provider] = provider_counts
today = self._today()
provider_counts[today] = int(provider_counts.get(today, 0)) + 1
def lookup_ip(self, ip: str) -> dict[str, object]: def lookup_ip(self, ip: str) -> dict[str, object]:
if not is_public_ip(ip): if not is_public_ip(ip):
return {"ip": ip, "provider": "local", "status": "skipped", "reason": "not_public_ip", "score": 0} return {"ip": ip, "provider": "local", "status": "skipped", "reason": "not_public_ip", "score": 0}
@@ -53,10 +95,18 @@ class ThreatIntelClient:
cache_key = f"{provider}:{ip}" cache_key = f"{provider}:{ip}"
cached = self.cache.get(cache_key) cached = self.cache.get(cache_key)
now = int(time.time()) now = int(time.time())
if cached and now - int(cached.get("cached_at", 0)) < self.ttl_seconds: if isinstance(cached, dict) and self._cache_fresh(cached, now):
return cached return cached
if not self.enabled: if not self.enabled:
return {"ip": ip, "provider": "none", "status": "disabled", "score": 0} return {"ip": ip, "provider": "none", "status": "disabled", "score": 0}
if self.daily_limit > 0 and self._requests_today(provider) >= self.daily_limit:
return {
"ip": ip,
"provider": provider,
"status": "daily_limit_reached",
"score": 0,
"reason": "external lookup budget reached; cached results remain available",
}
if provider == "abuseipdb": if provider == "abuseipdb":
if not self.abuseipdb_key: if not self.abuseipdb_key:
return {"ip": ip, "provider": "abuseipdb", "status": "missing_api_key", "score": 0} return {"ip": ip, "provider": "abuseipdb", "status": "missing_api_key", "score": 0}
@@ -68,6 +118,7 @@ class ThreatIntelClient:
else: else:
return {"ip": ip, "provider": provider, "status": "unsupported_provider", "score": 0} return {"ip": ip, "provider": provider, "status": "unsupported_provider", "score": 0}
self._record_request(provider)
result["cached_at"] = now result["cached_at"] = now
self.cache[cache_key] = result self.cache[cache_key] = result
self._write_cache() self._write_cache()

34
tests/test_recommendations.py
View File

@@ -1,4 +1,5 @@
import os import os
import tempfile
import unittest import unittest
from unittest.mock import patch from unittest.mock import patch
@@ -65,6 +66,39 @@ class RecommendationTests(unittest.TestCase):
self.assertEqual(client._select_provider(), "virustotal") self.assertEqual(client._select_provider(), "virustotal")
def test_successful_lookup_is_reused_from_cache(self):
with tempfile.TemporaryDirectory() as directory, patch.dict(
os.environ,
{"FGAI_THREAT_INTEL": "1", "VIRUSTOTAL_API_KEY": "test", "FGAI_THREAT_INTEL_PROVIDER": "virustotal"},
clear=True,
):
client = ThreatIntelClient(cache_file=f"{directory}/intel.json")
with patch.object(client, "_lookup_virustotal_ip", return_value={"ip": "8.8.8.8", "provider": "virustotal", "status": "ok", "score": 0}) as lookup:
client.lookup_ip("8.8.8.8")
client.lookup_ip("8.8.8.8")
self.assertEqual(lookup.call_count, 1)
self.assertEqual(client.status()["requests_today"], 1)
def test_daily_limit_prevents_new_external_lookups(self):
with tempfile.TemporaryDirectory() as directory, patch.dict(
os.environ,
{
"FGAI_THREAT_INTEL": "1",
"VIRUSTOTAL_API_KEY": "test",
"FGAI_THREAT_INTEL_PROVIDER": "virustotal",
"FGAI_THREAT_INTEL_DAILY_LIMIT": "1",
},
clear=True,
):
client = ThreatIntelClient(cache_file=f"{directory}/intel.json")
with patch.object(client, "_lookup_virustotal_ip", return_value={"ip": "8.8.8.8", "provider": "virustotal", "status": "ok", "score": 0}) as lookup:
client.lookup_ip("8.8.8.8")
limited = client.lookup_ip("1.1.1.1")
self.assertEqual(lookup.call_count, 1)
self.assertEqual(limited["status"], "daily_limit_reached")
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()