Public/fgAI
SHA256
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix retention

Browse Source
This commit is contained in:
larssand
2026-07-06 12:41:34 +02:00
parent 6a0df3bc74
commit 841f94dc9f
6 changed files with 159 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
README.md
View File

@@ -458,16 +458,19 @@ five-minute windows. Historical rate and hitcount-rate deviations then contribut
to its anomaly score. Set `FGAI_BASELINE_DB` to use another location.
SignalScope prunes old baseline buckets during each monitor cycle. The defaults
keep 14 days of buckets and dedupe history, prune stale one-off categorical
values after 7 days, and cap high-cardinality values per stream/entity/field.
keep 7 days of buckets and dedupe history, prune stale one-off categorical
values after 3 days, and cap high-cardinality values per stream/entity/field.
Very noisy fields such as raw messages, URLs, payloads, request/response bodies,
tokens, sessions, hashes and long values are still counted in bucket baselines
but are not stored as distinct rare-value candidates.
Tune these in the dashboard or in `state/fgai-config.json`:
```json
{
"baseline_training_days": 7,
"baseline_retention_days": 14,
"baseline_value_retention_days": 7,
"baseline_max_values_per_field": 2000
"baseline_retention_days": 7,
"baseline_value_retention_days": 3,
"baseline_max_values_per_field": 500
}
```
@@ -480,11 +483,12 @@ If an existing baseline database has already grown large, stop the monitor and
run a manual prune plus SQLite compaction:
```bash
signalscope baseline-maintenance --baseline-db state/fgai-baseline.sqlite3 --retention-days 14 --value-retention-days 7 --max-values-per-field 2000 --vacuum
signalscope baseline-maintenance --baseline-db state/fgai-baseline.sqlite3 --retention-days 7 --value-retention-days 3 --max-values-per-field 500 --vacuum
```
`VACUUM` can take time on a large database and should not be run while the
monitor is actively writing.
`VACUUM` can take time on a large database, needs free disk space roughly equal
to the database size, and should not be run while the monitor is actively
writing. Without `--vacuum`, SQLite may delete rows but keep the file size.
Analyze local logs: