Implemented the UI improvements. and roadmap

README.md

@@ -8,6 +8,8 @@ FortiGate is one supported example. The same workflow applies to DNS/AdGuard, Wi
 
 The Python module and legacy `fgai` command remain available for compatibility. New installations can use `signalscope`.
 
+The prioritized implementation plan is tracked in [ROADMAP.md](ROADMAP.md).
+
 Autoblocking is dry-run by default. The tool will not block RFC1918, loopback, multicast, link-local, reserved, or allowlisted addresses unless you change the code.
 
 ## Screenshots

ROADMAP.md

@@ -0,0 +1,113 @@
+# SignalScope Roadmap
+
+This roadmap tracks the path from the current local Graylog analytics agent to a
+production-ready, multi-source security analytics tool. Items are ordered by
+operational value and dependency, not by UI appeal.
+
+## Current Foundation
+
+- [x] Graylog MCP connection, stream discovery, field discovery, and stream selection.
+- [x] Per-stream profiles for entity, timestamp, categorical, and numeric fields.
+- [x] Local five-minute field baselines with duplicate-event protection.
+- [x] Time-aware baseline comparison using matching UTC weekday/hour when available.
+- [x] Generic entity correlation for IP addresses, users, and hostnames.
+- [x] Cross-stream correlation timelines and investigation incident grouping.
+- [x] Field-deviation review: expected, false positive, confirmed, note, and expiry.
+- [x] Local Ollama analyst assessment with incident and feedback context.
+- [x] Cached AbuseIPDB/VirusTotal enrichment with a daily request budget.
+- [x] Prometheus-compatible low-cardinality metrics endpoint.
+- [x] Local dashboard for status, findings, incidents, diagnostics, and configuration.
+
+## 1. Detection Quality
+
+Goal: make findings more accurate before adding more integrations.
+
+- [ ] Add baseline confidence based on sample count, profile age, and time-bucket coverage.
+- [ ] Add burst detection for event rate, authentication failures, DNS volume, and denied traffic.
+- [ ] Add rare-value detection with frequency thresholds and configurable field weights.
+- [ ] Add sequence detection, for example DNS lookup -> outbound connection -> authentication event.
+- [ ] Add per-stream detector enablement and thresholds in the UI.
+- [ ] Add a dry-run replay command to evaluate detector changes against a selected historic Graylog time range.
+
+Acceptance: each finding shows its detector, confidence, baseline sample count, current value, expected value, and a bounded set of raw-event references.
+
+## 2. Entity and Incident Investigation
+
+Goal: make one incident answer what happened, to whom, and across which sources.
+
+- [ ] Allow multiple entity fields per stream, such as user plus source IP plus hostname.
+- [ ] Add entity aliasing: map DHCP, VPN, DNS, and endpoint identities to the same host where evidence supports it.
+- [ ] Add configurable incident grouping windows and incident lifecycle: open, acknowledged, resolved, reopened.
+- [ ] Persist incident state and analyst notes separately from transient detection output.
+- [ ] Add direct Graylog query links or query details for each timeline event.
+- [ ] Add investigation export as JSON and Markdown report.
+
+Acceptance: an analyst can open an incident, see an ordered multi-stream timeline, review evidence, and record an outcome without losing it after the next monitor poll.
+
+## 3. Feedback and Learning Controls
+
+Goal: reduce repeat noise without hiding genuinely new behavior.
+
+- [ ] Show active feedback rules, expiry, scope, author, and matching findings in the UI.
+- [ ] Support feedback scope choices: exact value, entity and field, entire entity, or stream-wide rule.
+- [ ] Require an expiry for broad suppressions and warn before a broad rule is saved.
+- [ ] Track detector precision using confirmed versus false-positive feedback.
+- [ ] Include feedback-derived precision and suppression counts in metrics and diagnostics.
+
+Acceptance: a false-positive decision suppresses only the intended pattern and all suppressions are auditable and reversible.
+
+## 4. Threat Intelligence and Enrichment
+
+Goal: enrich public indicators predictably without exhausting provider quotas.
+
+- [ ] Make cache TTL and daily budget configurable in the dashboard.
+- [ ] Add provider health, cache hit ratio, and quota/budget visibility.
+- [ ] Add domain and URL enrichment behind separate provider budgets.
+- [ ] Keep enrichment asynchronous so a slow provider never delays monitoring.
+- [ ] Store enrichment provenance and observation timestamps with incident evidence.
+
+Acceptance: no refresh cycle causes uncontrolled external lookups, and every reputation result shows provider, timestamp, and cache state.
+
+## 5. Production Operations
+
+Goal: run reliably in a monitored environment.
+
+- [ ] Add systemd unit files for monitor, dashboard, and optional local syslog listener.
+- [ ] Add health and readiness endpoints with last successful Graylog fetch time.
+- [ ] Add structured application logs and configurable retention for status/history/baseline data.
+- [ ] Add backup and migration procedure for SQLite state.
+- [ ] Add Checkmk local-check output in addition to Prometheus metrics.
+- [ ] Add authentication/reverse-proxy guidance before exposing the dashboard beyond loopback.
+
+Acceptance: deployment, restart, upgrade, backup, and monitoring have documented, repeatable procedures.
+
+## 6. Dashboard Usability
+
+Goal: optimize the UI for security investigation rather than raw tables.
+
+- [ ] Replace remaining long tables with compact incident and entity cards where appropriate.
+- [ ] Add incident filters for stream, severity, entity type, review state, and time range.
+- [ ] Add baseline versus current charts per selected entity and field.
+- [ ] Add an interactive entity graph backed by actual correlations.
+- [ ] Preserve user view state, filters, expanded evidence, and selected tab across refreshes.
+- [ ] Add visual data-quality warnings for missing timestamps, entity fields, or truncated Graylog results.
+
+Acceptance: common triage can be completed from the dashboard without manually parsing raw JSON or searching for stream IDs.
+
+## 7. Integrations and Extensibility
+
+Goal: add log sources and outputs without adding source-specific logic everywhere.
+
+- [ ] Define versioned stream-profile templates for FortiGate, Windows, DNS/AdGuard, Nginx, Squid, VPN, and Proxmox.
+- [ ] Add import/export for profile templates and detector settings.
+- [ ] Separate source adapters, normalizers, detectors, enrichers, and output adapters into explicit extension interfaces.
+- [ ] Add optional webhook/SIEM ticket output for confirmed high-severity incidents.
+- [ ] Add role-aware configuration only if the dashboard is moved beyond single-user local operation.
+
+Acceptance: adding a new Graylog stream normally requires a profile template and configuration, not a core-code change.
+
+## Suggested Next Milestone
+
+Complete Detection Quality first: confidence scoring, burst/rare-value detectors,
+per-stream thresholds, and historical replay. These features determine whether
+the incident and dashboard work remains trustworthy as more log sources are added.

src/fgai/dashboard.py

@@ -60,6 +60,9 @@ HTML = """<!doctype html>
     .review-actions button[data-status="false_positive"] { border-color: #b7823a; color: #ffd36e; }
     .review-actions button[data-status="confirmed"] { border-color: #2a9b6e; color: #7be3ae; }
     .chart { width: 100%; height: 220px; background: #04182d; border: 1px solid #163b59; }
+    .graph { width: 100%; height: 300px; background: #04182d; border: 1px solid #163b59; }
+    .sort-button { border: 0; background: transparent; color: #83bce9; cursor: pointer; font: inherit; font-weight: 600; padding: 0; }
+    .sort-button:hover { color: #d9e8f7; }
     @media (max-width: 860px) { .hero, .split { grid-template-columns: 1fr; } .hero img { display: none; } }
   </style>
 </head>
@@ -74,10 +77,10 @@ HTML = """<!doctype html>
       </div>
     </section>
     <nav class="tabs" aria-label="Dashboard views"><button class="tab active" data-tab="overview">Overview</button><button class="tab" data-tab="findings">Findings</button><button class="tab" data-tab="diagnostics">Diagnostics</button><button class="tab" data-tab="settings">Settings</button></nav>
-    <div data-view="overview" class="active"><section class="split"><div class="panel"><h2>Events and Anomalies</h2><canvas id="trendChart" class="chart"></canvas></div><div class="panel"><h2>Baseline and Stream Health</h2><div id="health"></div></div></section><section class="panel"><h2>AI Assessment</h2><div id="llmAssessment" class="muted">LLM assessment disabled.</div></section><section class="panel"><h2>Investigation Incidents</h2><div id="incidents"></div></section><section class="split"><div class="panel"><h2>Anomalies</h2><div id="anomalies"></div></div><div class="panel"><h2>Recommendations</h2><div id="recommendations"></div></div></section></div>
+    <div data-view="overview" class="active"><section class="split"><div class="panel"><h2>Events and Anomalies</h2><canvas id="trendChart" class="chart"></canvas></div><div class="panel"><h2>Baseline and Stream Health</h2><div id="health"></div></div></section><section class="split"><div class="panel"><h2>Correlation Map</h2><canvas id="correlationGraph" class="graph"></canvas><div id="correlationGraphInfo" class="muted"></div></div><div class="panel"><h2>AI Assessment</h2><div id="llmAssessment" class="muted">LLM assessment disabled.</div></div></section><section class="panel"><h2>Investigation Incidents</h2><div id="incidents"></div></section><section class="split"><div class="panel"><h2>Anomalies</h2><div id="anomalies"></div></div><div class="panel"><h2>Recommendations</h2><div id="recommendations"></div></div></section></div>
     <div data-view="findings"><section class="panel"><h2>Field Baseline Deviations</h2><div id="feedbackNotice" class="muted" role="status"></div><div id="fieldDeviations"></div></section><section class="panel"><h2>Related Activity Across Sources</h2><div id="relatedActivity"></div></section><section class="split"><div class="panel"><h2>Block Candidates</h2><div id="blocks"></div></div><div class="panel"><h2>Threat Intelligence</h2><div id="reputation"></div></div></section><section class="panel"><h2>Policy Findings</h2><div id="policies"></div></section></div>
     <div data-view="diagnostics"><section class="panel"><h2>Diagnostics</h2><div id="diagnostics"></div></section></div>
-    <div data-view="settings"><section class="panel"><h2>Runtime Configuration</h2><form id="settingsForm"><div class="grid"><label>Log source<br><select name="log_source"><option value="local_syslog">Local syslog</option><option value="graylog_mcp">Graylog MCP</option></select></label><label>Graylog MCP URL<br><input name="graylog_mcp_url" type="url" placeholder="https://graylog.example/api/mcp"></label><label>Graylog streams<br><button type="button" id="loadStreams">Load streams</button><div id="streamPicker" class="muted">Load streams after URL and token are saved.</div></label><label>Discover fields<br><button type="button" id="loadFields">Load selected stream fields</button><div id="fieldPicker" class="muted">Select a stream first.</div></label><label>Graylog query<br><input name="graylog_query" placeholder="*"></label><label>Graylog field mapping (JSON)<br><textarea name="graylog_field_mapping" placeholder='{"srcip":"client_ip","dstip":"server_ip","action":"event_action"}'></textarea></label><label>Graylog MCP token<br><input name="graylog_mcp_token" type="password" placeholder="Leave blank to keep current token"></label><label>Ollama model<br><input name="llm_model" placeholder="llama3.1"></label><label><input name="llm_enabled" type="checkbox"> Enable Ollama analysis</label><label><input name="threat_intel_enabled" type="checkbox"> Enable threat intelligence</label></div><p><button type="submit">Save configuration</button> <span id="settingsResult" class="muted"></span></p></form></section></div>
+    <div data-view="settings"><section class="panel"><h2>Runtime Configuration</h2><form id="settingsForm"><div class="grid"><label>Log source<br><select name="log_source"><option value="local_syslog">Local syslog</option><option value="graylog_mcp">Graylog MCP</option></select></label><label>Graylog MCP URL<br><input name="graylog_mcp_url" type="url" placeholder="https://graylog.example/api/mcp"></label><label>Graylog streams<br><button type="button" id="loadStreams">Load streams</button><div id="streamPicker" class="muted">Load streams after URL and token are saved.</div></label><label>Discover fields<br><button type="button" id="loadFields">Load selected stream fields</button><div id="fieldPicker" class="muted">Select a stream first.</div></label><label>Profile name<br><input name="profile_name" placeholder="Example: Windows login behavior"></label><label>Graylog query<br><input name="graylog_query" placeholder="*"></label><label>Graylog field mapping (JSON)<br><textarea name="graylog_field_mapping" placeholder='{"srcip":"client_ip","dstip":"server_ip","action":"event_action"}'></textarea></label><label>Graylog MCP token<br><input name="graylog_mcp_token" type="password" placeholder="Leave blank to keep current token"></label><label>Ollama model<br><input name="llm_model" placeholder="llama3.1"></label><label><input name="llm_enabled" type="checkbox"> Enable Ollama analysis</label><label><input name="threat_intel_enabled" type="checkbox"> Enable threat intelligence</label></div><p><button type="submit">Save configuration</button> <span id="settingsResult" class="muted"></span></p></form></section></div>
   </main>
 <script>
 function esc(value) {
@@ -86,14 +89,18 @@ function esc(value) {
 function metric(label, value) {
   return `<div class="panel"><div class="metric">${esc(value)}</div><div class="label">${esc(label)}</div></div>`;
 }
-function table(rows, columns) {
+const tableSort = {};
+function table(rows, columns, id = '') {
   if (!rows || rows.length === 0) return '<p class="muted">No data.</p>';
-  const head = columns.map(c => `<th>${esc(c.label)}</th>`).join('');
+  const sort = tableSort[id];
-  const body = rows.map(row => `<tr>${columns.map(c => `<td>${c.render ? c.render(row) : esc(row[c.key])}</td>`).join('')}</tr>`).join('');
+  const sorted = sort ? [...rows].sort((left, right) => { const a=left[sort.key] ?? '', b=right[sort.key] ?? ''; const numeric=Number(a), numericB=Number(b); const compare=Number.isFinite(numeric) && Number.isFinite(numericB) && String(a).trim() !== '' && String(b).trim() !== '' ? numeric-numericB : String(a).localeCompare(String(b)); return sort.direction * compare; }) : rows;
+  const head = columns.map(c => `<th>${c.key && id ? `<button class="sort-button" data-sort-table="${esc(id)}" data-sort-key="${esc(c.key)}">${esc(c.label)}${sort && sort.key === c.key ? (sort.direction === 1 ? ' ▲' : ' ▼') : ''}</button>` : esc(c.label)}</th>`).join('');
+  const body = sorted.map(row => `<tr>${columns.map(c => `<td>${c.render ? c.render(row) : esc(row[c.key])}</td>`).join('')}</tr>`).join('');
   return `<div class="table-wrap"><table><thead><tr>${head}</tr></thead><tbody>${body}</tbody></table></div>`;
 }
 function capability(label, state, detail) { return `<span class="capability ${state}">${esc(label)}: ${esc(detail)}</span>`; }
-function drawTrend(history) { const canvas=document.getElementById('trendChart'), ctx=canvas.getContext('2d'), w=canvas.width=canvas.clientWidth*devicePixelRatio, h=canvas.height=canvas.clientHeight*devicePixelRatio; ctx.scale(devicePixelRatio,devicePixelRatio); const cw=canvas.clientWidth,ch=canvas.clientHeight; ctx.clearRect(0,0,cw,ch); const max=Math.max(1,...history.map(item=>item.events||0)); const line=(key,color)=>{ctx.strokeStyle=color;ctx.lineWidth=2;ctx.beginPath();history.forEach((item,index)=>{const x=12+index*Math.max(1,(cw-24)/Math.max(1,history.length-1));const y=ch-18-((item[key]||0)/max)*(ch-36);index?ctx.lineTo(x,y):ctx.moveTo(x,y)});ctx.stroke()}; line('events','#1ea9ff');line('anomalies','#ff5656'); }
+function drawTrend(history) { const canvas=document.getElementById('trendChart'), ctx=canvas.getContext('2d'), ratio=window.devicePixelRatio||1, cw=canvas.clientWidth,ch=canvas.clientHeight; canvas.width=cw*ratio; canvas.height=ch*ratio; ctx.scale(ratio,ratio); ctx.clearRect(0,0,cw,ch); if (!history.length) { ctx.fillStyle='#91abc4'; ctx.font='14px Arial'; ctx.fillText('Waiting for monitor history.', 16, 28); return; } const eventsMax=Math.max(1,...history.map(item=>Number(item.events)||0)), anomaliesMax=Math.max(1,...history.map(item=>Number(item.anomalies)||0)), left=40,right=40,top=28,bottom=24; ctx.strokeStyle='#163b59'; ctx.lineWidth=1; for(let index=0;index<4;index++){const y=top+index*(ch-top-bottom)/3;ctx.beginPath();ctx.moveTo(left,y);ctx.lineTo(cw-right,y);ctx.stroke();} const line=(key,max,color)=>{ctx.strokeStyle=color;ctx.lineWidth=2;ctx.beginPath();history.forEach((item,index)=>{const x=left+index*(cw-left-right)/Math.max(1,history.length-1);const y=ch-bottom-((Number(item[key])||0)/max)*(ch-top-bottom);index?ctx.lineTo(x,y):ctx.moveTo(x,y)});ctx.stroke();}; line('events',eventsMax,'#1ea9ff');line('anomalies',anomaliesMax,'#ff5656'); ctx.font='11px Arial';ctx.fillStyle='#1ea9ff';ctx.fillText(`Events max ${eventsMax}`,4,14);ctx.fillStyle='#ff5656';ctx.textAlign='right';ctx.fillText(`Anomalies max ${anomaliesMax}`,cw-4,14);ctx.textAlign='left'; }
+function drawCorrelationGraph(correlations) { const canvas=document.getElementById('correlationGraph'), ctx=canvas.getContext('2d'), ratio=window.devicePixelRatio||1, cw=canvas.clientWidth,ch=canvas.clientHeight; canvas.width=cw*ratio; canvas.height=ch*ratio; ctx.scale(ratio,ratio); ctx.clearRect(0,0,cw,ch); const items=(correlations||[]).slice(0,8); if (!items.length) { ctx.fillStyle='#91abc4'; ctx.font='14px Arial'; ctx.fillText('No multi-stream entities in the current analysis window.', 16, 28); return; } const streams=[...new Set(items.flatMap(item=>item.streams||[]))].slice(0,8); const entityPoint=(index,total)=>({x:Math.max(86,cw*.25),y:42+(index+0.5)*Math.max(36,(ch-84)/total)}); const streamPoint=(index,total)=>({x:Math.min(cw-96,cw*.75),y:42+(index+0.5)*Math.max(36,(ch-84)/total)}); const streamPositions=Object.fromEntries(streams.map((name,index)=>[name,streamPoint(index,streams.length)])); const short=value=>String(value).length>22?`${String(value).slice(0,19)}...`:String(value); items.forEach((item,index)=>{const point=entityPoint(index,items.length); (item.streams||[]).filter(name=>streamPositions[name]).forEach(name=>{const target=streamPositions[name]; ctx.strokeStyle=item.security_events?'#e66b6b':'#347fae'; ctx.lineWidth=Math.min(5,1+Number(item.security_events||0)); ctx.beginPath(); ctx.moveTo(point.x,point.y); ctx.lineTo(target.x,target.y); ctx.stroke();});}); items.forEach((item,index)=>{const point=entityPoint(index,items.length); ctx.fillStyle='#1388cc'; ctx.beginPath(); ctx.arc(point.x,point.y,12,0,Math.PI*2); ctx.fill(); ctx.fillStyle='#d9e8f7'; ctx.font='12px Arial'; ctx.textAlign='right'; ctx.fillText(short(item.entity||item.source_ip),point.x-18,point.y+4);}); streams.forEach((name,index)=>{const point=streamPositions[name]; ctx.fillStyle='#218957'; ctx.fillRect(point.x-10,point.y-10,20,20); ctx.fillStyle='#d9e8f7'; ctx.font='12px Arial'; ctx.textAlign='left'; ctx.fillText(short(name),point.x+16,point.y+4);}); ctx.textAlign='left'; }
 async function refresh() {
   const openDetails = new Set([...document.querySelectorAll('details[open][data-detail-id]')].map(item => item.dataset.detailId));
   const res = await fetch('/api/status', {cache: 'no-store'});
@@ -105,6 +112,8 @@ async function refresh() {
   const mcp = (data.capabilities || {}).graylog_mcp || {};
   const configuration = data.configuration || {};
   drawTrend(data.history || []);
+  drawCorrelationGraph(data.cross_source_correlations || []);
+  document.getElementById('correlationGraphInfo').textContent = `${(data.cross_source_correlations || []).length} entities correlated across enabled streams. Blue entities connect to green stream nodes; red edges carry security-event activity.`;
   document.getElementById('stamp').textContent = data.generated_at ? `Updated ${new Date(data.generated_at * 1000).toLocaleString()}` : 'Waiting for monitor data';
   document.getElementById('metrics').innerHTML = [
     metric('Total events', s.total || 0),
@@ -142,7 +151,7 @@ async function refresh() {
       return esc(`${rate}; dst ports: ${e.distinct_dst_ports || 0}; src ports: ${e.distinct_src_ports || 0}; hitcount: ${e.hitcount_total || 0}`);
     }},
     {label:'Reasons', render:r => esc((r.reasons || []).join('; '))}
-  ]);
+  ], 'anomalies');
   document.getElementById('recommendations').innerHTML = table(data.recommendations || [], [
     {label:'Subject', key:'subject'},
     {label:'Score', key:'score'},
@@ -151,27 +160,27 @@ async function refresh() {
     {label:'Recommendation', key:'recommendation'},
     {label:'Policies', render:r => esc((r.related_policy_ids || []).join(', '))},
     {label:'Services', render:r => esc((r.related_services || []).join(', '))}
-  ]);
+  ], 'recommendations');
   document.getElementById('incidents').innerHTML = table(data.incidents || [], [
-    {label:'Entity', render:r => esc(`${r.entity} (${r.entity_type || 'entity'})`)},
+    {label:'Entity', key:'entity', render:r => esc(`${r.entity} (${r.entity_type || 'entity'})`)},
     {label:'Score', key:'score'},
     {label:'Severity', render:r => `<span class="sev-${esc(r.severity)}">${esc(r.severity)}</span>`},
     {label:'Streams', render:r => esc((r.correlated_streams || []).join(', ') || 'single stream')},
     {label:'Evidence', render:r => esc((r.evidence || []).join('; '))},
     {label:'Timeline', render:r => { const rows=(r.timeline||[]).map(item => esc(`${item.timestamp || ''} | ${item.stream || ''} | ${item.action || ''} | ${item.destination || ''} | ${item.context || item.message || ''}`)).join('<br>'); const id=`incident:${r.entity}:${r.first_seen || ''}`; return rows ? `<details data-detail-id="${esc(id)}"><summary>${esc(`${r.first_seen || '-'} to ${r.last_seen || '-'}`)}</summary><p>${rows}</p></details>` : '-'; }}
-  ]);
+  ], 'incidents');
   document.getElementById('blocks').innerHTML = table(data.block_candidates || [], [
     {label:'Source', key:'src_ip'},
     {label:'Score', key:'score'},
     {label:'Reasons', render:r => esc((r.reasons || []).join('; '))}
-  ]);
+  ], 'blocks');
   const reputationRows = Object.entries(data.reputation || {}).map(([ip, intel]) => ({ip, ...intel}));
   const relatedRows = (data.cross_source_correlations || []).flatMap(correlation => (correlation.samples || []).map(sample => ({entity: correlation.entity || correlation.source_ip, source_ip: correlation.source_ip, ...sample})));
   const streamTitles = Object.fromEntries((configuration.graylog_streams || []).map(item => [item.id, item.title || item.id]));
   const fieldRows = Object.entries(data.field_deviations || {}).flatMap(([entity, deviations]) => (deviations || []).map(item => ({entity, stream_title: streamTitles[item.stream_id] || item.stream_id, ...item})));
   document.getElementById('fieldDeviations').innerHTML = table(fieldRows, [
     {label:'Entity', key:'entity'}, {label:'Stream', key:'stream_title'}, {label:'Field', key:'field'}, {label:'Score', key:'score'}, {label:'Review', render:r => esc(r.feedback || 'unreviewed')}, {label:'Evidence', render:r => { const summary=esc(`${r.reason}; current ${r.current ?? '-'} vs baseline ${r.baseline ?? '-'}; values: ${(r.sample_values || []).join(', ') || '-'}`); const events=(r.sample_events || []).map(item => esc(`${item.timestamp} | ${item.source} -> ${item.destination} | ${item.action} ${item.service} | ${item.value} | ${item.message}`)).join('<br>'); const id=`deviation:${r.entity}:${r.stream_id}:${r.field}:${r.value || ''}`; return events ? `<details data-detail-id="${esc(id)}"><summary>${summary}</summary><p>${events}</p></details>` : summary; }}, {label:'Review action', render:r => `<div class="review-actions"><button class="feedback" data-status="expected" data-entity="${esc(r.entity)}" data-stream="${esc(r.stream_id)}" data-field="${esc(r.field)}" data-value="${esc(r.value || '')}">Mark expected</button><button class="feedback" data-status="false_positive" data-entity="${esc(r.entity)}" data-stream="${esc(r.stream_id)}" data-field="${esc(r.field)}" data-value="${esc(r.value || '')}">Mark false positive</button><button class="feedback" data-status="confirmed" data-entity="${esc(r.entity)}" data-stream="${esc(r.stream_id)}" data-field="${esc(r.field)}" data-value="${esc(r.value || '')}">Mark confirmed</button></div>`}
-  ]);
+  ], 'field-deviations');
   document.querySelectorAll('.feedback').forEach(button => button.addEventListener('click', async () => {
     const note = prompt('Review note (optional):') || '';
     const days = prompt('Expiry in days (0 = no expiry):', '0') || '0';
@@ -180,10 +189,10 @@ async function refresh() {
     refresh();
   }));
   document.getElementById('relatedActivity').innerHTML = table(relatedRows, [
-    {label:'Entity', render:r => esc(r.entity || r.source_ip)}, {label:'Stream', key:'stream'}, {label:'Time', key:'timestamp'},
+    {label:'Entity', key:'entity', render:r => esc(r.entity || r.source_ip)}, {label:'Stream', key:'stream'}, {label:'Time', key:'timestamp'},
     {label:'Type', key:'type'}, {label:'Action', key:'action'}, {label:'Severity', key:'severity'},
     {label:'Destination', key:'destination'}, {label:'Service', key:'service'}, {label:'Context', key:'context'}
-  ]);
+  ], 'related-activity');
   document.getElementById('reputation').innerHTML = table(reputationRows, [
     {label:'IP', key:'ip'},
     {label:'Provider', key:'provider'},
@@ -191,22 +200,22 @@ async function refresh() {
     {label:'Score', key:'score'},
     {label:'Malicious', key:'malicious'},
     {label:'Suspicious', key:'suspicious'}
-  ]);
+  ], 'reputation');
   document.getElementById('policies').innerHTML = table(data.policy_findings || [], [
     {label:'Severity', render:r => `<span class="sev-${esc(r.severity)}">${esc(r.severity)}</span>`},
     {label:'Reference', key:'reference'},
     {label:'Title', key:'title'},
     {label:'Detail', key:'detail'}
-  ]);
+  ], 'policies');
   const d = data.diagnostics || {};
   const context = data.event_context || {};
   const quality = data.data_quality || {};
   const profileReadiness = (data.profile_readiness || []).map(item => ({...item, stream_title: streamTitles[item.stream_id] || item.stream_id}));
   const correlations = data.cross_source_correlations || [];
   document.getElementById('diagnostics').innerHTML =
-    '<h3>Cross-Source Correlations</h3>' + table(correlations, [{label:'Entity', render:r => esc(`${r.entity || r.source_ip} (${r.entity_type || 'ip'})`)}, {label:'Streams', render:r => esc((r.streams || []).join(', '))}, {label:'Events', key:'events'}, {label:'Security Events', key:'security_events'}]) +
+    '<h3>Cross-Source Correlations</h3>' + table(correlations, [{label:'Entity', key:'entity', render:r => esc(`${r.entity || r.source_ip} (${r.entity_type || 'ip'})`)}, {label:'Streams', render:r => esc((r.streams || []).join(', '))}, {label:'Events', key:'events'}, {label:'Security Events', key:'security_events'}], 'correlations') +
-    '<h3>Entities</h3>' + table(context.source_profiles || [], [{label:'Entity', key:'entity'}, {label:'Events', key:'events'}, {label:'UTM', key:'utm_events'}, {label:'Deny', key:'deny_or_threat_actions'}, {label:'Destinations', key:'distinct_destinations'}, {label:'Actions', render:r => esc((r.top_actions || []).join(', '))}]) +
+    '<h3>Entities</h3>' + table(context.source_profiles || [], [{label:'Entity', key:'entity'}, {label:'Events', key:'events'}, {label:'UTM', key:'utm_events'}, {label:'Deny', key:'deny_or_threat_actions'}, {label:'Destinations', key:'distinct_destinations'}, {label:'Actions', render:r => esc((r.top_actions || []).join(', '))}], 'entities') +
-    '<h3>Profile Baseline Readiness</h3>' + table(profileReadiness, [{label:'Stream', key:'stream_title'}, {label:'Field', key:'field'}, {label:'Buckets', key:'buckets'}, {label:'Ready', render:r => r.ready ? 'ready' : 'learning'}]) +
+    '<h3>Profile Baseline Readiness</h3>' + table(profileReadiness, [{label:'Stream', key:'stream_title'}, {label:'Field', key:'field'}, {label:'Buckets', key:'buckets'}, {label:'Ready', key:'ready', render:r => r.ready ? 'ready' : 'learning'}], 'profile-readiness') +
     '<h3>Data Quality</h3>' + table([quality], [{label:'Events', key:'events'}, {label:'Timestamp coverage', render:r => `${r.timestamp_coverage || 0}%`}, {label:'Source coverage', render:r => `${r.source_coverage || 0}%`}, {label:'Truncated streams', render:r => esc((r.truncated_streams || []).join(', ') || 'none')}]) +
     '<h3>Security Event Samples</h3>' + table(context.security_event_samples || [], [{label:'Entity', key:'entity'}, {label:'Type', key:'type'}, {label:'Action', key:'action'}, {label:'Severity', key:'severity'}, {label:'Destination', key:'dst'}, {label:'Service', key:'service'}]) +
     '<h3>Top Sources</h3>' + table(d.top_source_ips || [], [{label:'Value', key:'value'}, {label:'Count', key:'count'}]) +
@@ -217,6 +226,7 @@ async function refresh() {
     '<h3>Top Services</h3>' + table(d.top_services || [], [{label:'Value', key:'value'}, {label:'Count', key:'count'}]) +
     '<h3>Local-in Failures</h3>' + table(d.local_in_failures || [], [{label:'Source', key:'src_ip'}, {label:'Service', key:'service'}, {label:'Policy', key:'policy'}, {label:'Count', key:'count'}]);
   document.querySelectorAll('details[data-detail-id]').forEach(item => { if (openDetails.has(item.dataset.detailId)) item.open = true; });
+  document.querySelectorAll('[data-sort-table]').forEach(button => button.addEventListener('click', () => { const current=tableSort[button.dataset.sortTable]; tableSort[button.dataset.sortTable]={key:button.dataset.sortKey,direction:current && current.key===button.dataset.sortKey ? -current.direction : 1}; refresh(); }));
 }
 async function loadSettings() {
   const config = await (await fetch('/api/config', {cache: 'no-store'})).json();
@@ -243,6 +253,7 @@ document.getElementById('loadFields').addEventListener('click', async () => {
   if (!selected) { document.getElementById('fieldPicker').textContent = 'Load streams, tick one stream, then load its fields.'; return; }
   const payload = await (await fetch(`/api/graylog/fields?stream_id=${encodeURIComponent(selected.dataset.id)}`)).json();
   const profile = (window.streamProfiles || []).find(item => item.stream_id === selected.dataset.id) || {};
+  document.querySelector('[name="profile_name"]').value = profile.name || `${selected.dataset.title} profile`;
   const rows = (payload.fields || []).map(field => { const name=field.name||field.field, type=(field.type||{}).type||'', props=(field.type||{}).properties||[]; return `<div class="field-row"><code>${esc(name)}</code><span>${esc(type)}</span><span>${esc(props.join(', '))}</span><div class="field-controls"><label><input type="radio" name="profile_entity" value="${esc(name)}" ${profile.entity_field===name?'checked':''}> Entity</label><label><input type="radio" name="profile_timestamp" value="${esc(name)}" ${profile.timestamp_field===name?'checked':''}> Time</label>${props.includes('enumerable')?`<label><input class="profile-categorical" type="checkbox" value="${esc(name)}" ${(profile.categorical_fields||[]).includes(name)?'checked':''}> Categorical</label>`:''}${props.includes('numeric')?`<label><input class="profile-numeric" type="checkbox" value="${esc(name)}" ${(profile.numeric_fields||[]).includes(name)?'checked':''}> Numeric</label>`:''}</div></div>`; }).join('');
   document.getElementById('fieldPicker').innerHTML = rows ? `<div class="field-header"><span>Field</span><span>Type</span><span>Capabilities</span><span>Use In Profile</span></div>${rows}` : esc(payload.error || 'No fields found.');
   window.activeProfileStream = selected.dataset.id;
@@ -254,7 +265,7 @@ document.getElementById('settingsForm').addEventListener('submit', async event =
   values.llm_enabled = form.elements.llm_enabled.checked;
   values.threat_intel_enabled = form.elements.threat_intel_enabled.checked;
   values.graylog_streams = [...document.querySelectorAll('.graylog-stream')].map(item => ({id:item.dataset.id, title:item.dataset.title, enabled:item.checked}));
-  if (window.activeProfileStream) { const profile={stream_id:window.activeProfileStream,entity_field:form.querySelector('[name="profile_entity"]:checked')?.value||'',timestamp_field:form.querySelector('[name="profile_timestamp"]:checked')?.value||'timestamp',categorical_fields:[...form.querySelectorAll('.profile-categorical:checked')].map(item=>item.value),numeric_fields:[...form.querySelectorAll('.profile-numeric:checked')].map(item=>item.value)}; values.graylog_stream_profiles=[...(window.streamProfiles||[]).filter(item=>item.stream_id!==profile.stream_id),profile]; }
+  if (window.activeProfileStream) { const selectedStream=document.querySelector('.graylog-stream:checked'); const profile={stream_id:window.activeProfileStream,name:form.elements.profile_name.value.trim() || `${selectedStream?.dataset.title || window.activeProfileStream} profile`,entity_field:form.querySelector('[name="profile_entity"]:checked')?.value||'',timestamp_field:form.querySelector('[name="profile_timestamp"]:checked')?.value||'timestamp',categorical_fields:[...form.querySelectorAll('.profile-categorical:checked')].map(item=>item.value),numeric_fields:[...form.querySelectorAll('.profile-numeric:checked')].map(item=>item.value)}; values.graylog_stream_profiles=[...(window.streamProfiles||[]).filter(item=>item.stream_id!==profile.stream_id),profile]; }
   const response = await fetch('/api/config', {method:'POST', headers:{'Content-Type':'application/json'}, body:JSON.stringify(values)});
   document.getElementById('settingsResult').textContent = response.ok ? 'Saved. Monitor applies supported settings on its next cycle.' : 'Could not save configuration.';
   if (response.ok) loadSettings();

src/fgai/monitor.py

@@ -130,7 +130,7 @@ def build_status(
         "baseline": {"enabled": bool(baseline), "sources_ready": len(profiles), "new_events_recorded": baseline_events, "profile_fields_recorded": profile_baseline_fields},
         "capabilities": {"threat_intel": threat_intel_status, "graylog_mcp": mcp_status},
         "configuration": runtime_config,
-        "stream_profiles": [{"stream_id": item.stream_id, "entity_field": item.entity_field, "timestamp_field": item.timestamp_field, "categorical_fields": list(item.categorical_fields), "numeric_fields": list(item.numeric_fields)} for item in stream_profiles.values()],
+        "stream_profiles": [{"stream_id": item.stream_id, "name": item.name, "entity_field": item.entity_field, "timestamp_field": item.timestamp_field, "categorical_fields": list(item.categorical_fields), "numeric_fields": list(item.numeric_fields)} for item in stream_profiles.values()],
         "profile_readiness": profile_readiness,
         "diagnostics": {
             "top_source_ips": top_field_values(events, "srcip", limit=10),

src/fgai/stream_profiles.py

@@ -6,6 +6,7 @@ from dataclasses import dataclass
 @dataclass(frozen=True)
 class StreamProfile:
     stream_id: str
+    name: str
     entity_field: str
     timestamp_field: str
     categorical_fields: tuple[str, ...] = ()
@@ -23,7 +24,7 @@ def parse_profiles(value: object) -> dict[str, StreamProfile]:
         if not stream_id or not entity:
             continue
         profiles[stream_id] = StreamProfile(
-            stream_id, entity, timestamp,
+            stream_id, str(item.get("name", "")).strip() or stream_id, entity, timestamp,
             tuple(str(field) for field in item.get("categorical_fields", []) if field),
             tuple(str(field) for field in item.get("numeric_fields", []) if field),
         )

tests/test_stream_profiles.py

@@ -5,6 +5,7 @@ from fgai.stream_profiles import parse_profiles
 
 class StreamProfileTests(unittest.TestCase):
     def test_parses_field_level_profile(self):
-        profiles = parse_profiles([{"stream_id": "dns", "entity_field": "IP", "categorical_fields": ["QH"], "numeric_fields": ["Elapsed"]}])
+        profiles = parse_profiles([{"stream_id": "dns", "name": "DNS client behavior", "entity_field": "IP", "categorical_fields": ["QH"], "numeric_fields": ["Elapsed"]}])
+        self.assertEqual(profiles["dns"].name, "DNS client behavior")
         self.assertEqual(profiles["dns"].entity_field, "IP")
         self.assertEqual(profiles["dns"].numeric_fields, ("Elapsed",))