byrsapty
a24defe596
The alert5 description and both READMEs said "~4,600-4,800 msgs/10min" as the calibrated baseline - a transcription error, a digit short of the real figure (~46,000-48,000/10min, matching the ~278k/hour number right next to it, which was correct). The configured threshold (150,000) was actually computed from the correct number, so no behavior changed - only the description text was wrong. Also documents a real event from today: EX-NAS-1-2's first-ever Vector startup tripped this alert (430,982 msgs/10min) - Vector's file source replays existing log content from the beginning with no checkpoint yet, so a busy log's backlog shows up as a single burst rather than a real ongoing issue. Confirmed the sample messages were all routine DHCP/session churn timestamped around the same moment. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
21 lines
1.4 KiB
JSON
21 lines
1.4 KiB
JSON
{
|
|
"title": "WARNING: abnormal message volume from one server",
|
|
"description": "A single server is sending far more syslog messages than its established baseline - possible log loop, misconfiguration, or a genuine incident generating excessive events. Left unchecked this can fill the retention window's disk budget early. Threshold calibrated live on 2026-07-22 against real accel-ppp/RADIUS traffic: the one active NAS server was steadily sending ~46,000-48,000 msgs/10min (~278k/hour) under totally normal load, so the threshold is set to roughly 3x that baseline. Note: a server's first-ever Vector startup can trigger this once, harmlessly - Vector's file source reads existing log content from the start on a fresh run, producing a one-time burst of backlog rather than a real ongoing issue.",
|
|
"priority": 2,
|
|
"alert": true,
|
|
"config": {
|
|
"type": "aggregation-v1",
|
|
"query": "*",
|
|
"streams": ["__SERVERS_STREAM_ID__"],
|
|
"group_by": ["gl2_remote_ip"],
|
|
"series": [{"type": "count", "id": "count-", "field": null}],
|
|
"conditions": {"expression": {"expr": ">", "left": {"expr": "number-ref", "ref": "count-"}, "right": {"expr": "number", "value": 150000.0}}},
|
|
"search_within_ms": 600000,
|
|
"execute_every_ms": 600000,
|
|
"event_limit": 50
|
|
},
|
|
"field_spec": {},
|
|
"key_spec": [],
|
|
"notification_settings": {"grace_period_ms": 1800000, "backlog_size": 5},
|
|
"notifications": [{"notification_id": "__DISCORD_NOTIFICATION_ID__"}]
|
|
}
|