8 new rules (vendor=a10) wired into Network Equipment Parsing: LSN
TCP/Session/ICMP per-user quota exceeded (critical - real service
impact, drops new connections for that subscriber), BGP-4-MAXPFX
prefix-limit warning, and admin session open/close/timeout/auth-success
(aXAPI and CLI both covered by one pattern each).
Unlike the CSV-report-derived rules, these are built directly from
real captured A10 log output the user provided, so confidence is
higher - closer to the accel-ppp rules' provenance. Multi-entry LSN
lines (several 'ip(count)' pairs in one quota-exceeded message) only
have their first pair extracted into fields; the full list stays in
the raw message.
Not live-verified - the user is bringing the target system up
themselves this time rather than through the test container used
earlier in this branch of work.
alert8: any-match critical hardware/power alarms (dying_gasp, ONU
config/HAL/info failures, PSU/memory/ASIC faults, cold start) - same
fire-on-every-occurrence philosophy as alert4_juniper_chassis_alarm.
alert9: repeated-warning threshold on severity_tag:warning, grouped by
device+event_type, >5 in 5min - same philosophy as
alert7_accelppp_repeated_error_warn (a single occurrence can be
transient; repetition means a real problem). Covers signal degradation,
low optical power, BGP/LACP flapping, DDoS/SNMP violations, etc.
Deliberately excludes purely informational event_types (logins,
config_saved, ONU registration, alarm-cleared/recovery events) to avoid
noise.
Not yet verified live - same caveat as the rule27-84 commit, the test
container is still unreachable.
Closes the parsing gap the README explicitly called out (no D-Link
parsing, no ZTE ONU alarms) plus adds BDCOM GPON and expands Juniper
coverage (DDoS, PSU/memory/ASIC hardware faults, LACP/BGP/SNMP, config
commit). 58 new rules across 5 vendors, wired into Network Equipment
Parsing's stage 0 ahead of the generic_critical_severity fallback.
Where the same real-world event is reported by multiple vendors
(dying_gasp, onu_offline, optical_low_power, cli_login/cli_logout,
config_saved, interface_link_state, lag_state_change), rules share one
event_type value so dashboards can aggregate across vendors, same
normalization approach as accelppp_interface.
Built directly from the user's CSV signature report, not from real
device log samples - each rule's description says so explicitly. `when`
conditions use plain substring/contains matching on the report's own
pattern text to keep classification robust; regex field extraction is
only added where the source format is unambiguous. Passed offline
checks (JSON validity, every pipeline-referenced rule resolves to a
file, all regex patterns compile). Live compilation against a running
Graylog instance - which caught 2 real bugs during the dashboard/stream
fixes earlier this session - could NOT be completed: the test container
went unreachable mid-session. Re-run install-graylog.sh once it's back
up to confirm these compile before relying on them.
Both only checked title existence, so editing a stream's rules or a
dashboard's widgets under the same title silently skipped the update
forever. step_dashboard() now diffs the search's queries content
(delete+recreate view+search on change - no subscribers/history to lose,
unlike alerts or streams); step_streams() diffs rules individually by
(field,type,value,inverted) and PUT-updates metadata in place, since
delete+recreating a stream would briefly break live message routing.
Confirmed live on a local test deployment: a false "changed" positive
from search_types list-reordering and from search_servers.json's own
query-format inconsistency (bare string vs the {query_string,type} object
Graylog actually stores) had to be fixed before step_dashboard() settled
into a stable no-op; step_streams() was verified with a full inert
rule-add/remove and a metadata-change round-trip against the live API.
Confirmed live: "CRITICAL: unrecognized critical-severity syslog" (the
only alert with a 2-element streams list) kept reporting "config changed"
every single run even with no actual edit, because a plain dict/list ==
comparison in Python is order-sensitive and Graylog doesn't necessarily
return the streams array in the same order it was submitted in. List
values are now sorted before comparing on both sides, since list order
was never semantically meaningful here. Verified locally against a
same-elements-different-order case (now matches) and a genuinely-changed
case (still correctly detected as different) before pushing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The previous fix compared the whole live 'config' object against the
whole desired one - but Graylog's GET response fills in extra defaults
(query_parameters, filters, use_cron_scheduling, cron_expression,
cron_timezone, ...) that never appear in our alert JSON files. That made
every single alert compare as "different" on every single run, not just
the one actually edited - confirmed live: a re-run PUT-updated all 7
alerts when only alert7's query had changed.
Fixed by comparing only the keys our own files actually author (project
the live config down to just those keys before comparing), and switched
from interpolating JSON into python -c string literals to writing to temp
files and reading them - the former is fragile the moment a value contains
a quote or backslash.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same class of bug just fixed in step_pipeline_rules()/step_pipelines():
title-only existence check meant an alert's config (query, threshold,
group_by) could never actually change on a re-run once created. Confirmed
live: alert7's query was corrected in git (vendor:accel-ppp ->
vendor:accel-ppp AND event_type:*) but a subsequent re-run still reported
"already exists" with the old, buggy query untouched. Now compares by the
'config' object (not just title) and PUT-updates + re-enables the
schedule if it differs, same pattern as the other two fixes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- step_pipeline_rules() and step_pipelines() only ever created new
rules/pipelines by title, never updated existing ones whose content
changed under the same title. Confirmed live this session: 5 new rules
were added to rules/*.json and created fine in Graylog, but "Servers
Parsing"'s stage list was never updated to actually call them, since
the pipeline already existed. Both functions now compare by 'source'
content and PUT-update if it differs, matching the pattern already used
elsewhere (e.g. step_inputs()'s timezone self-heal).
- alert7's query (vendor:accel-ppp) was too broad: it also matches
routine traffic tagged only by the generic accelppp_interface_tag
fallback rule, which sets vendor but never event_type - producing false
"(Empty Value)" group_by matches on ordinary volume. Confirmed live.
Fixed to vendor:accel-ppp AND event_type:* - severity_tag was
considered instead but rejected, since accelppp_router_address_error
(the rule that inspired this alert) never set it either.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Analyzed a real 1GB accel-ppp log (2026-07-23): only 2,819 of its lines
were error:/warn:, and one pattern - "can't determine router address" -
repeated 2,746 times over ~4 hours for two specific subscriber interfaces
before self-resolving, completely unalerted since no alert covered it.
- New pipeline rules for 3 previously-unclassified real message types
(radius:dm_coa session not found, mac change detected, dhcpv4 short
packet) plus a text-based fallback pair (accelppp_unclassified_error/
warn) for anything not yet specifically classified - needed because
Vector-shipped accel-ppp lines have no real syslog PRI header, so the
numeric-severity generic_critical_severity rule never fires for this
source.
- New alert7: group by gl2_remote_ip + event_type, fires on >5 occurrences
in 5 minutes - low enough to have caught the real incident within its
first cycle, high enough to tolerate a single transient warning.
- vector-accel-ppp-setup.md gained a "keep only error/warn" filter option
(Step 2b), explicitly documented as a deliberate tradeoff: it also drops
RADIUS accounting, so the Servers & Sessions dashboard and session
correlation go empty for any server that applies it. Both the
with-filter and without-filter full configs are included so the choice
is per-server, not global.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Dropping every "[DHCPv4 " line unconditionally would also swallow a
genuine DHCP-side problem (pool exhaustion, lease timeout) if accel-ppp
ever logs one at error/warn level. Verified the refined condition against
sample lines before writing it: plain info-level DHCP still drops, RADIUS
lines are unaffected, and DHCP lines tagged error:/warn: now pass through.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Motivated by live capacity math: 2 active NAS servers measured at ~634
bytes/document in Graylog and ~75 msg/s each; projected to 10+ servers of
similar profile that's ~38.5 GiB/day, which the current 50GB disk / 14-21
day retention can't hold. DHCP request/response chatter (Discover/Request/
Offer/Ack) is the bulk of that volume and isn't needed for RADIUS/
accounting analytics, so dropping it via a Vector `filter` transform
(before the packet ever leaves the NAS server) is the cheapest place to
cut it - no wasted network/disk/CPU on data nobody queries.
Also cross-referenced two things diagnosed live earlier: the
read_from: beginning cold-start replay burst, and where to look if a
filtered-out message type stops appearing (intentional, not a bug).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The alert5 description and both READMEs said "~4,600-4,800 msgs/10min" as
the calibrated baseline - a transcription error, a digit short of the real
figure (~46,000-48,000/10min, matching the ~278k/hour number right next to
it, which was correct). The configured threshold (150,000) was actually
computed from the correct number, so no behavior changed - only the
description text was wrong.
Also documents a real event from today: EX-NAS-1-2's first-ever Vector
startup tripped this alert (430,982 msgs/10min) - Vector's file source
replays existing log content from the beginning with no checkpoint yet, so
a busy log's backlog shows up as a single burst rather than a real ongoing
issue. Confirmed the sample messages were all routine DHCP/session churn
timestamped around the same moment.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- step_compose_files() now uses GRAYLOG_ADMIN_PASSWORD as the actual admin
password on a fresh install if set, instead of always generating a
random one. Solves the CI secret-staleness problem at the root: pin a
password once and it's correct both at creation time and on every later
re-run, instead of a fresh install randomly generating a password the
stored secret then has to be manually kept in sync with.
- deploy-from-scratch.yml gained a `cores` workflow_dispatch input
(default 4), passed through to create-graylog-lxc.sh's --cores flag.
Only takes effect when the container is actually created fresh, same as
every other --cores usage in this project.
- Documented both in README.md/README.uk.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
deploy.yml already forwarded this secret; deploy-from-scratch.yml didn't,
so a re-run against an already-initialized Graylog (one-time credentials
file already consumed by an earlier successful run) died at
resolve_admin_password with "Cannot find admin password" - confirmed
live.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- New "Ways to run this" comparison table up top
- Step 9 split into 9a (in-container runner) / 9b (host-level runner,
with the RUNNER_USER=claude-deploy warning and single-line paste-safe
command) / 9c (secrets) / removal instructions
- CI/CD section split into deploy.yml and deploy-from-scratch.yml
subsections, each explaining its own runner/scope, plus what's common
to both
- File layout updated with every script and workflow file added this
session (bootstrap-host.sh, cleanup-host.sh, fix-lxc-apparmor.sh,
setup-forgejo-runner.sh, alerts/, dashboards/, .forgejo/workflows/)
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Introduced by an earlier edit to .admin_credentials_ONE_TIME's own
instructional text ("...you've stored the password: rm <path>"), which
itself contains "password: " - the same substring resolve_admin_password()
greps for. grep -oP matched BOTH occurrences; command substitution joined
them with a real newline, producing a corrupted two-line "password" that
never matched .env's GRAYLOG_ROOT_PASSWORD_SHA2, causing every gcurl call
to silently 401 and wait_for_api_ready to time out no matter how generous
the timeout was (confirmed by re-verifying live: the real password,
extracted correctly, hashes to exactly what .env already has).
Fixed both ends: reworded the instructional text to not repeat "password:",
and hardened the regex itself (^Graylog admin password: anchor, grep -m1)
so a future wording change can't reintroduce the same class of bug.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed live: on a CI-triggered run, the REST API genuinely took longer
than 60s to accept authenticated requests even though the container had
already reported healthy - verified after the fact that both the API and
the admin credentials were fine, this was purely insufficient margin
(likely due to concurrent load during the run), not a logic bug. 180s
matches the same order of magnitude as the existing 300s docker-health
wait elsewhere in this script.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
"the repo's Actions" inside FORGEJO_RUNNER_TOKEN's :? error message threw
off bash's parser even though the whole expression sits inside double
quotes - confirmed live via bisection (bash -n on truncated line ranges
pinpointed line 20 exactly, "unexpected EOF while looking for matching
`''`"). Single quotes inside \${VAR:?message} aren't neutralized by the
outer double quotes the way they would be in a plain string. Reworded to
avoid the apostrophe entirely.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Docker's healthcheck can report graylog-server "healthy" a few seconds
before the REST API is actually ready to serve authenticated requests -
confirmed live: the first gcurl call (step_index_retention) intermittently
got an empty response body, crashing the downstream `python3 -c
"json.load(sys.stdin)"` with "Expecting value: line 1 column 1".
Adds wait_for_api_ready(), polling the same endpoint step_index_retention
already needs (up to 60s) before proceeding, mirroring the existing
retry-loop pattern already used for docker compose pull.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
setup-forgejo-runner.sh gained RUNNER_LABEL/RUNNER_DIR/SERVICE_NAME/
RUNNER_USER params so the same script can register either kind of runner:
- inside the container (unchanged defaults, root - already scoped to just
that container)
- on the Proxmox host itself, where RUNNER_USER=claude-deploy is required:
a root-owned systemd service with no User= would hand every CI job
unrestricted root on the host, defeating the whole point of
claude-deploy's narrowly-scoped sudoers rules.
deploy-from-scratch.yml runs create-graylog-lxc.sh on the host-level
runner. Deliberately does NOT run pct destroy - that stays a manual,
deliberate human step. The idempotent create+install path is safe to
trigger any time: repairs an existing container in place, or fully
recreates one if it was destroyed beforehand.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- export LC_ALL=C.UTF-8 LANG=C.UTF-8 at the top of the script: the
container inherits LANG=en_US.UTF-8 from pct exec's calling shell but
never generates that locale, so every apt-get call printed "Setting
locale failed" warnings from perl/apt-listchanges. C.UTF-8 is glibc-
builtin, no locale-gen needed.
- The final summary now prints the generated admin user/password directly
and deletes .admin_credentials_ONE_TIME right after, instead of just
pointing at the file and leaving it for the operator to read and clean
up by hand.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Mirrors bootstrap-host.sh in reverse: removes the sudoers rule and the
fix-lxc-apparmor.sh script it installed, so the elevated grant only stands
for the duration of an active deployment instead of indefinitely.
create-graylog-lxc.sh already degrades cleanly to its manual fallback when
the automation isn't present, so this is a safe no-op-adjacent revoke.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds bootstrap-host.sh (one-time, run as root on a fresh Proxmox host) and
fix-lxc-apparmor.sh, the fixed-content script it installs. The sudoers
rule it wires up only ever invokes that one root-owned script with a VMID
argument - deliberately not a broader rule like `tee -a <conf>` or
`sh -c '...'`, since those only restrict the command's own argv, not
stdin/heredoc content, letting the caller write arbitrary lines to any
200-299 container's config instead of just this one fixed line.
create-graylog-lxc.sh now tries `sudo -n fix-lxc-apparmor.sh` first and
falls back to the existing manual instructions if that sudoers rule isn't
present yet - fully backward compatible with hosts that haven't run
bootstrap-host.sh.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The CI runner setup was done by hand this session (download, verify,
register, systemd unit) - script it the same idempotent way as
install-graylog.sh so a fresh deployment can reproduce it instead of
requiring manual SSH archaeology. Wire it into create-graylog-lxc.sh's
payload copy, and add step 9 to both READMEs covering the one-time setup.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Forgejo auto-issues a per-job, repo-scoped secrets.GITHUB_TOKEN (created at
workflow start, destroyed at completion, usable only against this repo);
use it in the clone URL instead of an anonymous HTTPS clone, which would
start failing with 401/403 the moment the repo's visibility changes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
actions/checkout@v4 is a Node.js-based action; the self-hosted runner
executes in host mode directly on the Graylog appliance container, which
has no Node.js and shouldn't need one just to check out a repo. A plain
git clone avoids the dependency entirely.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Two flood-detection alerts (per-source message volume, calibrated live
against real traffic) grouped by gl2_remote_ip
- Session correlation: accelppp_interface fallback tagging plus
radius_session_id/calling_station_id/radius_username extraction, so a
subscriber's full session lifecycle is searchable by one key
- Replace the single combined dashboard with three focused ones (Overview
& Alerts, Network Equipment, Servers & Sessions)
- Propagate GRAYLOG_ROOT_TIMEZONE and IP-in-alerts fixes into the reusable
install script and templates
- Add a Forgejo Actions workflow (manual trigger) that re-runs
install-graylog.sh on a self-hosted runner living in the container,
automating the deploy step this project has done by hand all along
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
