Graylog centralized log server — deployment scripts

Deploys a Debian 12 LXC container on Proxmox VE with a Docker Compose stack (MongoDB + OpenSearch + Graylog) for collecting syslog from network equipment (Juniper, ZTE OLT, D-Link) and servers (RADIUS, accel-ppp).

Quick start

Run on the Proxmox host as claude-deploy (or any user with the same restricted sudo rights on pct/pveam for VMIDs 200-299):

./create-graylog-lxc.sh \
  --ip 10.254.254.202/24 \
  --gw 10.254.254.235 \
  --vlan 1254 \
  --external-uri http://93.171.241.5:9000/ \
  --discord-webhook "https://discord.com/api/webhooks/xxx/yyy"

This is the only command you need. It creates the container, installs Docker, applies the firewall (nftables: only 9000/tcp, 514+1514/udp, 5140/udp open), brings up the stack, creates the Syslog inputs, imports the pipeline rules, and prints the admin credentials at the end.

Both scripts are idempotent — re-running after a failure (or to apply an update) picks up from where it left off instead of duplicating work.

Step-by-step deployment from scratch

0. Prerequisites

• SSH access to the Proxmox host as a user with sudo rights on pct create/set/start/stop/exec/status/list and pveam update/list/download (does not need to be root - this whole toolkit was built and tested under exactly that restricted scope).
• A storage on the Proxmox host with content type vztmpl enabled, holding (or able to download) a debian-12-standard template - default assumed name: local-btrfs. Check with:

  sudo pveam list local-btrfs   # replace with your storage name
  

  If it errors with "storage is disabled" or "does not exist", find the right one (sudo pveam list <name> for candidates) and pass it via --template-storage.
• A storage for the container's root filesystem (default assumed: EX-Ceph) with enough free space for --disk (default 50GB).
• The IP address, gateway, and VLAN tag (if any) for the network the container will live on - i.e. the same network your RADIUS/NAS servers and network equipment can reach.
• If you need remote access to the Web UI from outside that network: a spare external port to DNAT to the container's 9000/tcp (see the "reaching the Web UI from outside" note below - port 9000 collided with an existing ClickHouse listener during initial testing, so don't assume any specific port is free).
• (Optional) A Discord webhook URL, if you want Critical alerts pushed to a channel.

1. Get the scripts onto the Proxmox host

From your workstation:

tar czf - -C /path/to/graylog-deploy . | ssh claude-deploy@<proxmox-host> \
  "mkdir -p ~/graylog-deploy && tar xzf - -C ~/graylog-deploy && chmod +x ~/graylog-deploy/*.sh"

(Or git clone/scp the directory if you keep it in a repo - any method that gets the whole folder, including rules/, pipelines/, streams/, alerts/, onto the host works.)

2. Decide your parameters

Pick, in advance:

You need	Example	Why
A free VMID in 200-299	210	or omit --vmid to auto-pick the first free one
Container IP + CIDR on the management network	10.254.254.220/24	must be reachable by every device that will send syslog
Gateway on that network	10.254.254.235
VLAN tag (if the bridge is trunked)	1254	omit --vlan if untagged
Public URL for the Web UI	http://<public-ip>:<port>/	used both in Graylog's config and in Discord alert links
Discord webhook (optional)	https://discord.com/api/webhooks/.../...	leave unset to skip Discord entirely

Check the IP is actually free first - a duplicate IP on this network silently causes ARP flapping and intermittent, hard-to-diagnose routing weirdness (this happened during initial testing: .201 turned out to already be in use, causing traffic to randomly land on the wrong host). A quick way to check without touching anything:

ssh claude-deploy@<proxmox-host> "sudo /usr/sbin/pct exec <any-running-vmid> -- ping -c2 -W1 <candidate-ip>"

If you get replies, that IP is taken - pick another.

3. Run create-graylog-lxc.sh

ssh claude-deploy@<proxmox-host>
cd ~/graylog-deploy
./create-graylog-lxc.sh \
  --vmid 210 \
  --ip 10.254.254.220/24 \
  --gw 10.254.254.235 \
  --vlan 1254 \
  --external-uri http://<public-ip>:<port>/ \
  --discord-webhook "https://discord.com/api/webhooks/xxx/yyy"

What happens, in order: template check/download → pct create → start the container → wait for network → copy this whole folder into the container at /opt/graylog-deploy/ → run install-graylog.sh inside it, which installs Docker, applies the firewall, brings up MongoDB/OpenSearch/Graylog, waits for it to report healthy, creates the Syslog inputs, imports the pipeline rules/pipelines/streams, and (if a webhook was given) creates the Discord notification and the three alert definitions.

If it stops with an AppArmor error (open sysctl net.ipv4.ip_unprivileged_port_start: permission denied), that's expected on some Proxmox setups and needs one manual step from the host's root account - see "Docker-in-unprivileged-LXC AppArmor block" below. Do that, then just run the exact same command again; everything already done is skipped automatically.

Total time for a clean run: a few minutes, mostly waiting for image pulls and for Graylog to report healthy.

4. Read the admin credentials

ssh claude-deploy@<proxmox-host> "sudo pct exec 210 -- cat /opt/graylog/.admin_credentials_ONE_TIME"

Copy the password into your password manager, then delete the file:

ssh claude-deploy@<proxmox-host> "sudo pct exec 210 -- rm /opt/graylog/.admin_credentials_ONE_TIME"

Log into http://<public-ip>:<port>/ with user admin and that password.

5. Reach the Web UI from outside the management network (if needed)

If the container's IP isn't directly reachable from where you browse, add a DNAT rule on your edge router/firewall for 9000/tcp:

-A PREROUTING -d <public-ip>/32 -p tcp -m tcp --dport <port> -j DNAT --to-destination <container-ip>:9000

Pick a port that's actually free - during initial setup, port 9000 turned out to already be serving ClickHouse on the same public IP, and the response (Port 9000 is for clickhouse-client program...) looked like a Graylog problem for a while before that was found. If you get a suspicious/unexpected response on the port you chose, suspect a pre-existing service on that port before suspecting Graylog.

6. Point real equipment at the server

Source	Port	Notes
Network equipment (switches, OLTs, routers)	514/udp	the standard syslog port - almost no gear lets you pick a different one (logging <host> on a typical switch always uses 514)
Network equipment that can target a custom port	1514/udp	kept as a secondary input, same stream as 514
Servers (RADIUS, accel-ppp, conntrack/kernel messages)	5140/udp

On each device, this is normally a one-line config change (e.g. on a Cisco-like CLI: logging <container-ip>). No Graylog-side config is needed per device - the inputs and streams already listen on all three ports.

7. Verify data is actually arriving

1. Web UI → System → Inputs: each input shows live "Traffic Last Minute". If it stays at 0 after a device should have sent something, the problem is network/firewall, not Graylog - confirm with tcpdump inside the container before touching any Graylog config:

   sudo pct exec 210 -- tcpdump -i eth0 -n udp port 514
   

2. Web UI → Search, widen the time range (top-left), click any message to expand it. If vendor / event_type fields are populated, a pipeline rule matched it. If a message arrives but those fields are empty, it reached Graylog fine but no rule recognizes its format yet - that's a sign a new device/vendor needs a new rule (see "What's still NOT included" below for the process).
3. Web UI → Streams: the "Throughput" column shows live msg/s per stream.

8. (Optional) Trigger a real test alert

See "Verifying a test alert" further down - send one of the known-critical log lines via logger from inside the container and watch it land in Discord within about a minute.

Parameters

All flags have an environment-variable equivalent (see the top of create-graylog-lxc.sh), so you can also export MEMORY_MB=16384 etc. instead of passing flags.

Flag	Default	Notes
--ip	(required)	Static IP + CIDR for the container
--gw	(required)	Gateway IP
--external-uri	(required)	Public URL for the Graylog Web UI (used in GRAYLOG_HTTP_EXTERNAL_URI)
--vmid	first free 200-299
--hostname	graylog
--cores	4
--memory	8192 (MB)
--swap	512 (MB)
--disk	50 (GB)	rootfs size on --rootfs-storage
--bridge	vmbr0
--vlan	(none = untagged)
--nameserver	1.1.1.1
--searchdomain	(none)
--timezone	Europe/Kyiv
--template-storage	local-btrfs	must have content type vztmpl enabled
--rootfs-storage	EX-Ceph	container disk storage
--discord-webhook	(none)	passed through as DISCORD_WEBHOOK_URL to the in-container script

Dashboards

Three focused dashboards are created automatically instead of one combined view - each is scoped to what one kind of reader actually needs, so opening Graylog goes straight to something relevant instead of one big page mixing network gear, servers, and alerts together:

• Overview & Alerts (Dashboards → Overview & Alerts) - the landing page. Recent Alerts (last 24h, pulled straight from the "All events" stream so you see the actual fired alerts, not just counts), Message Volume by Source for the last hour (catches a flood visually before the flood alerts even fire), Events by Priority (24h), and Critical Events by Type (24h).
• Network Equipment (Dashboards → Network Equipment) - scoped to the Network Equipment stream only, 7-day window: messages over time by event type, vendor breakdown (Juniper vs. BDCOM), event type table, top devices by volume.
• Servers & Sessions (Dashboards → Servers & Sessions) - scoped to the Servers stream only, 7-day window: messages over time by event type, event type table, top servers by volume (same view the flood alerts are calibrated against), and a RADIUS accounting status breakdown (Start/Alive/Stop counts).

Each is built via the Views API (dashboards/search_<name>.json + dashboards/view_<name>.json pairs, one pair per dashboard) rather than Graylog's own widget-builder UI - that UI turned out to be difficult to drive reliably via browser automation (React combobox widgets that don't respond to plain keyboard/click events without also triggering React's internal state update), while the REST API accepted the same structure cleanly once the shape was reverse engineered from an existing dashboard's JSON. The one part that isn't a plain aggregation pivot - the Recent Alerts widget, type: "messages" instead of type: "aggregation" - needed its own bit of reverse engineering too: the widget-level sort field on a message-list widget must be [], not an object with a field/order pair, or Graylog rejects it with a Jackson polymorphism error (missing type id property 'type' - the sort DTO for message widgets doesn't have any registered subtypes in this Graylog version at all).

If you want to add a widget, either use the Graylog UI directly (a human using a mouse doesn't hit the automation issue) and then optionally export the result back into these JSON files, or extend a search_<name>.json/ view_<name>.json pair by hand - each widget needs a matching search_types entry (in search_<name>.json) and widgets + widget_mapping + positions + titles.widget entry (in view_<name>.json) sharing the same ID. step_dashboard() in install-graylog.sh picks up any search_*.json/view_*.json pair automatically (matched by filename), so a new pair just needs to exist in the dashboards/ directory - no script changes required. __NETWORK_STREAM_ID__ and __SERVERS_STREAM_ID__ placeholders are substituted the same way the alert templates already do it.

Known environment quirks this script works around

• Docker-in-unprivileged-LXC AppArmor block: containers fail with open sysctl net.ipv4.ip_unprivileged_port_start: permission denied unless the Proxmox host admin adds a raw LXC config line. pct set does not expose this option, so it cannot be automated within the claude-deploy sudo scope. If you hit this, run as root on the host:

  echo "lxc.apparmor.profile: unconfined" >> /etc/pve/lxc/<VMID>.conf
  pct reboot <VMID>
  

  then re-run create-graylog-lxc.sh (idempotent, will continue from there).

• vm.max_map_count: OpenSearch requires >= 262144. This is a host-wide kernel parameter, not namespaced per LXC, so it can't be set from inside the container either. install-graylog.sh only verifies it and fails with instructions if it's too low — on this deployment it was already 262144 by default, so no action was needed.

• Docker Hub TLS over IPv6: on this network, IPv6 paths to registry-1.docker.io intermittently get intercepted and served a mismatched certificate (*.docker.com). The install script disables IPv6 inside the container to force IPv4-only egress. Image pulls also retry up to 5x since even IPv4 occasionally hits a bad edge node.

• MongoDB version: Graylog 7.1 requires MongoDB >= 7.0 (some older docs still reference 6.0.x — don't trust cached documentation over what the running server actually reports).

• RFC3164 syslog timestamps have no timezone - Graylog assumes UTC by default: most network gear and accel-ppp send classic RFC3164 syslog (Jul 22 09:15:13, no year, no offset). Without an explicit timezone setting on the input, Graylog stores that bare timestamp as if it were already UTC - so a device logging in local Kyiv time (UTC+3) shows up 3 hours in the future in Graylog. Confirmed live: a raw test packet with Jul 22 09:15:13 was stored as 09:15:13Z (wrong) until each Syslog UDP input's timezone config was set to the container's own timezone (Europe/Kyiv here); afterwards it correctly stored as 06:15:13Z (09:15:13 Kyiv = 06:15:13 UTC). step_inputs() in install-graylog.sh sets this automatically from /etc/timezone for every input it creates, and self-heals it on existing inputs that predate this fix.

• The built-in admin user's displayed timezone is a separate setting from the input-level fix above: even after the RFC3164 fix, the Web UI can still show times in UTC for the read-only built-in admin account. That account's timezone is read_only: true and cannot be changed via PUT /api/users/admin (confirmed live - fails with "state should be: hexString has 24 characters", since that endpoint isn't meant for the special built-in account). It can only be set server-side via the root_timezone config option, i.e. Docker's GRAYLOG_ROOT_TIMEZONE env var. step_compose_files() sets this from /etc/timezone for fresh installs and self-heals it into any pre-existing .env that predates this fix; docker-compose.yml passes it through to the graylog service. A docker compose up -d re-run picks up the change and recreates the container automatically.

• Network gear sends syslog to port 514, not a custom port: most switches/OLTs (confirmed live with a BDCOM S5612) only support logging <host>, which always uses the standard UDP/514, with no way to point at a different port. There are therefore two "network equipment" inputs - 514 (what devices actually use) and 1514 (kept for any gear that can target a custom port) - both feeding the same "Network Equipment" stream (matching_type: OR). If you add new equipment and it doesn't show up, check with tcpdump -i eth0 udp port 514 inside the container before assuming the pipeline rules are wrong.

• nftables must never flush ruleset: an early version of this firewall step used flush ruleset, which also wipes Docker's own iptables-nft-managed nat/filter tables (DOCKER, DOCKER-USER, etc.), breaking container port publishing the next time docker compose up needs to add a rule (confirmed live - had to recover with a full docker compose down && up). nftables.conf here only ever does add table inet filter + flush table inet filter, which is scoped to just that one table and safe regardless of boot/service order relative to docker.service.

Alerting and Discord notifications

Six alerts are wired up out of the box (everything else stays quiet on purpose - routine auth failures, single dropped-session events, etc. are parsed and searchable but never page anyone):

Alert	Fires on	Priority
RADIUS server unreachable	radius: server(N) not responding or radius: no available servers (verified strings from accel-ppp source, radius/req.c)	High
conntrack table full (packet loss)	nf_conntrack: table full, dropping packet (standard Linux kernel message - active, ongoing packet loss)	High
Unrecognized critical-severity syslog	Any message (any vendor, either stream) with RFC5424/3164 syslog severity Emergency/Alert/Critical (0-2) that no specific pipeline rule already classified	Medium
Juniper chassis hardware alarm	CHASSISD_SNMP_TRAP/CHASSISD_SNMP_TRAP6 (over temperature, fan, power supply, etc.) - confirmed live against a real VC.MYRONIVKA chassis	High
Abnormal message volume from one server	A single server in the Servers stream sends more than 150,000 messages in a 10-minute window - see "Message-volume (flood) alerts" below	Medium
Abnormal syslog volume from network equipment	A single device in the Network Equipment stream sends more than 500 messages in a 5-minute window - see "Message-volume (flood) alerts" below	Medium

That third one is the "universal network equipment problem" catch-all: it doesn't depend on knowing any vendor's specific message format, just the standard syslog severity level every reasonable device already sends.

Message-volume (flood) alerts

The last two alerts protect against a single misbehaving source silently filling the retention window's disk budget - a log loop, a retry storm, or a debug-level setting left on by accident. They group by gl2_remote_ip (aggregation-v1, count() > threshold), so each individual source is compared against its own volume, not the whole stream's total.

The thresholds are not guessed - they were calibrated live on 2026-07-22 against real traffic, via a Views API pivot search grouped by gl2_remote_ip:

• The one active accel-ppp/RADIUS server was steadily sending ~4,600-4,800 messages per 10 minutes (~278k/hour) under normal load. The Servers stream threshold (150,000/10min) gives roughly 3x headroom above that.
• The one active network device was steadily sending ~30-60 messages per 5 minutes (~360/hour). The Network Equipment stream threshold (500/5min) gives roughly 10x headroom above that.

These are starting points based on partial rollout (1 server + 1 device active at calibration time). Revisit both thresholds once more of the planned ~10-15 servers and ~10-20 switches/OLTs are sending real traffic - what looks like 3x headroom today could be too tight or too loose once every server's individual baseline is known. Check current per-source volume any time with a query like:

gl2_remote_ip:<ip>

over a fixed time range in the Search page, or reuse the same pivot-search approach (grouped by gl2_remote_ip, count() series) via the Views API if you want exact numbers instead of eyeballing a graph.

Also parsed (searchable, but not alerted on since they're routine/expected volume, not incidents by themselves):

• accel-ppp: PPP authentication failed (ppp_auth.c)
• Standalone FreeRADIUS: Auth: (n) Login OK: [user] (from client X port P) / Auth: Login incorrect: [user] (from client X port P) (default auth_log format)

Session correlation (accel-ppp subscriber sessions)

Every accel-ppp log line for a given subscriber session - RADIUS Access-Request (auth attempt), Accounting-Request (start/interim/stop), DHCP discover/offer/request/ack, ipoe session create/start/finish/terminate

• gets tagged with the same accelppp_interface field (the vlanNNNN.NNN interface name accel-ppp itself uses per subscriber). This works even for message types with no other structured fields at all, via a fallback rule (accelppp_interface_tag) that only tags lines no more specific rule already classified.

To see a subscriber's full session lifecycle in one query, search:

accelppp_interface:"vlan1779.124"

sorted by time (default). This surfaces the DHCP handshake, the RADIUS auth/accounting exchange, and the eventual termination as one chronological list, instead of grep-ing for the interface name across raw text.

RADIUS Access-Request and Accounting-Request lines additionally get three richer correlation fields extracted directly from the RADIUS AVPs:

• radius_session_id - accel-ppp's Acct-Session-Id, stable for the entire session
• calling_station_id - the subscriber's MAC address
• radius_username - the subscriber's login (accel-ppp's User-Name, format <vlan>:<qinq> in this deployment)

These are useful when starting from a support ticket that has a MAC address or username but not the interface name, e.g.:

calling_station_id:"48:8f:5a:a4:f9:ba"

Confirmed live on 2026-07-22 against real EX-NAS-1-1 traffic: a single accelppp_interface value correctly tied together a DHCPv4 Ack, a DHCPv4 Request, a RADIUS Accounting-Response, and a RADIUS Accounting-Request, all belonging to the same subscriber session.

To wire up Discord, pass --discord-webhook (or set DISCORD_WEBHOOK_URL) when running create-graylog-lxc.sh. Under the hood this uses Graylog's built-in Slack notification type pointed at <your-webhook-url>/slack - Discord's Slack-compatibility endpoint - so no extra converter service is needed. The message template shows the event title/description plus the source, sending IP, and raw text of every matched message:

*${event_definition_title}*
${event_definition_description}
${event.message}
${if backlog}${foreach backlog message}• `${message.source}` (IP: ${message.fields.gl2_remote_ip}): ${message.message}
${end}${end}

${event.message} is Graylog's own auto-generated event summary - for plain critical alerts it just duplicates the title, but for the two flood alerts (grouped by gl2_remote_ip) this is where the specific source IP and the actual count() value show up, e.g. WARNING: ...: 93.171.243.4 - count()=278474.0.

gl2_remote_ip is a field Graylog attaches automatically to every message based on the actual UDP packet's source address, regardless of what hostname the device itself claims in the syslog source field.

Searching by IP address

Every message is searchable by the sending device's real IP via the same gl2_remote_ip field, in the Search page query bar:

gl2_remote_ip:93.171.243.4

source:<value> also works, but only matches if the device's self-reported hostname was used (some equipment sends its actual IP as the hostname, others send a configured name) - gl2_remote_ip is the reliable one since it's derived from the packet itself, not device-supplied data.

Verifying a test alert

# from inside the container, simulating each trigger:
docker exec -it graylog-server bash
logger -n 127.0.0.1 -P 5140 -d 'radius: server(1) not responding'
logger -n 127.0.0.1 -P 5140 -d 'kernel: nf_conntrack: table full, dropping packet'

The scheduler checks every 60s, so the Discord message can take up to a minute to arrive. Check Graylog's Alerts page and the configured Discord channel.

CI/CD via Forgejo Actions

This project's Forgejo repo (git.zotac.keenetic.link/zotac/graylog-deploy) has a manually-triggered deploy workflow: .forgejo/workflows/deploy.yml. Running it (Actions tab → Deploy Graylog config → Run workflow) clones the repo fresh and re-runs install-graylog.sh - the same idempotent script described throughout this README, just automated instead of scp'd by hand.

• Trigger is manual on purpose (workflow_dispatch only, no on: push) - this repo drives a production monitoring system, so a human reviews the diff and clicks "Run workflow" rather than every push silently redeploying.
• The runner lives inside the Graylog container itself (VMID 200, self-hosted:host label, installed at /usr/local/bin/forgejo-runner, running as a systemd service). No new SSH keys or cross-host access were needed - the workflow just clones the repo into /tmp/graylog-deploy-ci and runs the script locally, exactly like a human operator would.
• No actions/checkout - that action needs Node.js, which this appliance container doesn't have and shouldn't need just for CI. The workflow does a plain git clone --depth 1 instead.
• Secrets (GRAYLOG_ADMIN_PASSWORD, DISCORD_WEBHOOK_URL) are stored as repo-level Forgejo Actions secrets, not in any tracked file. GRAYLOG_EXTERNAL_URI isn't secret (it's the public Web UI address) so it's inlined directly in the workflow.

What's still NOT included

• D-Link switch parsing — no sample logs were available.
• ZTE OLT ONU online/offline + optical power alarms — no sample logs were available (only BDCOM OLT-style CLI logs were provided: privilege-mode/logout/ARP-move/config-write, which are parsed).
• Plain RADIUS Access-Accept/Reject from a specific real deployment — the standalone-FreeRADIUS rule above is based on FreeRADIUS's documented default log format, not a sample from this network's actual RADIUS boxes (no SSH access to them was available; the pattern is well-established upstream, but verify it against a real log line when one is available).

To fill these gaps: provide a few real raw log lines per source (D-Link syslog, ZTE OLT ONU up/down + optical alarms) and the equivalent rules/*.json + alert definitions can be added the same way the existing ones were built.

File layout

create-graylog-lxc.sh   # run on the Proxmox host
install-graylog.sh      # run inside the container (invoked automatically)
docker-compose.yml      # MongoDB + OpenSearch + Graylog stack definition
nftables.conf            # firewall ruleset applied inside the container
rules/*.json             # Graylog Pipeline Rule definitions (imported via API)
pipelines/*.json         # Graylog Pipeline definitions referencing the rules
streams/*.json           # Graylog Stream definitions (routes by input port)

Inside the container, everything lives under /opt/graylog/ (docker-compose.yml, .env with secrets) and /opt/graylog-deploy/ (a copy of this repo, used for re-running the install idempotently).

Credentials

install-graylog.sh generates GRAYLOG_PASSWORD_SECRET and a random admin password on first run, writing the admin password once to /opt/graylog/.admin_credentials_ONE_TIME inside the container — read it, store it in your password manager, then delete the file:

pct exec <VMID> -- cat /opt/graylog/.admin_credentials_ONE_TIME
pct exec <VMID> -- rm /opt/graylog/.admin_credentials_ONE_TIME