From aa5cfbfbd82305051ffaeea67ed1cfb8cfd029d3 Mon Sep 17 00:00:00 2001 From: byrsapty Date: Wed, 22 Jul 2026 04:51:39 +0300 Subject: [PATCH] first commit --- README.md | 381 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 381 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..24f8c25 --- /dev/null +++ b/README.md @@ -0,0 +1,381 @@ +# Graylog centralized log server — deployment scripts + +Deploys a Debian 12 LXC container on Proxmox VE with a Docker Compose stack +(MongoDB + OpenSearch + Graylog) for collecting syslog from network equipment +(Juniper, ZTE OLT, D-Link) and servers (RADIUS, accel-ppp). + +## Quick start + +Run on the Proxmox host as `claude-deploy` (or any user with the same +restricted sudo rights on `pct`/`pveam` for VMIDs 200-299): + +```bash +./create-graylog-lxc.sh \ + --ip 10.254.254.202/24 \ + --gw 10.254.254.235 \ + --vlan 1254 \ + --external-uri http://93.171.241.5:9000/ \ + --discord-webhook "https://discord.com/api/webhooks/xxx/yyy" +``` + +This is the **only command you need**. It creates the container, installs +Docker, applies the firewall (nftables: only 9000/tcp, 514+1514/udp, +5140/udp open), brings up the stack, creates the Syslog inputs, imports +the pipeline rules, and prints the admin credentials at the end. + +Both scripts are idempotent — re-running after a failure (or to apply an +update) picks up from where it left off instead of duplicating work. + +## Step-by-step deployment from scratch + +### 0. Prerequisites + +- SSH access to the Proxmox host as a user with sudo rights on `pct` + create/set/start/stop/exec/status/list and `pveam` update/list/download + (does **not** need to be root - this whole toolkit was built and tested + under exactly that restricted scope). +- A storage on the Proxmox host with content type `vztmpl` enabled, holding + (or able to download) a `debian-12-standard` template - default assumed + name: `local-btrfs`. Check with: + ```bash + sudo pveam list local-btrfs # replace with your storage name + ``` + If it errors with "storage is disabled" or "does not exist", find the + right one (`sudo pveam list ` for candidates) and pass it via + `--template-storage`. +- A storage for the container's root filesystem (default assumed: + `EX-Ceph`) with enough free space for `--disk` (default 50GB). +- The IP address, gateway, and VLAN tag (if any) for the network the + container will live on - i.e. the same network your RADIUS/NAS servers + and network equipment can reach. +- If you need remote access to the Web UI from outside that network: a + spare external port to DNAT to the container's `9000/tcp` (see the + "reaching the Web UI from outside" note below - port 9000 collided with + an existing ClickHouse listener during initial testing, so don't assume + any specific port is free). +- (Optional) A Discord webhook URL, if you want Critical alerts pushed to + a channel. + +### 1. Get the scripts onto the Proxmox host + +From your workstation: +```bash +tar czf - -C /path/to/graylog-deploy . | ssh claude-deploy@ \ + "mkdir -p ~/graylog-deploy && tar xzf - -C ~/graylog-deploy && chmod +x ~/graylog-deploy/*.sh" +``` +(Or `git clone`/`scp` the directory if you keep it in a repo - any method +that gets the whole folder, including `rules/`, `pipelines/`, `streams/`, +`alerts/`, onto the host works.) + +### 2. Decide your parameters + +Pick, in advance: + +| You need | Example | Why | +|---|---|---| +| A free VMID in 200-299 | `210` | or omit `--vmid` to auto-pick the first free one | +| Container IP + CIDR on the management network | `10.254.254.220/24` | must be reachable by every device that will send syslog | +| Gateway on that network | `10.254.254.235` | | +| VLAN tag (if the bridge is trunked) | `1254` | omit `--vlan` if untagged | +| Public URL for the Web UI | `http://:/` | used both in Graylog's config and in Discord alert links | +| Discord webhook (optional) | `https://discord.com/api/webhooks/.../...` | leave unset to skip Discord entirely | + +**Check the IP is actually free first** - a duplicate IP on this network +silently causes ARP flapping and intermittent, hard-to-diagnose routing +weirdness (this happened during initial testing: `.201` turned out to +already be in use, causing traffic to randomly land on the wrong host). +A quick way to check without touching anything: +```bash +ssh claude-deploy@ "sudo /usr/sbin/pct exec -- ping -c2 -W1 " +``` +If you get replies, that IP is taken - pick another. + +### 3. Run `create-graylog-lxc.sh` + +```bash +ssh claude-deploy@ +cd ~/graylog-deploy +./create-graylog-lxc.sh \ + --vmid 210 \ + --ip 10.254.254.220/24 \ + --gw 10.254.254.235 \ + --vlan 1254 \ + --external-uri http://:/ \ + --discord-webhook "https://discord.com/api/webhooks/xxx/yyy" +``` + +What happens, in order: template check/download → `pct create` → start the +container → wait for network → copy this whole folder into the container +at `/opt/graylog-deploy/` → run `install-graylog.sh` inside it, which +installs Docker, applies the firewall, brings up MongoDB/OpenSearch/Graylog, +waits for it to report healthy, creates the Syslog inputs, imports the +pipeline rules/pipelines/streams, and (if a webhook was given) creates the +Discord notification and the three alert definitions. + +**If it stops with an AppArmor error** (`open sysctl +net.ipv4.ip_unprivileged_port_start: permission denied`), that's expected +on some Proxmox setups and needs one manual step from the host's root +account - see "Docker-in-unprivileged-LXC AppArmor block" below. Do that, +then just run the exact same command again; everything already done is +skipped automatically. + +Total time for a clean run: a few minutes, mostly waiting for image pulls +and for Graylog to report healthy. + +### 4. Read the admin credentials + +```bash +ssh claude-deploy@ "sudo pct exec 210 -- cat /opt/graylog/.admin_credentials_ONE_TIME" +``` +Copy the password into your password manager, then delete the file: +```bash +ssh claude-deploy@ "sudo pct exec 210 -- rm /opt/graylog/.admin_credentials_ONE_TIME" +``` +Log into `http://:/` with user `admin` and that password. + +### 5. Reach the Web UI from outside the management network (if needed) + +If the container's IP isn't directly reachable from where you browse, add +a DNAT rule on your edge router/firewall for `9000/tcp`: +``` +-A PREROUTING -d /32 -p tcp -m tcp --dport -j DNAT --to-destination :9000 +``` +Pick a port that's actually free - during initial setup, port 9000 turned +out to already be serving ClickHouse on the same public IP, and the +response (`Port 9000 is for clickhouse-client program...`) looked like a +Graylog problem for a while before that was found. If you get a +suspicious/unexpected response on the port you chose, suspect a +pre-existing service on that port before suspecting Graylog. + +### 6. Point real equipment at the server + +| Source | Port | Notes | +|---|---|---| +| Network equipment (switches, OLTs, routers) | **514/udp** | the standard syslog port - almost no gear lets you pick a different one (`logging ` on a typical switch always uses 514) | +| Network equipment that *can* target a custom port | 1514/udp | kept as a secondary input, same stream as 514 | +| Servers (RADIUS, accel-ppp, conntrack/kernel messages) | **5140/udp** | | + +On each device, this is normally a one-line config change (e.g. on a +Cisco-like CLI: `logging `). No Graylog-side config is +needed per device - the inputs and streams already listen on all three +ports. + +### 7. Verify data is actually arriving + +1. Web UI → **System → Inputs**: each input shows live "Traffic Last + Minute". If it stays at 0 after a device should have sent something, + the problem is network/firewall, not Graylog - confirm with `tcpdump` + inside the container before touching any Graylog config: + ```bash + sudo pct exec 210 -- tcpdump -i eth0 -n udp port 514 + ``` +2. Web UI → **Search**, widen the time range (top-left), click any message + to expand it. If `vendor` / `event_type` fields are populated, a + pipeline rule matched it. If a message arrives but those fields are + empty, it reached Graylog fine but no rule recognizes its format yet - + that's a sign a new device/vendor needs a new rule (see "What's still + NOT included" below for the process). +3. Web UI → **Streams**: the "Throughput" column shows live msg/s per + stream. + +### 8. (Optional) Trigger a real test alert + +See "Verifying a test alert" further down - send one of the known-critical +log lines via `logger` from inside the container and watch it land in +Discord within about a minute. + +## Parameters + +All flags have an environment-variable equivalent (see the top of +`create-graylog-lxc.sh`), so you can also `export MEMORY_MB=16384` etc. +instead of passing flags. + +| Flag | Default | Notes | +|---|---|---| +| `--ip` | *(required)* | Static IP + CIDR for the container | +| `--gw` | *(required)* | Gateway IP | +| `--external-uri` | *(required)* | Public URL for the Graylog Web UI (used in `GRAYLOG_HTTP_EXTERNAL_URI`) | +| `--vmid` | first free 200-299 | | +| `--hostname` | `graylog` | | +| `--cores` | `4` | | +| `--memory` | `8192` (MB) | | +| `--swap` | `512` (MB) | | +| `--disk` | `50` (GB) | rootfs size on `--rootfs-storage` | +| `--bridge` | `vmbr0` | | +| `--vlan` | *(none = untagged)* | | +| `--nameserver` | `1.1.1.1` | | +| `--searchdomain` | *(none)* | | +| `--timezone` | `Europe/Kyiv` | | +| `--template-storage` | `local-btrfs` | must have content type `vztmpl` enabled | +| `--rootfs-storage` | `EX-Ceph` | container disk storage | +| `--discord-webhook` | *(none)* | passed through as `DISCORD_WEBHOOK_URL` to the in-container script | + +## Dashboard + +A "Network & RADIUS Monitoring" dashboard is created automatically +(Dashboards → Network & RADIUS Monitoring), with five widgets over a +default 7-day window: + +- **Messages Over Time by Stream** - stacked bar, so you can see network + equipment vs. server volume at a glance +- **Vendor Breakdown** - pie chart by the `vendor` field the pipeline + rules set +- **Top Event Types** - table of `event_type` counts +- **Critical Events by Type** - the same, but filtered to + `severity_tag:critical` - i.e. only what the three alerts above care about +- **Top Sources** - which devices/servers are sending the most volume + +It's built via the Views API (`dashboards/search.json` + `dashboards/view.json`) +rather than Graylog's own widget-builder UI - that UI turned out to be +difficult to drive reliably via browser automation (React `combobox` +widgets that don't respond to plain keyboard/click events without also +triggering React's internal state update), while the REST API accepted the +same structure cleanly on the first attempt once the shape was reverse +engineered from an existing dashboard's JSON. If you want to add a widget, +either use the Graylog UI directly (a human using a mouse doesn't hit the +automation issue) and then optionally export the result back into these +two JSON files, or extend `dashboards/search.json`/`view.json` by hand - +each widget needs a matching `search_types` entry (in `search.json`) and +`widgets` + `widget_mapping` + `positions` + `titles.widget` entry (in +`view.json`) sharing the same ID. + +## Known environment quirks this script works around + +- **Docker-in-unprivileged-LXC AppArmor block**: containers fail with + `open sysctl net.ipv4.ip_unprivileged_port_start: permission denied` + unless the Proxmox host admin adds a raw LXC config line. `pct set` + does not expose this option, so it cannot be automated within the + `claude-deploy` sudo scope. If you hit this, run as root on the host: + ```bash + echo "lxc.apparmor.profile: unconfined" >> /etc/pve/lxc/.conf + pct reboot + ``` + then re-run `create-graylog-lxc.sh` (idempotent, will continue from there). + +- **`vm.max_map_count`**: OpenSearch requires >= 262144. This is a + host-wide kernel parameter, not namespaced per LXC, so it can't be set + from inside the container either. `install-graylog.sh` only verifies it + and fails with instructions if it's too low — on this deployment it was + already 262144 by default, so no action was needed. + +- **Docker Hub TLS over IPv6**: on this network, IPv6 paths to + `registry-1.docker.io` intermittently get intercepted and served a + mismatched certificate (`*.docker.com`). The install script disables + IPv6 inside the container to force IPv4-only egress. Image pulls also + retry up to 5x since even IPv4 occasionally hits a bad edge node. + +- **MongoDB version**: Graylog 7.1 requires MongoDB >= 7.0 (some older + docs still reference 6.0.x — don't trust cached documentation over + what the running server actually reports). + +- **Network gear sends syslog to port 514, not a custom port**: most + switches/OLTs (confirmed live with a BDCOM S5612) only support + `logging `, which always uses the standard UDP/514, with no way to + point at a different port. There are therefore **two** "network + equipment" inputs - 514 (what devices actually use) and 1514 (kept for + any gear that *can* target a custom port) - both feeding the same + "Network Equipment" stream (`matching_type: OR`). If you add new + equipment and it doesn't show up, check with `tcpdump -i eth0 udp port + 514` inside the container before assuming the pipeline rules are wrong. + +- **nftables must never `flush ruleset`**: an early version of this + firewall step used `flush ruleset`, which also wipes Docker's own + iptables-nft-managed nat/filter tables (`DOCKER`, `DOCKER-USER`, etc.), + breaking container port publishing the next time `docker compose up` + needs to add a rule (confirmed live - had to recover with a full + `docker compose down && up`). `nftables.conf` here only ever does + `add table inet filter` + `flush table inet filter`, which is scoped to + just that one table and safe regardless of boot/service order relative + to `docker.service`. + +## Alerting and Discord notifications + +Three CRITICAL-only alerts are wired up out of the box (everything else stays +quiet on purpose - routine auth failures, single dropped-session events, etc. +are parsed and searchable but never page anyone): + +| Alert | Fires on | Priority | +|---|---|---| +| RADIUS server unreachable | `radius: server(N) not responding` or `radius: no available servers` (verified strings from accel-ppp source, `radius/req.c`) | High | +| conntrack table full (packet loss) | `nf_conntrack: table full, dropping packet` (standard Linux kernel message - active, ongoing packet loss) | High | +| Unrecognized critical-severity syslog | Any message (any vendor, either stream) with RFC5424/3164 syslog severity Emergency/Alert/Critical (0-2) that no specific pipeline rule already classified | Medium | + +That third one is the "universal network equipment problem" catch-all: it +doesn't depend on knowing any vendor's specific message format, just the +standard syslog severity level every reasonable device already sends. + +Also parsed (searchable, but not alerted on since they're routine/expected +volume, not incidents by themselves): +- accel-ppp: PPP authentication failed (`ppp_auth.c`) +- Standalone FreeRADIUS: `Auth: (n) Login OK: [user] (from client X port P)` / + `Auth: Login incorrect: [user] (from client X port P)` (default `auth_log` + format) + +To wire up Discord, pass `--discord-webhook` (or set `DISCORD_WEBHOOK_URL`) +when running `create-graylog-lxc.sh`. Under the hood this uses Graylog's +built-in **Slack** notification type pointed at +`/slack` - Discord's Slack-compatibility endpoint - so no +extra converter service is needed. The message template shows the event +title/description plus the source and raw text of every matched message: +``` +*${event_definition_title}* +${event_definition_description} +${if backlog}${foreach backlog message}• `${message.source}`: ${message.message} +${end}${end} +``` + +### Verifying a test alert + +```bash +# from inside the container, simulating each trigger: +docker exec -it graylog-server bash +logger -n 127.0.0.1 -P 5140 -d 'radius: server(1) not responding' +logger -n 127.0.0.1 -P 5140 -d 'kernel: nf_conntrack: table full, dropping packet' +``` +The scheduler checks every 60s, so the Discord message can take up to a +minute to arrive. Check Graylog's Alerts page and the configured Discord +channel. + +## What's still NOT included + +- **D-Link switch parsing** — no sample logs were available. +- **ZTE OLT ONU online/offline + optical power alarms** — no sample + logs were available (only BDCOM OLT-style CLI logs were provided: + privilege-mode/logout/ARP-move/config-write, which *are* parsed). +- **Plain RADIUS Access-Accept/Reject from a specific real deployment** — + the standalone-FreeRADIUS rule above is based on FreeRADIUS's documented + default log format, not a sample from this network's actual RADIUS boxes + (no SSH access to them was available; the pattern is well-established + upstream, but verify it against a real log line when one is available). + +To fill these gaps: provide a few real raw log lines per source (D-Link +syslog, ZTE OLT ONU up/down + optical alarms) and the equivalent +`rules/*.json` + alert definitions can be added the same way the existing +ones were built. + +## File layout + +``` +create-graylog-lxc.sh # run on the Proxmox host +install-graylog.sh # run inside the container (invoked automatically) +docker-compose.yml # MongoDB + OpenSearch + Graylog stack definition +nftables.conf # firewall ruleset applied inside the container +rules/*.json # Graylog Pipeline Rule definitions (imported via API) +pipelines/*.json # Graylog Pipeline definitions referencing the rules +streams/*.json # Graylog Stream definitions (routes by input port) +``` + +Inside the container, everything lives under `/opt/graylog/` +(`docker-compose.yml`, `.env` with secrets) and `/opt/graylog-deploy/` +(a copy of this repo, used for re-running the install idempotently). + +## Credentials + +`install-graylog.sh` generates `GRAYLOG_PASSWORD_SECRET` and a random +admin password on first run, writing the admin password once to +`/opt/graylog/.admin_credentials_ONE_TIME` inside the container — read +it, store it in your password manager, then delete the file: +```bash +pct exec -- cat /opt/graylog/.admin_credentials_ONE_TIME +pct exec -- rm /opt/graylog/.admin_credentials_ONE_TIME +```