graylog-deploy/README.md

# Graylog centralized log server — deployment scripts

Deploys a Debian 12 LXC container on Proxmox VE with a Docker Compose stack
(MongoDB + OpenSearch + Graylog) for collecting syslog from network equipment
(Juniper, ZTE OLT, D-Link) and servers (RADIUS, accel-ppp).

## Quick start

Run on the Proxmox host as `claude-deploy` (or any user with the same
restricted sudo rights on `pct`/`pveam` for VMIDs 200-299):

```bash
./create-graylog-lxc.sh \
  --ip 10.254.254.202/24 \
  --gw 10.254.254.235 \
  --vlan 1254 \
  --external-uri http://93.171.241.5:9000/ \
  --discord-webhook "https://discord.com/api/webhooks/xxx/yyy"
```

This is the **only command you need**. It creates the container, installs
Docker, applies the firewall (nftables: only 9000/tcp, 514+1514/udp,
5140/udp open), brings up the stack, creates the Syslog inputs, imports
the pipeline rules, and prints the admin credentials at the end.

Both scripts are idempotent — re-running after a failure (or to apply an
update) picks up from where it left off instead of duplicating work.

## Step-by-step deployment from scratch

### 0. Prerequisites

- SSH access to the Proxmox host as a user with sudo rights on `pct`
  create/set/start/stop/exec/status/list and `pveam` update/list/download
  (does **not** need to be root - this whole toolkit was built and tested
  under exactly that restricted scope).
- A storage on the Proxmox host with content type `vztmpl` enabled, holding
  (or able to download) a `debian-12-standard` template - default assumed
  name: `local-btrfs`. Check with:
  ```bash
  sudo pveam list local-btrfs   # replace with your storage name
  ```
  If it errors with "storage is disabled" or "does not exist", find the
  right one (`sudo pveam list <name>` for candidates) and pass it via
  `--template-storage`.
- A storage for the container's root filesystem (default assumed:
  `EX-Ceph`) with enough free space for `--disk` (default 50GB).
- The IP address, gateway, and VLAN tag (if any) for the network the
  container will live on - i.e. the same network your RADIUS/NAS servers
  and network equipment can reach.
- If you need remote access to the Web UI from outside that network: a
  spare external port to DNAT to the container's `9000/tcp` (see the
  "reaching the Web UI from outside" note below - port 9000 collided with
  an existing ClickHouse listener during initial testing, so don't assume
  any specific port is free).
- (Optional) A Discord webhook URL, if you want Critical alerts pushed to
  a channel.

### 1. Get the scripts onto the Proxmox host

From your workstation:
```bash
tar czf - -C /path/to/graylog-deploy . | ssh claude-deploy@<proxmox-host> \
  "mkdir -p ~/graylog-deploy && tar xzf - -C ~/graylog-deploy && chmod +x ~/graylog-deploy/*.sh"
```
(Or `git clone`/`scp` the directory if you keep it in a repo - any method
that gets the whole folder, including `rules/`, `pipelines/`, `streams/`,
`alerts/`, onto the host works.)

### 2. Decide your parameters

Pick, in advance:

| You need | Example | Why |
|---|---|---|
| A free VMID in 200-299 | `210` | or omit `--vmid` to auto-pick the first free one |
| Container IP + CIDR on the management network | `10.254.254.220/24` | must be reachable by every device that will send syslog |
| Gateway on that network | `10.254.254.235` | |
| VLAN tag (if the bridge is trunked) | `1254` | omit `--vlan` if untagged |
| Public URL for the Web UI | `http://<public-ip>:<port>/` | used both in Graylog's config and in Discord alert links |
| Discord webhook (optional) | `https://discord.com/api/webhooks/.../...` | leave unset to skip Discord entirely |

**Check the IP is actually free first** - a duplicate IP on this network
silently causes ARP flapping and intermittent, hard-to-diagnose routing
weirdness (this happened during initial testing: `.201` turned out to
already be in use, causing traffic to randomly land on the wrong host).
A quick way to check without touching anything:
```bash
ssh claude-deploy@<proxmox-host> "sudo /usr/sbin/pct exec <any-running-vmid> -- ping -c2 -W1 <candidate-ip>"
```
If you get replies, that IP is taken - pick another.

### 3. Run `create-graylog-lxc.sh`

```bash
ssh claude-deploy@<proxmox-host>
cd ~/graylog-deploy
./create-graylog-lxc.sh \
  --vmid 210 \
  --ip 10.254.254.220/24 \
  --gw 10.254.254.235 \
  --vlan 1254 \
  --external-uri http://<public-ip>:<port>/ \
  --discord-webhook "https://discord.com/api/webhooks/xxx/yyy"
```

What happens, in order: template check/download → `pct create` → start the
container → wait for network → copy this whole folder into the container
at `/opt/graylog-deploy/` → run `install-graylog.sh` inside it, which
installs Docker, applies the firewall, brings up MongoDB/OpenSearch/Graylog,
waits for it to report healthy, creates the Syslog inputs, imports the
pipeline rules/pipelines/streams, and (if a webhook was given) creates the
Discord notification and the three alert definitions.

**If it stops with an AppArmor error** (`open sysctl
net.ipv4.ip_unprivileged_port_start: permission denied`), that's expected
on some Proxmox setups and needs one manual step from the host's root
account - see "Docker-in-unprivileged-LXC AppArmor block" below. Do that,
then just run the exact same command again; everything already done is
skipped automatically.

Total time for a clean run: a few minutes, mostly waiting for image pulls
and for Graylog to report healthy.

### 4. Read the admin credentials

```bash
ssh claude-deploy@<proxmox-host> "sudo pct exec 210 -- cat /opt/graylog/.admin_credentials_ONE_TIME"
```
Copy the password into your password manager, then delete the file:
```bash
ssh claude-deploy@<proxmox-host> "sudo pct exec 210 -- rm /opt/graylog/.admin_credentials_ONE_TIME"
```
Log into `http://<public-ip>:<port>/` with user `admin` and that password.

### 5. Reach the Web UI from outside the management network (if needed)

If the container's IP isn't directly reachable from where you browse, add
a DNAT rule on your edge router/firewall for `9000/tcp`:
```
-A PREROUTING -d <public-ip>/32 -p tcp -m tcp --dport <port> -j DNAT --to-destination <container-ip>:9000
```
Pick a port that's actually free - during initial setup, port 9000 turned
out to already be serving ClickHouse on the same public IP, and the
response (`Port 9000 is for clickhouse-client program...`) looked like a
Graylog problem for a while before that was found. If you get a
suspicious/unexpected response on the port you chose, suspect a
pre-existing service on that port before suspecting Graylog.

### 6. Point real equipment at the server

| Source | Port | Notes |
|---|---|---|
| Network equipment (switches, OLTs, routers) | **514/udp** | the standard syslog port - almost no gear lets you pick a different one (`logging <host>` on a typical switch always uses 514) |
| Network equipment that *can* target a custom port | 1514/udp | kept as a secondary input, same stream as 514 |
| Servers (RADIUS, accel-ppp, conntrack/kernel messages) | **5140/udp** | |

On each device, this is normally a one-line config change (e.g. on a
Cisco-like CLI: `logging <container-ip>`). No Graylog-side config is
needed per device - the inputs and streams already listen on all three
ports.

### 7. Verify data is actually arriving

1. Web UI → **System → Inputs**: each input shows live "Traffic Last
   Minute". If it stays at 0 after a device should have sent something,
   the problem is network/firewall, not Graylog - confirm with `tcpdump`
   inside the container before touching any Graylog config:
   ```bash
   sudo pct exec 210 -- tcpdump -i eth0 -n udp port 514
   ```
2. Web UI → **Search**, widen the time range (top-left), click any message
   to expand it. If `vendor` / `event_type` fields are populated, a
   pipeline rule matched it. If a message arrives but those fields are
   empty, it reached Graylog fine but no rule recognizes its format yet -
   that's a sign a new device/vendor needs a new rule (see "What's still
   NOT included" below for the process).
3. Web UI → **Streams**: the "Throughput" column shows live msg/s per
   stream.

### 8. (Optional) Trigger a real test alert

See "Verifying a test alert" further down - send one of the known-critical
log lines via `logger` from inside the container and watch it land in
Discord within about a minute.

## Parameters

All flags have an environment-variable equivalent (see the top of
`create-graylog-lxc.sh`), so you can also `export MEMORY_MB=16384` etc.
instead of passing flags.

| Flag | Default | Notes |
|---|---|---|
| `--ip` | *(required)* | Static IP + CIDR for the container |
| `--gw` | *(required)* | Gateway IP |
| `--external-uri` | *(required)* | Public URL for the Graylog Web UI (used in `GRAYLOG_HTTP_EXTERNAL_URI`) |
| `--vmid` | first free 200-299 | |
| `--hostname` | `graylog` | |
| `--cores` | `4` | |
| `--memory` | `8192` (MB) | |
| `--swap` | `512` (MB) | |
| `--disk` | `50` (GB) | rootfs size on `--rootfs-storage` |
| `--bridge` | `vmbr0` | |
| `--vlan` | *(none = untagged)* | |
| `--nameserver` | `1.1.1.1` | |
| `--searchdomain` | *(none)* | |
| `--timezone` | `Europe/Kyiv` | |
| `--template-storage` | `local-btrfs` | must have content type `vztmpl` enabled |
| `--rootfs-storage` | `EX-Ceph` | container disk storage |
| `--discord-webhook` | *(none)* | passed through as `DISCORD_WEBHOOK_URL` to the in-container script |

## Dashboard

A "Network & RADIUS Monitoring" dashboard is created automatically
(Dashboards → Network & RADIUS Monitoring), with five widgets over a
default 7-day window:

- **Messages Over Time by Stream** - stacked bar, so you can see network
  equipment vs. server volume at a glance
- **Vendor Breakdown** - pie chart by the `vendor` field the pipeline
  rules set
- **Top Event Types** - table of `event_type` counts
- **Critical Events by Type** - the same, but filtered to
  `severity_tag:critical` - i.e. only what the three alerts above care about
- **Top Sources** - which devices/servers are sending the most volume

It's built via the Views API (`dashboards/search.json` + `dashboards/view.json`)
rather than Graylog's own widget-builder UI - that UI turned out to be
difficult to drive reliably via browser automation (React `combobox`
widgets that don't respond to plain keyboard/click events without also
triggering React's internal state update), while the REST API accepted the
same structure cleanly on the first attempt once the shape was reverse
engineered from an existing dashboard's JSON. If you want to add a widget,
either use the Graylog UI directly (a human using a mouse doesn't hit the
automation issue) and then optionally export the result back into these
two JSON files, or extend `dashboards/search.json`/`view.json` by hand -
each widget needs a matching `search_types` entry (in `search.json`) and
`widgets` + `widget_mapping` + `positions` + `titles.widget` entry (in
`view.json`) sharing the same ID.

## Known environment quirks this script works around

- **Docker-in-unprivileged-LXC AppArmor block**: containers fail with
  `open sysctl net.ipv4.ip_unprivileged_port_start: permission denied`
  unless the Proxmox host admin adds a raw LXC config line. `pct set`
  does not expose this option, so it cannot be automated within the
  `claude-deploy` sudo scope. If you hit this, run as root on the host:
  ```bash
  echo "lxc.apparmor.profile: unconfined" >> /etc/pve/lxc/<VMID>.conf
  pct reboot <VMID>
  ```
  then re-run `create-graylog-lxc.sh` (idempotent, will continue from there).

- **`vm.max_map_count`**: OpenSearch requires >= 262144. This is a
  host-wide kernel parameter, not namespaced per LXC, so it can't be set
  from inside the container either. `install-graylog.sh` only verifies it
  and fails with instructions if it's too low — on this deployment it was
  already 262144 by default, so no action was needed.

- **Docker Hub TLS over IPv6**: on this network, IPv6 paths to
  `registry-1.docker.io` intermittently get intercepted and served a
  mismatched certificate (`*.docker.com`). The install script disables
  IPv6 inside the container to force IPv4-only egress. Image pulls also
  retry up to 5x since even IPv4 occasionally hits a bad edge node.

- **MongoDB version**: Graylog 7.1 requires MongoDB >= 7.0 (some older
  docs still reference 6.0.x — don't trust cached documentation over
  what the running server actually reports).

- **Network gear sends syslog to port 514, not a custom port**: most
  switches/OLTs (confirmed live with a BDCOM S5612) only support
  `logging <host>`, which always uses the standard UDP/514, with no way to
  point at a different port. There are therefore **two** "network
  equipment" inputs - 514 (what devices actually use) and 1514 (kept for
  any gear that *can* target a custom port) - both feeding the same
  "Network Equipment" stream (`matching_type: OR`). If you add new
  equipment and it doesn't show up, check with `tcpdump -i eth0 udp port
  514` inside the container before assuming the pipeline rules are wrong.

- **nftables must never `flush ruleset`**: an early version of this
  firewall step used `flush ruleset`, which also wipes Docker's own
  iptables-nft-managed nat/filter tables (`DOCKER`, `DOCKER-USER`, etc.),
  breaking container port publishing the next time `docker compose up`
  needs to add a rule (confirmed live - had to recover with a full
  `docker compose down && up`). `nftables.conf` here only ever does
  `add table inet filter` + `flush table inet filter`, which is scoped to
  just that one table and safe regardless of boot/service order relative
  to `docker.service`.

## Alerting and Discord notifications

Three CRITICAL-only alerts are wired up out of the box (everything else stays
quiet on purpose - routine auth failures, single dropped-session events, etc.
are parsed and searchable but never page anyone):

| Alert | Fires on | Priority |
|---|---|---|
| RADIUS server unreachable | `radius: server(N) not responding` or `radius: no available servers` (verified strings from accel-ppp source, `radius/req.c`) | High |
| conntrack table full (packet loss) | `nf_conntrack: table full, dropping packet` (standard Linux kernel message - active, ongoing packet loss) | High |
| Unrecognized critical-severity syslog | Any message (any vendor, either stream) with RFC5424/3164 syslog severity Emergency/Alert/Critical (0-2) that no specific pipeline rule already classified | Medium |

That third one is the "universal network equipment problem" catch-all: it
doesn't depend on knowing any vendor's specific message format, just the
standard syslog severity level every reasonable device already sends.

Also parsed (searchable, but not alerted on since they're routine/expected
volume, not incidents by themselves):
- accel-ppp: PPP authentication failed (`ppp_auth.c`)
- Standalone FreeRADIUS: `Auth: (n) Login OK: [user] (from client X port P)` /
  `Auth: Login incorrect: [user] (from client X port P)` (default `auth_log`
  format)

To wire up Discord, pass `--discord-webhook` (or set `DISCORD_WEBHOOK_URL`)
when running `create-graylog-lxc.sh`. Under the hood this uses Graylog's
built-in **Slack** notification type pointed at
`<your-webhook-url>/slack` - Discord's Slack-compatibility endpoint - so no
extra converter service is needed. The message template shows the event
title/description plus the source and raw text of every matched message:
```
*${event_definition_title}*
${event_definition_description}
${if backlog}${foreach backlog message}• `${message.source}`: ${message.message}
${end}${end}
```

### Verifying a test alert

```bash
# from inside the container, simulating each trigger:
docker exec -it graylog-server bash
logger -n 127.0.0.1 -P 5140 -d 'radius: server(1) not responding'
logger -n 127.0.0.1 -P 5140 -d 'kernel: nf_conntrack: table full, dropping packet'
```
The scheduler checks every 60s, so the Discord message can take up to a
minute to arrive. Check Graylog's Alerts page and the configured Discord
channel.

## What's still NOT included

- **D-Link switch parsing** — no sample logs were available.
- **ZTE OLT ONU online/offline + optical power alarms** — no sample
  logs were available (only BDCOM OLT-style CLI logs were provided:
  privilege-mode/logout/ARP-move/config-write, which *are* parsed).
- **Plain RADIUS Access-Accept/Reject from a specific real deployment** —
  the standalone-FreeRADIUS rule above is based on FreeRADIUS's documented
  default log format, not a sample from this network's actual RADIUS boxes
  (no SSH access to them was available; the pattern is well-established
  upstream, but verify it against a real log line when one is available).

To fill these gaps: provide a few real raw log lines per source (D-Link
syslog, ZTE OLT ONU up/down + optical alarms) and the equivalent
`rules/*.json` + alert definitions can be added the same way the existing
ones were built.

## File layout

```
create-graylog-lxc.sh   # run on the Proxmox host
install-graylog.sh      # run inside the container (invoked automatically)
docker-compose.yml      # MongoDB + OpenSearch + Graylog stack definition
nftables.conf            # firewall ruleset applied inside the container
rules/*.json             # Graylog Pipeline Rule definitions (imported via API)
pipelines/*.json         # Graylog Pipeline definitions referencing the rules
streams/*.json           # Graylog Stream definitions (routes by input port)
```

Inside the container, everything lives under `/opt/graylog/`
(`docker-compose.yml`, `.env` with secrets) and `/opt/graylog-deploy/`
(a copy of this repo, used for re-running the install idempotently).

## Credentials

`install-graylog.sh` generates `GRAYLOG_PASSWORD_SECRET` and a random
admin password on first run, writing the admin password once to
`/opt/graylog/.admin_credentials_ONE_TIME` inside the container — read
it, store it in your password manager, then delete the file:
```bash
pct exec <VMID> -- cat /opt/graylog/.admin_credentials_ONE_TIME
pct exec <VMID> -- rm /opt/graylog/.admin_credentials_ONE_TIME
```