graylog-deploy/README.md

# Graylog centralized log server — deployment scripts

Deploys a Debian 12 LXC container on Proxmox VE with a Docker Compose stack
(MongoDB + OpenSearch + Graylog) for collecting syslog from network equipment
(Juniper, ZTE OLT, D-Link) and servers (RADIUS, accel-ppp).

## Quick start

Run on the Proxmox host as `claude-deploy` (or any user with the same
restricted sudo rights on `pct`/`pveam` for VMIDs 200-299):

```bash
./create-graylog-lxc.sh \
  --ip 10.254.254.202/24 \
  --gw 10.254.254.235 \
  --vlan 1254 \
  --external-uri http://93.171.241.5:9000/ \
  --discord-webhook "https://discord.com/api/webhooks/xxx/yyy"
```

This is the **only command you need**. It creates the container, installs
Docker, applies the firewall (nftables: only 9000/tcp, 514+1514/udp,
5140/udp open), brings up the stack, creates the Syslog inputs, imports
the pipeline rules, and prints the admin credentials at the end.

Both scripts are idempotent — re-running after a failure (or to apply an
update) picks up from where it left off instead of duplicating work.

## Ways to run this

Four ways to apply this project, from "type one command" to "click a
button in a browser" - pick whichever fits what changed and how much
setup you're willing to do first.

| Way | Run from | What it does | One-time setup | When to use |
|---|---|---|---|---|
| **Manual, full** | terminal on the Proxmox host | `create-graylog-lxc.sh` → creates/repairs the LXC container, installs everything | none | first-ever deploy, or when you'd rather watch it happen |
| **Manual, config-only** | terminal, `pct exec` into the container | re-run `install-graylog.sh` directly | none | quick config-only fix without touching git |
| **CI, config-only** | Forgejo web UI | workflow `deploy.yml` → re-runs `install-graylog.sh` from the latest git commit | `setup-forgejo-runner.sh` *inside the container* (see step 9) | pushed a new rule/alert/dashboard, container already exists |
| **CI, full** | Forgejo web UI | workflow `deploy-from-scratch.yml` → runs `create-graylog-lxc.sh` from the latest git commit | `setup-forgejo-runner.sh` *on the Proxmox host* (see step 9) | one-click recreate after a manual `pct destroy`, no terminal needed |

All four end up running the exact same idempotent scripts - there's no
"CI-only" behavior to diverge from what you'd type by hand. Neither CI
workflow runs `pct destroy` - that's always a deliberate, manual step (see
"CI/CD via Forgejo Actions" below for why).

## Step-by-step deployment from scratch

### 0. Prerequisites

- SSH access to the Proxmox host as a user with sudo rights on `pct`
  create/set/start/stop/exec/status/list and `pveam` update/list/download
  (does **not** need to be root - this whole toolkit was built and tested
  under exactly that restricted scope).
- A storage on the Proxmox host with content type `vztmpl` enabled, holding
  (or able to download) a `debian-12-standard` template - default assumed
  name: `local-btrfs`. Check with:
  ```bash
  sudo pveam list local-btrfs   # replace with your storage name
  ```
  If it errors with "storage is disabled" or "does not exist", find the
  right one (`sudo pveam list <name>` for candidates) and pass it via
  `--template-storage`.
- A storage for the container's root filesystem (default assumed:
  `EX-Ceph`) with enough free space for `--disk` (default 50GB).
- The IP address, gateway, and VLAN tag (if any) for the network the
  container will live on - i.e. the same network your RADIUS/NAS servers
  and network equipment can reach.
- If you need remote access to the Web UI from outside that network: a
  spare external port to DNAT to the container's `9000/tcp` (see the
  "reaching the Web UI from outside" note below - port 9000 collided with
  an existing ClickHouse listener during initial testing, so don't assume
  any specific port is free).
- (Optional) A Discord webhook URL, if you want Critical alerts pushed to
  a channel.

### 1. Get the scripts onto the Proxmox host

From your workstation:
```bash
tar czf - -C /path/to/graylog-deploy . | ssh claude-deploy@<proxmox-host> \
  "mkdir -p ~/graylog-deploy && tar xzf - -C ~/graylog-deploy && chmod +x ~/graylog-deploy/*.sh"
```
(Or `git clone`/`scp` the directory if you keep it in a repo - any method
that gets the whole folder, including `rules/`, `pipelines/`, `streams/`,
`alerts/`, onto the host works.)

### 2. Decide your parameters

Pick, in advance:

| You need | Example | Why |
|---|---|---|
| A free VMID in 200-299 | `210` | or omit `--vmid` to auto-pick the first free one |
| Container IP + CIDR on the management network | `10.254.254.220/24` | must be reachable by every device that will send syslog |
| Gateway on that network | `10.254.254.235` | |
| VLAN tag (if the bridge is trunked) | `1254` | omit `--vlan` if untagged |
| Public URL for the Web UI | `http://<public-ip>:<port>/` | used both in Graylog's config and in Discord alert links |
| Discord webhook (optional) | `https://discord.com/api/webhooks/.../...` | leave unset to skip Discord entirely |

**Check the IP is actually free first** - a duplicate IP on this network
silently causes ARP flapping and intermittent, hard-to-diagnose routing
weirdness (this happened during initial testing: `.201` turned out to
already be in use, causing traffic to randomly land on the wrong host).
A quick way to check without touching anything:
```bash
ssh claude-deploy@<proxmox-host> "sudo /usr/sbin/pct exec <any-running-vmid> -- ping -c2 -W1 <candidate-ip>"
```
If you get replies, that IP is taken - pick another.

### 3. Run `create-graylog-lxc.sh`

```bash
ssh claude-deploy@<proxmox-host>
cd ~/graylog-deploy
./create-graylog-lxc.sh \
  --vmid 210 \
  --ip 10.254.254.220/24 \
  --gw 10.254.254.235 \
  --vlan 1254 \
  --external-uri http://<public-ip>:<port>/ \
  --discord-webhook "https://discord.com/api/webhooks/xxx/yyy"
```

What happens, in order: template check/download → `pct create` → start the
container → wait for network → copy this whole folder into the container
at `/opt/graylog-deploy/` → run `install-graylog.sh` inside it, which
installs Docker, applies the firewall, brings up MongoDB/OpenSearch/Graylog,
waits for it to report healthy, creates the Syslog inputs, imports the
pipeline rules/pipelines/streams, and (if a webhook was given) creates the
Discord notification and the three alert definitions.

**If it stops with an AppArmor error** (`open sysctl
net.ipv4.ip_unprivileged_port_start: permission denied`), that's expected
on some Proxmox setups and needs one manual step from the host's root
account - see "Docker-in-unprivileged-LXC AppArmor block" below. Do that,
then just run the exact same command again; everything already done is
skipped automatically.

Total time for a clean run: a few minutes, mostly waiting for image pulls
and for Graylog to report healthy.

### 4. Read the admin credentials

`install-graylog.sh` prints the generated admin user/password once, at the
end of a successful fresh run, and deletes the one-time credentials file
right after - nothing further to do. If you missed it (or the run died
before reaching that point), it's still sitting at
`/opt/graylog/.admin_credentials_ONE_TIME` until you read it:
```bash
ssh claude-deploy@<proxmox-host> "sudo pct exec 210 -- cat /opt/graylog/.admin_credentials_ONE_TIME"
```
Log into `http://<public-ip>:<port>/` with user `admin` and that password.

### 5. Reach the Web UI from outside the management network (if needed)

If the container's IP isn't directly reachable from where you browse, add
a DNAT rule on your edge router/firewall for `9000/tcp`:
```
-A PREROUTING -d <public-ip>/32 -p tcp -m tcp --dport <port> -j DNAT --to-destination <container-ip>:9000
```
Pick a port that's actually free - during initial setup, port 9000 turned
out to already be serving ClickHouse on the same public IP, and the
response (`Port 9000 is for clickhouse-client program...`) looked like a
Graylog problem for a while before that was found. If you get a
suspicious/unexpected response on the port you chose, suspect a
pre-existing service on that port before suspecting Graylog.

### 6. Point real equipment at the server

| Source | Port | Notes |
|---|---|---|
| Network equipment (switches, OLTs, routers) | **514/udp** | the standard syslog port - almost no gear lets you pick a different one (`logging <host>` on a typical switch always uses 514) |
| Network equipment that *can* target a custom port | 1514/udp | kept as a secondary input, same stream as 514 |
| Servers (RADIUS, accel-ppp, conntrack/kernel messages) | **5140/udp** | |

On each device, this is normally a one-line config change (e.g. on a
Cisco-like CLI: `logging <container-ip>`). No Graylog-side config is
needed per device - the inputs and streams already listen on all three
ports.

### 7. Verify data is actually arriving

1. Web UI → **System → Inputs**: each input shows live "Traffic Last
   Minute". If it stays at 0 after a device should have sent something,
   the problem is network/firewall, not Graylog - confirm with `tcpdump`
   inside the container before touching any Graylog config:
   ```bash
   sudo pct exec 210 -- tcpdump -i eth0 -n udp port 514
   ```
2. Web UI → **Search**, widen the time range (top-left), click any message
   to expand it. If `vendor` / `event_type` fields are populated, a
   pipeline rule matched it. If a message arrives but those fields are
   empty, it reached Graylog fine but no rule recognizes its format yet -
   that's a sign a new device/vendor needs a new rule (see "What's still
   NOT included" below for the process).
3. Web UI → **Streams**: the "Throughput" column shows live msg/s per
   stream.

### 8. (Optional) Trigger a real test alert

See "Verifying a test alert" further down - send one of the known-critical
log lines via `logger` from inside the container and watch it land in
Discord within about a minute.

### 9. (Optional) Set up Forgejo Actions CI

`setup-forgejo-runner.sh` registers a self-hosted Actions runner - it can
run in two different places, one script for both, just different
parameters. See "CI/CD via Forgejo Actions" further down for what each
workflow does; this step is just the one-time runner setup. Both are
optional and independent - set up either, neither, or both.

**Shared first step:** repo → Settings → Actions → Runners → "Create new
Runner" to get a registration token (one-time value, not something to
guess or reuse from another repo - get a fresh one for each runner you
register).

**9a. Runner inside the container** (for `deploy.yml` - config-only
redeploys):
```bash
pct exec <VMID> -- env \
  FORGEJO_URL="https://your-forgejo-instance" \
  FORGEJO_RUNNER_TOKEN="<token>" \
  bash /opt/graylog-deploy/setup-forgejo-runner.sh
```

**9b. Runner on the Proxmox host itself** (for `deploy-from-scratch.yml` -
full recreate). `git` must be installed on the host first (`apt-get
install -y git` - the container has it, the bare host usually doesn't).
`RUNNER_USER=claude-deploy` is required here, not optional: without it the
systemd service runs as root, handing every CI job unrestricted root on
the host instead of claude-deploy's narrowly-scoped sudo:
```bash
sudo env FORGEJO_URL="https://your-forgejo-instance" FORGEJO_RUNNER_TOKEN="<token>" RUNNER_NAME="proxmox-host-runner" RUNNER_LABEL="proxmox-host:host" RUNNER_DIR="/opt/forgejo-runner-host" SERVICE_NAME="forgejo-runner-host" RUNNER_USER="claude-deploy" bash /home/claude-deploy/graylog-deploy/setup-forgejo-runner.sh
```
(One line, no `\` continuations - multi-line pastes with backslashes have
been observed dropping the continuation in some terminals, silently
running `sudo env` with no arguments and none of the variables set.)

**9c. Secrets** the workflows need (repo → Settings → Actions → Secrets):
`GRAYLOG_ADMIN_PASSWORD` and `DISCORD_WEBHOOK_URL`. A third value,
`secrets.GITHUB_TOKEN`, doesn't need creating - Forgejo issues it
automatically per job, scoped to just this repo.

**Removing a runner later** (e.g. the host-level one, if you'd rather not
leave it registered when not actively deploying):
```bash
sudo systemctl disable --now forgejo-runner-host
sudo rm -f /etc/systemd/system/forgejo-runner-host.service
sudo systemctl daemon-reload
sudo rm -rf /opt/forgejo-runner-host
```
then delete it from repo → Settings → Actions → Runners in the UI too.
Re-registering later needs a fresh token from that same page - the old
one won't be reused automatically.

## Parameters

All flags have an environment-variable equivalent (see the top of
`create-graylog-lxc.sh`), so you can also `export MEMORY_MB=16384` etc.
instead of passing flags.

| Flag | Default | Notes |
|---|---|---|
| `--ip` | *(required)* | Static IP + CIDR for the container |
| `--gw` | *(required)* | Gateway IP |
| `--external-uri` | *(required)* | Public URL for the Graylog Web UI (used in `GRAYLOG_HTTP_EXTERNAL_URI`) |
| `--vmid` | first free 200-299 | |
| `--hostname` | `graylog` | |
| `--cores` | `4` | |
| `--memory` | `8192` (MB) | |
| `--swap` | `512` (MB) | |
| `--disk` | `50` (GB) | rootfs size on `--rootfs-storage` |
| `--bridge` | `vmbr0` | |
| `--vlan` | *(none = untagged)* | |
| `--nameserver` | `1.1.1.1` | |
| `--searchdomain` | *(none)* | |
| `--timezone` | `Europe/Kyiv` | |
| `--template-storage` | `local-btrfs` | must have content type `vztmpl` enabled |
| `--rootfs-storage` | `EX-Ceph` | container disk storage |
| `--discord-webhook` | *(none)* | passed through as `DISCORD_WEBHOOK_URL` to the in-container script |

## Dashboards

Three focused dashboards are created automatically instead of one combined
view - each is scoped to what one kind of reader actually needs, so opening
Graylog goes straight to something relevant instead of one big page mixing
network gear, servers, and alerts together:

- **Overview & Alerts** (Dashboards → Overview & Alerts) - the landing
  page. Recent Alerts (last 24h, pulled straight from the "All events"
  stream so you see the actual fired alerts, not just counts), Message
  Volume by Source for the last hour (catches a flood visually before the
  flood alerts even fire), Events by Priority (24h), and Critical Events by
  Type (24h).
- **Network Equipment** (Dashboards → Network Equipment) - scoped to the
  Network Equipment stream only, 7-day window: messages over time by
  event type, vendor breakdown (Juniper vs. BDCOM), event type table, top
  devices by volume.
- **Servers & Sessions** (Dashboards → Servers & Sessions) - scoped to the
  Servers stream only, 7-day window: messages over time by event type,
  event type table, top servers by volume (same view the flood alerts are
  calibrated against), and a RADIUS accounting status breakdown
  (Start/Alive/Stop counts).

Each is built via the Views API (`dashboards/search_<name>.json` +
`dashboards/view_<name>.json` pairs, one pair per dashboard) rather than
Graylog's own widget-builder UI - that UI turned out to be difficult to
drive reliably via browser automation (React `combobox` widgets that don't
respond to plain keyboard/click events without also triggering React's
internal state update), while the REST API accepted the same structure
cleanly once the shape was reverse engineered from an existing dashboard's
JSON. The one part that isn't a plain aggregation pivot - the Recent Alerts
widget, `type: "messages"` instead of `type: "aggregation"` - needed its
own bit of reverse engineering too: the widget-level `sort` field on a
message-list widget must be `[]`, not an object with a `field`/`order`
pair, or Graylog rejects it with a Jackson polymorphism error
(`missing type id property 'type'` - the sort DTO for message widgets
doesn't have any registered subtypes in this Graylog version at all).

If you want to add a widget, either use the Graylog UI directly (a human
using a mouse doesn't hit the automation issue) and then optionally export
the result back into these JSON files, or extend a `search_<name>.json`/
`view_<name>.json` pair by hand - each widget needs a matching
`search_types` entry (in `search_<name>.json`) and `widgets` +
`widget_mapping` + `positions` + `titles.widget` entry (in
`view_<name>.json`) sharing the same ID. `step_dashboard()` in
`install-graylog.sh` picks up any `search_*.json`/`view_*.json` pair
automatically (matched by filename), so a new pair just needs to exist in
the `dashboards/` directory - no script changes required. `__NETWORK_STREAM_ID__`
and `__SERVERS_STREAM_ID__` placeholders are substituted the same way the
alert templates already do it.

## Known environment quirks this script works around

- **Docker-in-unprivileged-LXC AppArmor block**: containers fail with
  `open sysctl net.ipv4.ip_unprivileged_port_start: permission denied`
  unless the Proxmox host admin adds a raw LXC config line. `pct set`
  does not expose this option (checked: `--features` covers
  `nesting`/`keyctl`/`mount`/`fuse`/`mknod`/`force_rw_sys`, nothing for
  AppArmor), so it can't be automated within the plain `claude-deploy`
  sudo scope of `pct create/set/start/stop/exec/status/list`. If you hit
  this, run as root on the host:
  ```bash
  echo "lxc.apparmor.profile: unconfined" >> /etc/pve/lxc/<VMID>.conf
  pct reboot <VMID>
  ```
  then re-run `create-graylog-lxc.sh` (idempotent, will continue from there).

  **Automating this fix (optional, one-time per Proxmox host):** run
  `bootstrap-host.sh` once as root. It installs `fix-lxc-apparmor.sh` to
  `/usr/local/sbin/` (root-owned, `chmod 700` - not writable by
  `claude-deploy`) plus a narrow sudoers rule scoped to exactly that script
  and the VMID 200-299 range:
  ```
  claude-deploy ALL=(root) NOPASSWD: /usr/local/sbin/fix-lxc-apparmor.sh 2[0-9][0-9]
  ```
  Deliberately *not* a broader rule like `tee -a /etc/pve/lxc/2[0-9][0-9].conf`
  or `sh -c '...'`: sudoers only restricts a command's own argv, not
  stdin/heredoc content, so either of those would let the caller append
  *arbitrary* lines to any 200-299 container's config (e.g.
  `lxc.mount.entry` to bind-mount host paths in) - a much wider grant than
  intended. Pinning the exact line inside a fixed, root-owned script is
  what keeps the sudoers rule's effect as narrow as its pattern suggests.
  Once bootstrapped, `create-graylog-lxc.sh` calls
  `sudo -n fix-lxc-apparmor.sh` automatically and only falls back to the
  manual instructions above if that sudoers rule isn't present yet.

  **Revoking it again:** run `cleanup-host.sh` as root once the deployment
  is done - it removes both the sudoers rule and the script, so the
  elevated grant only stands for the duration of an active deployment
  rather than indefinitely. `create-graylog-lxc.sh` degrades cleanly back
  to the manual fallback if it's ever run again without the automation in
  place; re-run `bootstrap-host.sh` whenever you need it back (e.g. before
  recreating a container from scratch).

- **`vm.max_map_count`**: OpenSearch requires >= 262144. This is a
  host-wide kernel parameter, not namespaced per LXC, so it can't be set
  from inside the container either. `install-graylog.sh` only verifies it
  and fails with instructions if it's too low — on this deployment it was
  already 262144 by default, so no action was needed.

- **Docker Hub TLS over IPv6**: on this network, IPv6 paths to
  `registry-1.docker.io` intermittently get intercepted and served a
  mismatched certificate (`*.docker.com`). The install script disables
  IPv6 inside the container to force IPv4-only egress. Image pulls also
  retry up to 5x since even IPv4 occasionally hits a bad edge node.

- **MongoDB version**: Graylog 7.1 requires MongoDB >= 7.0 (some older
  docs still reference 6.0.x — don't trust cached documentation over
  what the running server actually reports).

- **RFC3164 syslog timestamps have no timezone - Graylog assumes UTC by
  default**: most network gear and accel-ppp send classic RFC3164 syslog
  (`Jul 22 09:15:13`, no year, no offset). Without an explicit `timezone`
  setting on the input, Graylog stores that bare timestamp as if it were
  already UTC - so a device logging in local Kyiv time (UTC+3) shows up
  3 hours in the future in Graylog. Confirmed live: a raw test packet with
  `Jul 22 09:15:13` was stored as `09:15:13Z` (wrong) until each Syslog UDP
  input's `timezone` config was set to the container's own timezone
  (`Europe/Kyiv` here); afterwards it correctly stored as `06:15:13Z`
  (`09:15:13` Kyiv = `06:15:13` UTC). `step_inputs()` in
  `install-graylog.sh` sets this automatically from `/etc/timezone` for
  every input it creates, and self-heals it on existing inputs that
  predate this fix.

- **The built-in `admin` user's displayed timezone is a separate setting
  from the input-level fix above**: even after the RFC3164 fix, the Web UI
  can still show times in UTC for the read-only built-in `admin` account.
  That account's timezone is `read_only: true` and **cannot** be changed
  via `PUT /api/users/admin` (confirmed live - fails with `"state should
  be: hexString has 24 characters"`, since that endpoint isn't meant for
  the special built-in account). It can only be set server-side via the
  `root_timezone` config option, i.e. Docker's `GRAYLOG_ROOT_TIMEZONE` env
  var. `step_compose_files()` sets this from `/etc/timezone` for fresh
  installs and self-heals it into any pre-existing `.env` that predates
  this fix; `docker-compose.yml` passes it through to the `graylog`
  service. A `docker compose up -d` re-run picks up the change and
  recreates the container automatically.

- **Network gear sends syslog to port 514, not a custom port**: most
  switches/OLTs (confirmed live with a BDCOM S5612) only support
  `logging <host>`, which always uses the standard UDP/514, with no way to
  point at a different port. There are therefore **two** "network
  equipment" inputs - 514 (what devices actually use) and 1514 (kept for
  any gear that *can* target a custom port) - both feeding the same
  "Network Equipment" stream (`matching_type: OR`). If you add new
  equipment and it doesn't show up, check with `tcpdump -i eth0 udp port
  514` inside the container before assuming the pipeline rules are wrong.

- **nftables must never `flush ruleset`**: an early version of this
  firewall step used `flush ruleset`, which also wipes Docker's own
  iptables-nft-managed nat/filter tables (`DOCKER`, `DOCKER-USER`, etc.),
  breaking container port publishing the next time `docker compose up`
  needs to add a rule (confirmed live - had to recover with a full
  `docker compose down && up`). `nftables.conf` here only ever does
  `add table inet filter` + `flush table inet filter`, which is scoped to
  just that one table and safe regardless of boot/service order relative
  to `docker.service`.

## Alerting and Discord notifications

Six alerts are wired up out of the box (everything else stays quiet on
purpose - routine auth failures, single dropped-session events, etc. are
parsed and searchable but never page anyone):

| Alert | Fires on | Priority |
|---|---|---|
| RADIUS server unreachable | `radius: server(N) not responding` or `radius: no available servers` (verified strings from accel-ppp source, `radius/req.c`) | High |
| conntrack table full (packet loss) | `nf_conntrack: table full, dropping packet` (standard Linux kernel message - active, ongoing packet loss) | High |
| Unrecognized critical-severity syslog | Any message (any vendor, either stream) with RFC5424/3164 syslog severity Emergency/Alert/Critical (0-2) that no specific pipeline rule already classified | Medium |
| Juniper chassis hardware alarm | `CHASSISD_SNMP_TRAP`/`CHASSISD_SNMP_TRAP6` (over temperature, fan, power supply, etc.) - confirmed live against a real `VC.MYRONIVKA` chassis | High |
| Abnormal message volume from one server | A single server in the Servers stream sends more than 150,000 messages in a 10-minute window - see "Message-volume (flood) alerts" below | Medium |
| Abnormal syslog volume from network equipment | A single device in the Network Equipment stream sends more than 500 messages in a 5-minute window - see "Message-volume (flood) alerts" below | Medium |
| Repeated accel-ppp error/warning from one server | The same `event_type` (from `error:`/`warn:` accel-ppp lines) recurs more than 5 times in 5 minutes from one server - catches a stuck session or ongoing condition rather than a single transient warning. Threshold confirmed live against a real incident: `can't determine router address` repeating 2,746 times over ~4 hours for two specific interfaces | Medium |

That third one is the "universal network equipment problem" catch-all: it
doesn't depend on knowing any vendor's specific message format, just the
standard syslog severity level every reasonable device already sends.

### Message-volume (flood) alerts

The last two alerts protect against a single misbehaving source silently
filling the retention window's disk budget - a log loop, a retry storm, or
a debug-level setting left on by accident. They group by `gl2_remote_ip`
(aggregation-v1, `count() > threshold`), so each *individual* source is
compared against its own volume, not the whole stream's total.

The thresholds are not guessed - they were calibrated live on 2026-07-22
against real traffic, via a Views API pivot search grouped by
`gl2_remote_ip`:
- The one active accel-ppp/RADIUS server was steadily sending **~46,000-48,000
  messages per 10 minutes** (~278k/hour) under normal load. The Servers
  stream threshold (150,000/10min) gives roughly 3x headroom above that.
- The one active network device was steadily sending **~30-60 messages per
  5 minutes** (~360/hour). The Network Equipment stream threshold
  (500/5min) gives roughly 10x headroom above that.

These are starting points based on partial rollout (1 server + 1 device
active at calibration time). Revisit both thresholds once more of the
planned ~10-15 servers and ~10-20 switches/OLTs are sending real traffic -
what looks like 3x headroom today could be too tight or too loose once
every server's individual baseline is known.

**A server's first-ever Vector startup can trip this once, harmlessly** -
confirmed live on 2026-07-23 when a second NAS server (EX-NAS-1-2) came
online: Vector's file source reads existing log content from the
beginning on a fresh run (no checkpoint yet), so a busy accel-ppp log
already containing hours of routine DHCP/session churn gets replayed as a
single burst - 430,982 messages in one 10-minute window in that case, all
ordinary (`DHCPv4 Ack`, `ipoe stop/remove`), all timestamped right around
the same moment. Not a loop, not a misconfiguration - just backlog replay
on cold start. Check whether it's settled with the same per-source query
below; if volume stays elevated past the first window or two, treat it as
a real incident instead.

Check current per-source volume any time with a query like:
```
gl2_remote_ip:<ip>
```
over a fixed time range in the Search page, or reuse the same pivot-search
approach (grouped by `gl2_remote_ip`, `count()` series) via the Views API
if you want exact numbers instead of eyeballing a graph.

Also parsed (searchable, but not alerted on since they're routine/expected
volume, not incidents by themselves):
- accel-ppp: PPP authentication failed (`ppp_auth.c`)
- Standalone FreeRADIUS: `Auth: (n) Login OK: [user] (from client X port P)` /
  `Auth: Login incorrect: [user] (from client X port P)` (default `auth_log`
  format)

## Session correlation (accel-ppp subscriber sessions)

Every accel-ppp log line for a given subscriber session - RADIUS
Access-Request (auth attempt), Accounting-Request (start/interim/stop),
DHCP discover/offer/request/ack, ipoe session create/start/finish/terminate
- gets tagged with the same `accelppp_interface` field (the `vlanNNNN.NNN`
interface name accel-ppp itself uses per subscriber). This works even for
message types with no other structured fields at all, via a fallback rule
(`accelppp_interface_tag`) that only tags lines no more specific rule
already classified.

To see a subscriber's full session lifecycle in one query, search:
```
accelppp_interface:"vlan1779.124"
```
sorted by time (default). This surfaces the DHCP handshake, the RADIUS
auth/accounting exchange, and the eventual termination as one chronological
list, instead of grep-ing for the interface name across raw text.

RADIUS Access-Request and Accounting-Request lines additionally get three
richer correlation fields extracted directly from the RADIUS AVPs:
- `radius_session_id` - accel-ppp's `Acct-Session-Id`, stable for the
  entire session
- `calling_station_id` - the subscriber's MAC address
- `radius_username` - the subscriber's login (accel-ppp's `User-Name`,
  format `<vlan>:<qinq>` in this deployment)

These are useful when starting from a support ticket that has a MAC address
or username but not the interface name, e.g.:
```
calling_station_id:"48:8f:5a:a4:f9:ba"
```
Confirmed live on 2026-07-22 against real EX-NAS-1-1 traffic: a single
`accelppp_interface` value correctly tied together a DHCPv4 Ack, a DHCPv4
Request, a RADIUS Accounting-Response, and a RADIUS Accounting-Request, all
belonging to the same subscriber session.

To wire up Discord, pass `--discord-webhook` (or set `DISCORD_WEBHOOK_URL`)
when running `create-graylog-lxc.sh`. Under the hood this uses Graylog's
built-in **Slack** notification type pointed at
`<your-webhook-url>/slack` - Discord's Slack-compatibility endpoint - so no
extra converter service is needed. The message template shows the event
title/description plus the source, sending IP, and raw text of every
matched message:
```
*${event_definition_title}*
${event_definition_description}
${event.message}
${if backlog}${foreach backlog message}• `${message.source}` (IP: ${message.fields.gl2_remote_ip}): ${message.message}
${end}${end}
```
`${event.message}` is Graylog's own auto-generated event summary - for
plain critical alerts it just duplicates the title, but for the two flood
alerts (grouped by `gl2_remote_ip`) this is where the specific source IP
and the actual `count()` value show up, e.g.
`WARNING: ...: 93.171.243.4 - count()=278474.0`.

`gl2_remote_ip` is a field Graylog attaches automatically to every message
based on the actual UDP packet's source address, regardless of what
hostname the device itself claims in the syslog `source` field.

### Searching by IP address

Every message is searchable by the sending device's real IP via the same
`gl2_remote_ip` field, in the Search page query bar:
```
gl2_remote_ip:93.171.243.4
```
`source:<value>` also works, but only matches if the device's self-reported
hostname was used (some equipment sends its actual IP as the hostname,
others send a configured name) - `gl2_remote_ip` is the reliable one since
it's derived from the packet itself, not device-supplied data.

### Verifying a test alert

```bash
# from inside the container, simulating each trigger:
docker exec -it graylog-server bash
logger -n 127.0.0.1 -P 5140 -d 'radius: server(1) not responding'
logger -n 127.0.0.1 -P 5140 -d 'kernel: nf_conntrack: table full, dropping packet'
```
The scheduler checks every 60s, so the Discord message can take up to a
minute to arrive. Check Graylog's Alerts page and the configured Discord
channel.

## CI/CD via Forgejo Actions

This project's Forgejo repo (`git.zotac.keenetic.link/zotac/graylog-deploy`)
has two manually-triggered workflows, each with its own runner (see step 9
above for setup). Both just automate scripts already described elsewhere
in this README - there's nothing CI-specific about their actual behavior.

### `deploy.yml` — config-only redeploy

Clones the repo fresh and re-runs `install-graylog.sh` inside the
container. Use this after pushing a new rule/alert/dashboard/pipeline
change to an already-existing container.

**Running it:** repo → **Actions** tab → **Deploy Graylog config** in the
left sidebar → **Run workflow** button → confirm.

- **The runner lives inside the Graylog container itself** (VMID 200,
  `self-hosted:host` label, installed at `/usr/local/bin/forgejo-runner`,
  running as a systemd service). No new SSH keys or cross-host access were
  needed - the workflow just clones the repo into `/tmp/graylog-deploy-ci`
  and runs the script locally, exactly like a human operator would.

### `deploy-from-scratch.yml` — full recreate

Clones the repo fresh and re-runs `create-graylog-lxc.sh` **on the Proxmox
host**. Use this after a manual `pct destroy` (or on a container that was
never created), when you'd rather click a button than type the full
`create-graylog-lxc.sh` invocation.

**Running it:** repo → **Actions** tab → **Deploy Graylog from scratch
(host-level)** in the left sidebar → **Run workflow** button → a `cores`
field appears (default `4`) - override it or leave as-is → confirm. Only
takes effect when the container is actually being created fresh; a no-op
on one that already exists (same rule as the `--cores` flag everywhere
else in this project - see "Parameters" above).

- **The runner lives on the Proxmox host itself**, running as
  `claude-deploy` (not root - see step 9b above for why that matters),
  registered with a distinct label (`proxmox-host:host`) so it only picks
  up jobs meant for this workflow, not `deploy.yml`'s.
- **`pct destroy` is deliberately NOT part of this workflow.** Recreating
  infrastructure automatically is fine because `create-graylog-lxc.sh` is
  idempotent (repairs in place if the container already exists, creates
  fresh if it doesn't); destroying it is irreversible data loss and stays
  a human typing the command on purpose. A CI button that can destroy
  production with one accidental click was considered and rejected.
  `pct destroy` isn't even in `claude-deploy`'s sudo scope, on purpose.
- **`git` must be installed on the bare host** - unlike the container
  (which has it from the Debian 12 template), a fresh Proxmox host
  usually doesn't (`apt-get install -y git`, one-time, root).

### Common to both workflows

- **Trigger is manual on purpose** (`workflow_dispatch` only, no
  `on: push`) - these drive a production monitoring system, so a human
  reviews the diff and clicks "Run workflow" rather than every push
  silently redeploying.
- **No `actions/checkout`** - that action needs Node.js, which neither
  runner has and shouldn't need just for CI. Both workflows do a plain
  `git clone --depth 1` instead.
- **The clone is authenticated with `secrets.GITHUB_TOKEN`** - a
  short-lived token Forgejo creates automatically at the start of each
  workflow run and destroys when it finishes, scoped to only this repo
  (confirmed against Forgejo's own docs: it works even when the repo is
  private, and any attempt to use it against a different repo returns
  404). Nothing to configure - it's provided by the platform, not a secret
  this project manages. This is what keeps the clone working if the repo
  is later made private.
- **Secrets** (`GRAYLOG_ADMIN_PASSWORD`, `DISCORD_WEBHOOK_URL`) are stored
  as repo-level Forgejo Actions secrets, not in any tracked file.
  `GRAYLOG_EXTERNAL_URI` isn't secret (it's the public Web UI address) so
  it's inlined directly in both workflows.

## What's still NOT included

- **D-Link switch parsing** — no sample logs were available.
- **ZTE OLT ONU online/offline + optical power alarms** — no sample
  logs were available (only BDCOM OLT-style CLI logs were provided:
  privilege-mode/logout/ARP-move/config-write, which *are* parsed).
- **Plain RADIUS Access-Accept/Reject from a specific real deployment** —
  the standalone-FreeRADIUS rule above is based on FreeRADIUS's documented
  default log format, not a sample from this network's actual RADIUS boxes
  (no SSH access to them was available; the pattern is well-established
  upstream, but verify it against a real log line when one is available).

To fill these gaps: provide a few real raw log lines per source (D-Link
syslog, ZTE OLT ONU up/down + optical alarms) and the equivalent
`rules/*.json` + alert definitions can be added the same way the existing
ones were built.

## File layout

```
create-graylog-lxc.sh        # run on the Proxmox host - creates/repairs the LXC container
install-graylog.sh           # run inside the container (invoked automatically)
bootstrap-host.sh            # one-time, root, Proxmox host - automates the AppArmor fix
cleanup-host.sh               # reverses bootstrap-host.sh
fix-lxc-apparmor.sh           # installed by bootstrap-host.sh, not run directly
setup-forgejo-runner.sh       # registers a CI runner - in the container or on the host
docker-compose.yml            # MongoDB + OpenSearch + Graylog stack definition
nftables.conf                 # firewall ruleset applied inside the container
rules/*.json                  # Graylog Pipeline Rule definitions (imported via API)
pipelines/*.json              # Graylog Pipeline definitions referencing the rules
streams/*.json                # Graylog Stream definitions (routes by input port)
alerts/*.json                 # Event Definitions + the Discord notification template
dashboards/search_*.json      # Views API search objects, one pair per dashboard
dashboards/view_*.json        # Views API dashboard/widget layout, paired with search_*.json
.forgejo/workflows/deploy.yml               # CI: config-only redeploy (in-container runner)
.forgejo/workflows/deploy-from-scratch.yml  # CI: full recreate (host-level runner)
```

Inside the container, everything lives under `/opt/graylog/`
(`docker-compose.yml`, `.env` with secrets) and `/opt/graylog-deploy/`
(a copy of this repo, used for re-running the install idempotently).

## Credentials

`install-graylog.sh` generates `GRAYLOG_PASSWORD_SECRET` and an admin
password on first run, writing the admin password once to
`/opt/graylog/.admin_credentials_ONE_TIME` inside the container. The
script itself prints it and deletes the file automatically at the end of
a successful run - store it in your password manager then. If the run
died before reaching that point, read and remove the file by hand:
```bash
pct exec <VMID> -- cat /opt/graylog/.admin_credentials_ONE_TIME
pct exec <VMID> -- rm /opt/graylog/.admin_credentials_ONE_TIME
```

**The admin password itself** is random by default, but `GRAYLOG_ADMIN_PASSWORD`
(env var, same one `create-graylog-lxc.sh` forwards through) can pin it to
a known value instead - set on a fresh install, that value becomes the
actual password, not just a fallback. Two different roles for the same
variable depending on when it's set:
- **On a fresh install** (`.env` doesn't exist yet): if set, becomes the
  real password instead of a random one. If unset, random as before.
- **On any later re-run** (`.env` already exists, one-time file already
  consumed by an earlier successful run): required, must match whatever
  password is already configured - there's no other way for the script to
  authenticate to the running instance.

This matters most for CI: pin a password once via the `GRAYLOG_ADMIN_PASSWORD`
Forgejo secret, and it never goes stale - the same value is correct at
creation time and on every re-run after, instead of having to notice a
fresh install generated a new random password and update the secret to
match (confirmed live: a `deploy-from-scratch.yml` re-run against an
already-initialized Graylog with a stale secret died at
`resolve_admin_password` with "Cannot find admin password").