From d76f1e766b047bd8101078fb850a2d186a3ee5b1 Mon Sep 17 00:00:00 2001 From: byrsapty Date: Thu, 23 Jul 2026 21:00:06 +0300 Subject: [PATCH] Fix the same update-gap bug in step_streams()/step_dashboard() as rules/pipelines/alerts Both only checked title existence, so editing a stream's rules or a dashboard's widgets under the same title silently skipped the update forever. step_dashboard() now diffs the search's queries content (delete+recreate view+search on change - no subscribers/history to lose, unlike alerts or streams); step_streams() diffs rules individually by (field,type,value,inverted) and PUT-updates metadata in place, since delete+recreating a stream would briefly break live message routing. Confirmed live on a local test deployment: a false "changed" positive from search_types list-reordering and from search_servers.json's own query-format inconsistency (bare string vs the {query_string,type} object Graylog actually stores) had to be fixed before step_dashboard() settled into a stable no-op; step_streams() was verified with a full inert rule-add/remove and a metadata-change round-trip against the live API. --- README.md | 32 +++++- README.uk.md | 34 ++++-- dashboards/search_servers.json | 2 +- install-graylog.sh | 204 +++++++++++++++++++++++++++++---- 4 files changed, 238 insertions(+), 34 deletions(-) diff --git a/README.md b/README.md index f1fa019..07463a9 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,14 @@ Docker, applies the firewall (nftables: only 9000/tcp, 514+1514/udp, the pipeline rules, and prints the admin credentials at the end. Both scripts are idempotent — re-running after a failure (or to apply an -update) picks up from where it left off instead of duplicating work. +update) picks up from where it left off instead of duplicating work. This +means detecting content changes, not just presence: editing a rule's +regex, a pipeline's stage list, a stream's matching rules, a dashboard's +widgets, or an alert's threshold and re-running applies the change in +place - it won't silently skip the update just because a same-named +resource already exists. Confirmed live (2026-07-23) for every resource +type this script manages, including a full create/delete round-trip on a +live stream's rules and a dashboard's widgets. ## Ways to run this @@ -54,15 +61,28 @@ workflow runs `pct destroy` - that's always a deliberate, manual step (see under exactly that restricted scope). - A storage on the Proxmox host with content type `vztmpl` enabled, holding (or able to download) a `debian-12-standard` template - default assumed - name: `local-btrfs`. Check with: + name: `local-btrfs`. Check that it's active and actually supports this + content type: ```bash - sudo pveam list local-btrfs # replace with your storage name + pvesm status # all storages; Status column should read "active" + pvesm status --content vztmpl # only storages supporting vztmpl - your --template-storage should be in here + sudo pveam list local-btrfs # the template list itself (can be empty - the script will download one) ``` - If it errors with "storage is disabled" or "does not exist", find the - right one (`sudo pveam list ` for candidates) and pass it via + If your storage isn't listed under `--content vztmpl`, either pick a + different candidate from the list, or enable this content type on it: + `pvesm set --content vztmpl,rootdir,...` (this call + **overwrites the entire** content type list, so include whatever was + already there - check `/etc/pve/storage.cfg` first). If it still errors + with "storage is disabled" or "does not exist", pass the right name via `--template-storage`. - A storage for the container's root filesystem (default assumed: - `EX-Ceph`) with enough free space for `--disk` (default 50GB). + `EX-Ceph`) with content type `rootdir` and enough free space for + `--disk` (default 50GB). Check with: + ```bash + pvesm status --content rootdir # your --rootfs-storage should be in here; Avail column is free space in KiB + ``` + If the default `EX-Ceph` isn't listed or doesn't have enough space, pass + the right name via `--rootfs-storage`. - The IP address, gateway, and VLAN tag (if any) for the network the container will live on - i.e. the same network your RADIUS/NAS servers and network equipment can reach. diff --git a/README.uk.md b/README.uk.md index 9d94fa4..5131f52 100644 --- a/README.uk.md +++ b/README.uk.md @@ -25,7 +25,14 @@ в кінці виводить дані адміністратора. Обидва скрипти ідемпотентні — повторний запуск після збою (або для -оновлення) продовжує з того місця, де зупинився, а не дублює роботу. +оновлення) продовжує з того місця, де зупинився, а не дублює роботу. Це +означає виявлення змін у вмісті, а не лише факту існування: якщо +відредагувати regex у правилі, список стейджів у pipeline, правила +збігу в стрімі, віджети в дашборді чи поріг в алерті — повторний запуск +застосує зміну на місці, а не тихо пропустить оновлення лише тому, що +ресурс з такою самою назвою вже існує. Підтверджено наживо (2026-07-23) +для кожного типу ресурсу, яким керує цей скрипт, включно з повним циклом +створення/видалення правил стріму та віджетів дашборду. ## Варіанти запуску @@ -55,15 +62,28 @@ протестований саме в такому обмеженому обсязі прав). - Storage на хості Proxmox з увімкненим content type `vztmpl`, де вже є (або можна завантажити) template `debian-12-standard` — типова назва за - замовчуванням: `local-btrfs`. Перевірте: + замовчуванням: `local-btrfs`. Перевірте, що він взагалі активний і + підтримує саме цей content type: ```bash - sudo pveam list local-btrfs # замініть на назву вашого storage + pvesm status # усі storage; колонка Status має бути "active" + pvesm status --content vztmpl # тільки ті, що підтримують vztmpl - серед них має бути ваш --template-storage + sudo pveam list local-btrfs # сам список шаблонів (може бути порожнім - скрипт довантажить) ``` - Якщо помилка "storage is disabled" чи "does not exist" — знайдіть - правильний (`sudo pveam list ` для кандидатів) і передайте через - `--template-storage`. + Якщо потрібного storage немає у виводі `--content vztmpl` — або оберіть + інший кандидата зі списку, або додайте йому цей content type: `pvesm set + --content vztmpl,rootdir,...` (цей виклик **перезаписує весь** + список content type, тож перелічіть і ті, що там вже були — гляньте + поточний список в `/etc/pve/storage.cfg` перед цим). Якщо після цього + все ще помилка "storage is disabled" чи "does not exist" — передайте + правильну назву через `--template-storage`. - Storage для кореневої файлової системи контейнера (типово: `EX-Ceph`) з - достатнім вільним місцем під `--disk` (за замовчуванням 50GB). + content type `rootdir` і достатнім вільним місцем під `--disk` (за + замовчуванням 50GB). Перевірте: + ```bash + pvesm status --content rootdir # тут має бути ваш --rootfs-storage; колонка Avail - вільне місце в KiB + ``` + Якщо типового `EX-Ceph` немає у списку або вільного місця замало — + передайте правильну назву через `--rootfs-storage`. - IP-адреса, шлюз і VLAN-тег (якщо є) мережі, де житиме контейнер — тобто та сама мережа, звідки ваші RADIUS/NAS-сервери й мережеве обладнання зможуть до нього достукатись. diff --git a/dashboards/search_servers.json b/dashboards/search_servers.json index e5d51c5..34a990b 100644 --- a/dashboards/search_servers.json +++ b/dashboards/search_servers.json @@ -60,7 +60,7 @@ "type": "pivot", "name": "chart", "timerange": {"type": "relative", "range": 604800}, - "query": "event_type:radius_accounting", + "query": {"type": "elasticsearch", "query_string": "event_type:radius_accounting"}, "streams": ["__SERVERS_STREAM_ID__"], "stream_categories": [], "series": [{"type": "count", "id": "Message count", "field": null}], diff --git a/install-graylog.sh b/install-graylog.sh index 53f94c5..421b7ad 100644 --- a/install-graylog.sh +++ b/install-graylog.sh @@ -390,30 +390,116 @@ step_pipelines() { step_streams() { log "Creating streams from $SCRIPT_DIR/streams/*.json (idempotent)..." - local index_set_id existing + # Same "compare content, don't just check the title exists" fix as + # step_pipeline_rules()/step_pipelines()/step_alerts()/step_dashboard() - a + # stream's title never changes when its rules do, so a plain title-exists + # check would silently skip the update forever. Unlike a dashboard, a + # stream's rules live in their own sub-resource + # (POST/PUT/DELETE /streams/{id}/rules/{ruleId}), not a single comparable + # field - and delete+recreating the whole stream would briefly break live + # message routing (alerts/dashboards go blind until it's reconnected to + # its pipeline). So changed rules are diffed individually - identity is + # (field, type, value, inverted) since desired rules carry no id - missing + # ones are created, extra ones are deleted; title/description/matching_type/ + # index_set_id are PUT-updated in place same as a pipeline's 'source'. + local index_set_id existing_file index_set_id="$(gcurl GET /system/indices/index_sets | python3 -c "import json,sys;d=json.load(sys.stdin);print(next(s['id'] for s in d['index_sets'] if s.get('default')))")" - existing="$(gcurl GET /streams)" + existing_file="$(mktemp)" + gcurl GET /streams > "$existing_file" # $2.. are extra `sed -e` substitution expressions applied on top of the # index-set-id one, so each template can carry however many input-id # placeholders it needs. render_and_create() { local tmpl="$1" title="$2"; shift 2 - local id - id="$(echo "$existing" | python3 -c "import json,sys;d=json.load(sys.stdin);print(next((s['id'] for s in d['streams'] if s['title']=='$title'),''))")" - if [ -n "$id" ]; then - skip "stream '$title' already exists ($id)" - echo "$id" - return - fi local sed_args=(-e "s/__DEFAULT_INDEX_SET_ID__/$index_set_id/") local expr for expr in "$@"; do sed_args+=(-e "$expr"); done - local body - body="$(sed "${sed_args[@]}" "$tmpl")" - id="$(gcurl POST /streams "$body" | python3 -c "import json,sys;print(json.load(sys.stdin)['stream_id'])")" + local desired_file id + desired_file="$(mktemp)" + sed "${sed_args[@]}" "$tmpl" > "$desired_file" + + id="$(python3 -c " +import json +d = json.load(open('$existing_file')) +print(next((s['id'] for s in d['streams'] if s['title'] == '$title'), '')) +")" + + if [ -n "$id" ]; then + local plan_file + plan_file="$(mktemp)" + python3 -c " +import json + +d = json.load(open('$existing_file')) +current = next(s for s in d['streams'] if s['id'] == '$id') +desired = json.load(open('$desired_file'))['entity'] + +meta_keys = ['title', 'description', 'matching_type', 'index_set_id', 'remove_matches_from_default_stream'] +metadata_changed = any(current.get(k) != desired.get(k) for k in meta_keys) +metadata_body = {k: desired[k] for k in meta_keys if k in desired} + +def rule_key(r): + return (r['field'], r['type'], r['value'], r['inverted']) + +current_rules = current.get('rules') or [] +desired_rules = desired.get('rules') or [] +current_by_key = {rule_key(r): r['id'] for r in current_rules} +desired_key_set = {rule_key(r) for r in desired_rules} + +rules_create = [r for r in desired_rules if rule_key(r) not in current_by_key] +rules_delete = [rid for k, rid in current_by_key.items() if k not in desired_key_set] + +json.dump({ + 'metadata_changed': metadata_changed, + 'metadata_body': metadata_body, + 'rules_create': rules_create, + 'rules_delete': rules_delete, +}, open('$plan_file', 'w')) +" + local metadata_changed rules_create_count rules_delete_count + metadata_changed="$(python3 -c "import json;print(json.load(open('$plan_file'))['metadata_changed'])")" + rules_create_count="$(python3 -c "import json;print(len(json.load(open('$plan_file'))['rules_create']))")" + rules_delete_count="$(python3 -c "import json;print(len(json.load(open('$plan_file'))['rules_delete']))")" + + if [ "$metadata_changed" = "True" ]; then + gcurl PUT "/streams/$id" "$(python3 -c "import json;print(json.dumps(json.load(open('$plan_file'))['metadata_body']))")" >/dev/null + fi + + if [ "$rules_delete_count" -gt 0 ]; then + local rid + while IFS= read -r rid; do + [ -n "$rid" ] || continue + gcurl DELETE "/streams/$id/rules/$rid" >/dev/null + done < <(python3 -c "import json;print('\n'.join(json.load(open('$plan_file'))['rules_delete']))") + fi + + if [ "$rules_create_count" -gt 0 ]; then + local rule_body + while IFS= read -r rule_body; do + gcurl POST "/streams/$id/rules" "$rule_body" >/dev/null + done < <(python3 -c " +import json +for r in json.load(open('$plan_file'))['rules_create']: + print(json.dumps(r)) +") + fi + + if [ "$metadata_changed" = "True" ] || [ "$rules_create_count" -gt 0 ] || [ "$rules_delete_count" -gt 0 ]; then + ok "updated stream '$title' ($id) - $rules_create_count rule(s) added, $rules_delete_count removed$([ "$metadata_changed" = "True" ] && echo ", metadata changed")" + else + skip "stream '$title' already up to date ($id)" + fi + + rm -f "$plan_file" "$desired_file" + echo "$id" + return + fi + + id="$(gcurl POST /streams "$(cat "$desired_file")" | python3 -c "import json,sys;print(json.load(sys.stdin)['stream_id'])")" ok "created stream '$title' ($id)" gcurl POST "/streams/$id/resume" "" >/dev/null + rm -f "$desired_file" echo "$id" } @@ -422,6 +508,7 @@ step_streams() { "s/__NETWORK_INPUT_ID_1514__/$NETWORK_INPUT_ID_1514/")" SERVERS_STREAM_ID="$(render_and_create "$SCRIPT_DIR/streams/stream2_servers.json" "Servers" \ "s/__SERVERS_INPUT_ID__/$SERVERS_INPUT_ID/")" + rm -f "$existing_file" } step_connect_pipelines() { @@ -445,28 +532,105 @@ step_dashboard() { # different reader (landing/alerts overview, network-equipment-only, # servers-only) so opening Graylog goes straight to what's relevant # instead of one big mixed-stream page. - local existing_titles - existing_titles="$(gcurl GET /views | python3 -c "import json,sys;print('\n'.join(v['title'] for v in json.load(sys.stdin)['views']))")" + # + # Same "compare content, don't just check the title exists" fix as + # step_pipeline_rules()/step_pipelines()/step_alerts() - a dashboard's + # title never changes when a widget/query in search_*.json does, so a + # plain title-exists check silently skips the update forever. Unlike + # rules/pipelines/alerts there's no safe in-place PUT that keeps a view + # and its search in sync here, so a changed dashboard is deleted and + # recreated instead - low risk since dashboards have no subscribers or + # history the way alerts or streams do. Compares only the 'queries' key + # our search_*.json files author, same reasoning as step_alerts(): GET + # fills in extra defaults (owner, created_at, requires, ...) that never + # appear in our files, so a whole-object compare would always say + # "changed". + local existing_views_file + existing_views_file="$(mktemp)" + gcurl GET /views > "$existing_views_file" - local search_file view_file title id search_id body + local search_file view_file title existing_id existing_search_id desired_search_file current_search_file search_id id body for search_file in "$SCRIPT_DIR"/dashboards/search_*.json; do view_file="${search_file/search_/view_}" [ -f "$view_file" ] || die "missing $view_file for $search_file" title="$(python3 -c "import json;print(json.load(open('$view_file'))['entity']['title'])")" - if echo "$existing_titles" | grep -qx "$title"; then - skip "dashboard '$title' already exists" - continue + + desired_search_file="$(mktemp)" + sed -e "s/__NETWORK_STREAM_ID__/$NETWORK_STREAM_ID/g" -e "s/__SERVERS_STREAM_ID__/$SERVERS_STREAM_ID/g" "$search_file" > "$desired_search_file" + + existing_id="$(python3 -c " +import json +d = json.load(open('$existing_views_file')) +print(next((v['id'] for v in d['views'] if v['title'] == '$title'), '')) +")" + + if [ -n "$existing_id" ]; then + existing_search_id="$(python3 -c " +import json +d = json.load(open('$existing_views_file')) +print(next(v['search_id'] for v in d['views'] if v['id'] == '$existing_id')) +")" + current_search_file="$(mktemp)" + gcurl GET "/views/search/$existing_search_id" > "$current_search_file" + + if python3 -c " +import json, sys + +def sort_search_types(q): + # search_types order isn't semantically meaningful (widgets are matched + # to view widgets by 'id', not position) but Graylog doesn't return them + # in the same order they were created in - confirmed live 2026-07-23: + # this caused a false 'changed' positive for every single dashboard on + # every single run, identical to the list-order bug already worked + # around in step_alerts(). + q = dict(q) + if 'search_types' in q: + q['search_types'] = sorted(q['search_types'], key=lambda st: st.get('id', '')) + return q + +def filter_to(desired_obj, current_obj): + # Keep only the keys/structure our search_*.json files actually author - + # confirmed live 2026-07-23: a 'messages'-type search_type (the message + # list widget on the Overview dashboard) comes back from GET with a pile + # of extra default keys (decorators, fields, filter, name, timerange, + # stream_categories, ...) that never appear in our file, so a whole- + # object compare always said 'changed' for that one dashboard even with + # sort_search_types() already applied - same reasoning as step_alerts(). + if isinstance(desired_obj, dict): + cur = current_obj if isinstance(current_obj, dict) else {} + return {k: filter_to(v, cur.get(k)) for k, v in desired_obj.items()} + if isinstance(desired_obj, list): + cur = current_obj if isinstance(current_obj, list) else [] + return [filter_to(dv, cur[i] if i < len(cur) else None) for i, dv in enumerate(desired_obj)] + return current_obj + +current = json.load(open('$current_search_file')) +desired = json.load(open('$desired_search_file')) +current_q = [sort_search_types(q) for q in current.get('queries', [])] +desired_q = [sort_search_types(q) for q in desired.get('queries', [])] +sys.exit(0 if filter_to(desired_q, current_q) == desired_q else 1) +"; then + skip "dashboard '$title' already up to date ($existing_id)" + rm -f "$desired_search_file" "$current_search_file" + continue + fi + rm -f "$current_search_file" + + gcurl DELETE "/views/$existing_id" >/dev/null + gcurl DELETE "/views/search/$existing_search_id" >/dev/null + ok "removed stale dashboard '$title' ($existing_id) - content changed, recreating" fi - body="$(sed -e "s/__NETWORK_STREAM_ID__/$NETWORK_STREAM_ID/g" -e "s/__SERVERS_STREAM_ID__/$SERVERS_STREAM_ID/g" "$search_file")" - search_id="$(gcurl POST /views/search "$body" | python3 -c "import json,sys;print(json.load(sys.stdin)['id'])")" + search_id="$(gcurl POST /views/search "$(cat "$desired_search_file")" | python3 -c "import json,sys;print(json.load(sys.stdin)['id'])")" [ -n "$search_id" ] || die "search creation failed for dashboard '$title'" + rm -f "$desired_search_file" body="$(sed -e "s/__NETWORK_STREAM_ID__/$NETWORK_STREAM_ID/g" -e "s/__SERVERS_STREAM_ID__/$SERVERS_STREAM_ID/g" -e "s/__SEARCH_ID__/$search_id/" "$view_file")" id="$(gcurl POST /views "$body" | python3 -c "import json,sys;print(json.load(sys.stdin).get('id',''))")" [ -n "$id" ] || die "dashboard '$title' failed to create" ok "created dashboard '$title' ($id)" done + rm -f "$existing_views_file" } step_alerts() {