Fix the same update-gap bug in step_streams()/step_dashboard() as rules/pipelines/alerts

Both only checked title existence, so editing a stream's rules or a dashboard's widgets under the same title silently skipped the update forever. step_dashboard() now diffs the search's queries content (delete+recreate view+search on change - no subscribers/history to lose, unlike alerts or streams); step_streams() diffs rules individually by (field,type,value,inverted) and PUT-updates metadata in place, since delete+recreating a stream would briefly break live message routing. Confirmed live on a local test deployment: a false "changed" positive from search_types list-reordering and from search_servers.json's own query-format inconsistency (bare string vs the {query_string,type} object Graylog actually stores) had to be fixed before step_dashboard() settled into a stable no-op; step_streams() was verified with a full inert rule-add/remove and a metadata-change round-trip against the live API.
2026-07-23 21:00:06 +03:00 · 2026-07-23 21:00:06 +03:00 · d76f1e766b
commit d76f1e766b
parent 32f2a922e8
4 changed files with 238 additions and 34 deletions
--- a/README.md
+++ b/README.md
@ -24,7 +24,14 @@ Docker, applies the firewall (nftables: only 9000/tcp, 514+1514/udp,
 the pipeline rules, and prints the admin credentials at the end.

 Both scripts are idempotent — re-running after a failure (or to apply an
-update) picks up from where it left off instead of duplicating work.
+update) picks up from where it left off instead of duplicating work. This
+means detecting content changes, not just presence: editing a rule's
+regex, a pipeline's stage list, a stream's matching rules, a dashboard's
+widgets, or an alert's threshold and re-running applies the change in
+place - it won't silently skip the update just because a same-named
+resource already exists. Confirmed live (2026-07-23) for every resource
+type this script manages, including a full create/delete round-trip on a
+live stream's rules and a dashboard's widgets.

 ## Ways to run this

@ -54,15 +61,28 @@ workflow runs `pct destroy` - that's always a deliberate, manual step (see
  under exactly that restricted scope).
 - A storage on the Proxmox host with content type `vztmpl` enabled, holding
  (or able to download) a `debian-12-standard` template - default assumed
-  name: `local-btrfs`. Check with:
+  name: `local-btrfs`. Check that it's active and actually supports this
+  content type:
  ```bash
-  sudo pveam list local-btrfs   # replace with your storage name
+  pvesm status                    # all storages; Status column should read "active"
+  pvesm status --content vztmpl   # only storages supporting vztmpl - your --template-storage should be in here
+  sudo pveam list local-btrfs     # the template list itself (can be empty - the script will download one)
  ```
-  If it errors with "storage is disabled" or "does not exist", find the
-  right one (`sudo pveam list <name>` for candidates) and pass it via
+  If your storage isn't listed under `--content vztmpl`, either pick a
+  different candidate from the list, or enable this content type on it:
+  `pvesm set <storage> --content vztmpl,rootdir,...` (this call
+  **overwrites the entire** content type list, so include whatever was
+  already there - check `/etc/pve/storage.cfg` first). If it still errors
+  with "storage is disabled" or "does not exist", pass the right name via
  `--template-storage`.
 - A storage for the container's root filesystem (default assumed:
-  `EX-Ceph`) with enough free space for `--disk` (default 50GB).
+  `EX-Ceph`) with content type `rootdir` and enough free space for
+  `--disk` (default 50GB). Check with:
+  ```bash
+  pvesm status --content rootdir   # your --rootfs-storage should be in here; Avail column is free space in KiB
+  ```
+  If the default `EX-Ceph` isn't listed or doesn't have enough space, pass
+  the right name via `--rootfs-storage`.
 - The IP address, gateway, and VLAN tag (if any) for the network the
  container will live on - i.e. the same network your RADIUS/NAS servers
  and network equipment can reach.
--- a/README.uk.md
+++ b/README.uk.md
@ -25,7 +25,14 @@
 в кінці виводить дані адміністратора.

 Обидва скрипти ідемпотентні — повторний запуск після збою (або для
-оновлення) продовжує з того місця, де зупинився, а не дублює роботу.
+оновлення) продовжує з того місця, де зупинився, а не дублює роботу. Це
+означає виявлення змін у вмісті, а не лише факту існування: якщо
+відредагувати regex у правилі, список стейджів у pipeline, правила
+збігу в стрімі, віджети в дашборді чи поріг в алерті — повторний запуск
+застосує зміну на місці, а не тихо пропустить оновлення лише тому, що
+ресурс з такою самою назвою вже існує. Підтверджено наживо (2026-07-23)
+для кожного типу ресурсу, яким керує цей скрипт, включно з повним циклом
+створення/видалення правил стріму та віджетів дашборду.

 ## Варіанти запуску

@ -55,15 +62,28 @@
  протестований саме в такому обмеженому обсязі прав).
 - Storage на хості Proxmox з увімкненим content type `vztmpl`, де вже є
  (або можна завантажити) template `debian-12-standard` — типова назва за
-  замовчуванням: `local-btrfs`. Перевірте:
+  замовчуванням: `local-btrfs`. Перевірте, що він взагалі активний і
+  підтримує саме цей content type:
  ```bash
-  sudo pveam list local-btrfs   # замініть на назву вашого storage
+  pvesm status                    # усі storage; колонка Status має бути "active"
+  pvesm status --content vztmpl   # тільки ті, що підтримують vztmpl - серед них має бути ваш --template-storage
+  sudo pveam list local-btrfs     # сам список шаблонів (може бути порожнім - скрипт довантажить)
  ```
-  Якщо помилка "storage is disabled" чи "does not exist" — знайдіть
-  правильний (`sudo pveam list <name>` для кандидатів) і передайте через
-  `--template-storage`.
+  Якщо потрібного storage немає у виводі `--content vztmpl` — або оберіть
+  інший кандидата зі списку, або додайте йому цей content type: `pvesm set
+  <storage> --content vztmpl,rootdir,...` (цей виклик **перезаписує весь**
+  список content type, тож перелічіть і ті, що там вже були — гляньте
+  поточний список в `/etc/pve/storage.cfg` перед цим). Якщо після цього
+  все ще помилка "storage is disabled" чи "does not exist" — передайте
+  правильну назву через `--template-storage`.
 - Storage для кореневої файлової системи контейнера (типово: `EX-Ceph`) з
-  достатнім вільним місцем під `--disk` (за замовчуванням 50GB).
+  content type `rootdir` і достатнім вільним місцем під `--disk` (за
+  замовчуванням 50GB). Перевірте:
+  ```bash
+  pvesm status --content rootdir   # тут має бути ваш --rootfs-storage; колонка Avail - вільне місце в KiB
+  ```
+  Якщо типового `EX-Ceph` немає у списку або вільного місця замало —
+  передайте правильну назву через `--rootfs-storage`.
 - IP-адреса, шлюз і VLAN-тег (якщо є) мережі, де житиме контейнер — тобто
  та сама мережа, звідки ваші RADIUS/NAS-сервери й мережеве обладнання
  зможуть до нього достукатись.
--- a/dashboards/search_servers.json
+++ b/dashboards/search_servers.json
@ -60,7 +60,7 @@
          "type": "pivot",
          "name": "chart",
          "timerange": {"type": "relative", "range": 604800},
-          "query": "event_type:radius_accounting",
+          "query": {"type": "elasticsearch", "query_string": "event_type:radius_accounting"},
          "streams": ["__SERVERS_STREAM_ID__"],
          "stream_categories": [],
          "series": [{"type": "count", "id": "Message count", "field": null}],
--- a/install-graylog.sh
+++ b/install-graylog.sh
@ -390,30 +390,116 @@ step_pipelines() {

 step_streams() {
  log "Creating streams from $SCRIPT_DIR/streams/*.json (idempotent)..."
-  local index_set_id existing
+  # Same "compare content, don't just check the title exists" fix as
+  # step_pipeline_rules()/step_pipelines()/step_alerts()/step_dashboard() - a
+  # stream's title never changes when its rules do, so a plain title-exists
+  # check would silently skip the update forever. Unlike a dashboard, a
+  # stream's rules live in their own sub-resource
+  # (POST/PUT/DELETE /streams/{id}/rules/{ruleId}), not a single comparable
+  # field - and delete+recreating the whole stream would briefly break live
+  # message routing (alerts/dashboards go blind until it's reconnected to
+  # its pipeline). So changed rules are diffed individually - identity is
+  # (field, type, value, inverted) since desired rules carry no id - missing
+  # ones are created, extra ones are deleted; title/description/matching_type/
+  # index_set_id are PUT-updated in place same as a pipeline's 'source'.
+  local index_set_id existing_file
  index_set_id="$(gcurl GET /system/indices/index_sets | python3 -c "import json,sys;d=json.load(sys.stdin);print(next(s['id'] for s in d['index_sets'] if s.get('default')))")"
-  existing="$(gcurl GET /streams)"
+  existing_file="$(mktemp)"
+  gcurl GET /streams > "$existing_file"

  # $2.. are extra `sed -e` substitution expressions applied on top of the
  # index-set-id one, so each template can carry however many input-id
  # placeholders it needs.
  render_and_create() {
    local tmpl="$1" title="$2"; shift 2
-    local id
-    id="$(echo "$existing" | python3 -c "import json,sys;d=json.load(sys.stdin);print(next((s['id'] for s in d['streams'] if s['title']=='$title'),''))")"
-    if [ -n "$id" ]; then
-      skip "stream '$title' already exists ($id)"
-      echo "$id"
-      return
-    fi
    local sed_args=(-e "s/__DEFAULT_INDEX_SET_ID__/$index_set_id/")
    local expr
    for expr in "$@"; do sed_args+=(-e "$expr"); done
-    local body
-    body="$(sed "${sed_args[@]}" "$tmpl")"
-    id="$(gcurl POST /streams "$body" | python3 -c "import json,sys;print(json.load(sys.stdin)['stream_id'])")"
+    local desired_file id
+    desired_file="$(mktemp)"
+    sed "${sed_args[@]}" "$tmpl" > "$desired_file"
+
+    id="$(python3 -c "
+import json
+d = json.load(open('$existing_file'))
+print(next((s['id'] for s in d['streams'] if s['title'] == '$title'), ''))
+")"
+
+    if [ -n "$id" ]; then
+      local plan_file
+      plan_file="$(mktemp)"
+      python3 -c "
+import json
+
+d = json.load(open('$existing_file'))
+current = next(s for s in d['streams'] if s['id'] == '$id')
+desired = json.load(open('$desired_file'))['entity']
+
+meta_keys = ['title', 'description', 'matching_type', 'index_set_id', 'remove_matches_from_default_stream']
+metadata_changed = any(current.get(k) != desired.get(k) for k in meta_keys)
+metadata_body = {k: desired[k] for k in meta_keys if k in desired}
+
+def rule_key(r):
+    return (r['field'], r['type'], r['value'], r['inverted'])
+
+current_rules = current.get('rules') or []
+desired_rules = desired.get('rules') or []
+current_by_key = {rule_key(r): r['id'] for r in current_rules}
+desired_key_set = {rule_key(r) for r in desired_rules}
+
+rules_create = [r for r in desired_rules if rule_key(r) not in current_by_key]
+rules_delete = [rid for k, rid in current_by_key.items() if k not in desired_key_set]
+
+json.dump({
+    'metadata_changed': metadata_changed,
+    'metadata_body': metadata_body,
+    'rules_create': rules_create,
+    'rules_delete': rules_delete,
+}, open('$plan_file', 'w'))
+"
+      local metadata_changed rules_create_count rules_delete_count
+      metadata_changed="$(python3 -c "import json;print(json.load(open('$plan_file'))['metadata_changed'])")"
+      rules_create_count="$(python3 -c "import json;print(len(json.load(open('$plan_file'))['rules_create']))")"
+      rules_delete_count="$(python3 -c "import json;print(len(json.load(open('$plan_file'))['rules_delete']))")"
+
+      if [ "$metadata_changed" = "True" ]; then
+        gcurl PUT "/streams/$id" "$(python3 -c "import json;print(json.dumps(json.load(open('$plan_file'))['metadata_body']))")" >/dev/null
+      fi
+
+      if [ "$rules_delete_count" -gt 0 ]; then
+        local rid
+        while IFS= read -r rid; do
+          [ -n "$rid" ] || continue
+          gcurl DELETE "/streams/$id/rules/$rid" >/dev/null
+        done < <(python3 -c "import json;print('\n'.join(json.load(open('$plan_file'))['rules_delete']))")
+      fi
+
+      if [ "$rules_create_count" -gt 0 ]; then
+        local rule_body
+        while IFS= read -r rule_body; do
+          gcurl POST "/streams/$id/rules" "$rule_body" >/dev/null
+        done < <(python3 -c "
+import json
+for r in json.load(open('$plan_file'))['rules_create']:
+    print(json.dumps(r))
+")
+      fi
+
+      if [ "$metadata_changed" = "True" ] || [ "$rules_create_count" -gt 0 ] || [ "$rules_delete_count" -gt 0 ]; then
+        ok "updated stream '$title' ($id) - $rules_create_count rule(s) added, $rules_delete_count removed$([ "$metadata_changed" = "True" ] && echo ", metadata changed")"
+      else
+        skip "stream '$title' already up to date ($id)"
+      fi
+
+      rm -f "$plan_file" "$desired_file"
+      echo "$id"
+      return
+    fi
+
+    id="$(gcurl POST /streams "$(cat "$desired_file")" | python3 -c "import json,sys;print(json.load(sys.stdin)['stream_id'])")"
    ok "created stream '$title' ($id)"
    gcurl POST "/streams/$id/resume" "" >/dev/null
+    rm -f "$desired_file"
    echo "$id"
  }

@ -422,6 +508,7 @@ step_streams() {
    "s/__NETWORK_INPUT_ID_1514__/$NETWORK_INPUT_ID_1514/")"
  SERVERS_STREAM_ID="$(render_and_create "$SCRIPT_DIR/streams/stream2_servers.json" "Servers" \
    "s/__SERVERS_INPUT_ID__/$SERVERS_INPUT_ID/")"
+  rm -f "$existing_file"
 }

 step_connect_pipelines() {
@ -445,28 +532,105 @@ step_dashboard() {
  # different reader (landing/alerts overview, network-equipment-only,
  # servers-only) so opening Graylog goes straight to what's relevant
  # instead of one big mixed-stream page.
-  local existing_titles
-  existing_titles="$(gcurl GET /views | python3 -c "import json,sys;print('\n'.join(v['title'] for v in json.load(sys.stdin)['views']))")"
+  #
+  # Same "compare content, don't just check the title exists" fix as
+  # step_pipeline_rules()/step_pipelines()/step_alerts() - a dashboard's
+  # title never changes when a widget/query in search_*.json does, so a
+  # plain title-exists check silently skips the update forever. Unlike
+  # rules/pipelines/alerts there's no safe in-place PUT that keeps a view
+  # and its search in sync here, so a changed dashboard is deleted and
+  # recreated instead - low risk since dashboards have no subscribers or
+  # history the way alerts or streams do. Compares only the 'queries' key
+  # our search_*.json files author, same reasoning as step_alerts(): GET
+  # fills in extra defaults (owner, created_at, requires, ...) that never
+  # appear in our files, so a whole-object compare would always say
+  # "changed".
+  local existing_views_file
+  existing_views_file="$(mktemp)"
+  gcurl GET /views > "$existing_views_file"

-  local search_file view_file title id search_id body
+  local search_file view_file title existing_id existing_search_id desired_search_file current_search_file search_id id body
  for search_file in "$SCRIPT_DIR"/dashboards/search_*.json; do
    view_file="${search_file/search_/view_}"
    [ -f "$view_file" ] || die "missing $view_file for $search_file"
    title="$(python3 -c "import json;print(json.load(open('$view_file'))['entity']['title'])")"
-    if echo "$existing_titles" | grep -qx "$title"; then
-      skip "dashboard '$title' already exists"
-      continue
+
+    desired_search_file="$(mktemp)"
+    sed -e "s/__NETWORK_STREAM_ID__/$NETWORK_STREAM_ID/g" -e "s/__SERVERS_STREAM_ID__/$SERVERS_STREAM_ID/g" "$search_file" > "$desired_search_file"
+
+    existing_id="$(python3 -c "
+import json
+d = json.load(open('$existing_views_file'))
+print(next((v['id'] for v in d['views'] if v['title'] == '$title'), ''))
+")"
+
+    if [ -n "$existing_id" ]; then
+      existing_search_id="$(python3 -c "
+import json
+d = json.load(open('$existing_views_file'))
+print(next(v['search_id'] for v in d['views'] if v['id'] == '$existing_id'))
+")"
+      current_search_file="$(mktemp)"
+      gcurl GET "/views/search/$existing_search_id" > "$current_search_file"
+
+      if python3 -c "
+import json, sys
+
+def sort_search_types(q):
+    # search_types order isn't semantically meaningful (widgets are matched
+    # to view widgets by 'id', not position) but Graylog doesn't return them
+    # in the same order they were created in - confirmed live 2026-07-23:
+    # this caused a false 'changed' positive for every single dashboard on
+    # every single run, identical to the list-order bug already worked
+    # around in step_alerts().
+    q = dict(q)
+    if 'search_types' in q:
+        q['search_types'] = sorted(q['search_types'], key=lambda st: st.get('id', ''))
+    return q
+
+def filter_to(desired_obj, current_obj):
+    # Keep only the keys/structure our search_*.json files actually author -
+    # confirmed live 2026-07-23: a 'messages'-type search_type (the message
+    # list widget on the Overview dashboard) comes back from GET with a pile
+    # of extra default keys (decorators, fields, filter, name, timerange,
+    # stream_categories, ...) that never appear in our file, so a whole-
+    # object compare always said 'changed' for that one dashboard even with
+    # sort_search_types() already applied - same reasoning as step_alerts().
+    if isinstance(desired_obj, dict):
+        cur = current_obj if isinstance(current_obj, dict) else {}
+        return {k: filter_to(v, cur.get(k)) for k, v in desired_obj.items()}
+    if isinstance(desired_obj, list):
+        cur = current_obj if isinstance(current_obj, list) else []
+        return [filter_to(dv, cur[i] if i < len(cur) else None) for i, dv in enumerate(desired_obj)]
+    return current_obj
+
+current = json.load(open('$current_search_file'))
+desired = json.load(open('$desired_search_file'))
+current_q = [sort_search_types(q) for q in current.get('queries', [])]
+desired_q = [sort_search_types(q) for q in desired.get('queries', [])]
+sys.exit(0 if filter_to(desired_q, current_q) == desired_q else 1)
+"; then
+        skip "dashboard '$title' already up to date ($existing_id)"
+        rm -f "$desired_search_file" "$current_search_file"
+        continue
+      fi
+      rm -f "$current_search_file"
+
+      gcurl DELETE "/views/$existing_id" >/dev/null
+      gcurl DELETE "/views/search/$existing_search_id" >/dev/null
+      ok "removed stale dashboard '$title' ($existing_id) - content changed, recreating"
    fi

-    body="$(sed -e "s/__NETWORK_STREAM_ID__/$NETWORK_STREAM_ID/g" -e "s/__SERVERS_STREAM_ID__/$SERVERS_STREAM_ID/g" "$search_file")"
-    search_id="$(gcurl POST /views/search "$body" | python3 -c "import json,sys;print(json.load(sys.stdin)['id'])")"
+    search_id="$(gcurl POST /views/search "$(cat "$desired_search_file")" | python3 -c "import json,sys;print(json.load(sys.stdin)['id'])")"
    [ -n "$search_id" ] || die "search creation failed for dashboard '$title'"
+    rm -f "$desired_search_file"

    body="$(sed -e "s/__NETWORK_STREAM_ID__/$NETWORK_STREAM_ID/g" -e "s/__SERVERS_STREAM_ID__/$SERVERS_STREAM_ID/g" -e "s/__SEARCH_ID__/$search_id/" "$view_file")"
    id="$(gcurl POST /views "$body" | python3 -c "import json,sys;print(json.load(sys.stdin).get('id',''))")"
    [ -n "$id" ] || die "dashboard '$title' failed to create"
    ok "created dashboard '$title' ($id)"
  done
+  rm -f "$existing_views_file"
 }

 step_alerts() {