From 00d492a5046cba6540a527a1dfbfe13cc5c579a7 Mon Sep 17 00:00:00 2001 From: Alex Godbehere Date: Fri, 17 Jul 2026 14:04:29 +0100 Subject: [PATCH 1/2] Gate the openid restart on openid.enabled The restart loop hardcoded 'for D in mqtt openid', while every other openid reference in this file is wrapped in an openid.enabled guard. With openid.enabled=false - which the v6 release notes document as supported - the deployment does not exist, so the rollout restart exits 1. The Job is restartPolicy OnFailure with backoffLimit 9999, and each retry starts by restarting mqtt again, so the broker is bounced every few minutes for weeks, every edge agent reconnect-loops, and the Job never completes. helm upgrade reports success throughout, because the Job is a plain resource rather than a hook. --- deploy/templates/service-setup.yaml | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/deploy/templates/service-setup.yaml b/deploy/templates/service-setup.yaml index 42259c32c..23978309b 100644 --- a/deploy/templates/service-setup.yaml +++ b/deploy/templates/service-setup.yaml @@ -135,8 +135,14 @@ spec: # SPI's cached JAAS Subject. Without a restart # the SPI's outbound calls to F+ Auth time # out until the pod is bounced manually. + # openid is gated: with openid.enabled=false the + # deployment does not exist, and an ungated restart would + # exit 1 on every run. The Job is restartPolicy OnFailure + # with backoffLimit 9999, so each retry would bounce mqtt + # again - restarting the broker every few minutes for + # weeks while helm upgrade reports success. NAMESPACE={{.Release.Namespace}} - for D in mqtt openid; do + for D in mqtt{{ if .Values.openid.enabled }} openid{{ end }}; do echo "Restarting deployment: $D" kubectl rollout restart "deployment/$D" -n "$NAMESPACE" || exit 1 kubectl rollout status "deployment/$D" -n "$NAMESPACE" --timeout=120s || exit 1 From ad09bad8ba50a4671b68ecd6988b50f7d75c53d7 Mon Sep 17 00:00:00 2001 From: Alex Godbehere Date: Fri, 17 Jul 2026 14:38:23 +0100 Subject: [PATCH 2/2] Ship Grafana 12.4.5 rather than 13.1.0 Grafana 13 deleted the sqlstore migration that adds playlist.created_at/updated_at, which existed from 10.2.0 to 12.4.x, while still running a unified-storage playlist migration whose SQL requires them. Any database created before 10.2.0 - including every ACS v5.1.0 site, which shipped Grafana 10.0.1 - cannot start under 13. Skipping that migration, as the previous fix did, is not an option: Grafana 13 removed the legacy playlist service entirely and serves playlists from unified storage, so skipping it makes every existing playlist disappear from the UI. ACS installations do use playlists, to rotate dashboards on wall displays. 12.4.x still ships playlist_mig.go, so Grafana's own framework adds the columns from whatever schema a site starts on, then completes the playlist migration and records it in unifiedstorage_migration_log. A later move to 13 is safe, because 13 skips the migration on that row. Everything the 13 bump was for survives: 10.x is still two majors behind on security fixes, and auth.jwt with its dev-mode JWKS exemption is present and identical at 12.4.5. Also pin autoMigrationThreshold for folders and dashboards. Grafana 12.4.x auto-migrates both into unified storage when a site has fewer dashboards than the default threshold of 10, one-way. ACS provisions five, so without this a fresh install takes a migration an established site does not - splitting the estate on a row count and leaving CI exercising a different path from production. --- deploy/values.yaml | 45 ++++++++++++++++++--------------- docs/reference/release-notes.md | 22 +++++++++++++--- 2 files changed, 43 insertions(+), 24 deletions(-) diff --git a/deploy/values.yaml b/deploy/values.yaml index 342edd4f9..a35280ec4 100644 --- a/deploy/values.yaml +++ b/deploy/values.yaml @@ -454,7 +454,17 @@ grafana: # v13 migrates dashboards to a new schema ON FIRST USE and the migration # is one-way — see "Grafana upgraded from v10 to v13" in the release # notes before downgrading this on an existing installation. - tag: 13.1.0 + # Not 13.x. Grafana 13 deleted the sqlstore migration that adds + # playlist.created_at/updated_at (present 10.2.0 - 12.4.x) while + # still running a unified-storage playlist migration whose SQL + # requires those columns, so any database created before 10.2.0 - + # including every ACS v5.1.0 site, which shipped Grafana 10.0.1 - + # cannot start under 13. 12.4.x still ships the migration, adds the + # columns through Grafana's own framework, and then completes the + # unified-storage migration itself, recording it in + # unifiedstorage_migration_log. A later move to 13 is then safe, + # because 13 skips the migration on that log row. + tag: 12.4.5 enabled: true persistence: enabled: true @@ -533,25 +543,20 @@ grafana: # revoking Grafana.Perm.* in F+ downgrades on next login rather # than letting the 60s SPI cache staleness become permanent. role_attribute_strict: true - # Grafana 13 runs a unified-storage data migration for playlists on - # boot, enabled by default (MigratedUnifiedResources in - # pkg/setting/setting_unified_storage.go). Its SQL selects - # playlist.created_at, but the sqlstore migration which adds that - # column existed only in 11.x/12.x: v13 deleted playlist_mig.go - # entirely, so a database created by Grafana 10.x and upgraded - # straight to 13.x never gains the column. Grafana then dies on boot - # with "no such column: p.created_at" - the SELECT fails at prepare, - # so having no playlists does not help. - # - # ACS v5.1.0 shipped Grafana 10.0.1, so every site upgrading with a - # kept Grafana PVC hits this. Fresh installs do not: v13 seeds the - # column from its schema snapshot, which is why CI stayed green. - # - # Skipping the migration leaves the legacy playlist table untouched - # and unified storage with no playlists. ACS does not use playlists. - # Drop this key and add the columns by hand if you ever need them. - unified_storage.playlists.playlist.grafana.app: - enableMigration: false + # Keep folders and dashboards on legacy storage on every site, not + # just the big ones. Grafana 12.4.x auto-migrates them into unified + # storage when a site has fewer dashboards than + # DefaultAutoMigrationThreshold (10, setting_unified_storage.go), and + # that migration is one-way. ACS provisions five dashboards, so + # without this a fresh install silently takes a migration that an + # established site does not, splitting the estate into two storage + # modes on a row count - and leaving CI exercising a different path + # from production. A threshold of 1 is below any real dashboard + # count, so shouldAutoMigrate is false everywhere. + unified_storage.folders.folder.grafana.app: + autoMigrationThreshold: 1 + unified_storage.dashboards.dashboard.grafana.app: + autoMigrationThreshold: 1 extraConfigmapMounts: - name: acs-grafana-config mountPath: /etc/acs-config diff --git a/docs/reference/release-notes.md b/docs/reference/release-notes.md index 960800413..12a065c3a 100644 --- a/docs/reference/release-notes.md +++ b/docs/reference/release-notes.md @@ -94,21 +94,35 @@ grafana: disable_login_form: false ``` -### Grafana upgraded from v10 to v13 +### Grafana upgraded from v10 to v12 -The bundled Grafana image moves from 10.0.1 to 13.1.0. Grafana 10.x +The bundled Grafana image moves from 10.0.1 to 12.4.5. Grafana 10.x left support in 2024, so this catches up two majors of security fixes, and the newer `auth.jwt` support is what lets machine-to-machine consumers sign in with tokens (see below). +This release deliberately stops at 12.4.x rather than 13.x. Grafana 13 +removed the schema migration that adds the `playlist.created_at` and +`updated_at` columns, which were introduced in 10.2.0, while still +running a playlist migration that requires them. A database created by +the Grafana 10.0.1 that ACS v5.1.0 shipped therefore cannot start under +Grafana 13 at all. Grafana 12.4.x still ships that migration, so it +adds the columns and completes the playlist migration itself; a later +move to 13 will then be safe. + Three upstream Grafana changes are worth knowing about when upgrading an existing installation: - Dashboards are migrated to Grafana's new dashboard schema as they are - used, and the migration is one-way: once v13 has written to the - Grafana database you cannot return to the previous image without + used, and the migration is one-way: once the new image has written to + the Grafana database you cannot return to the previous one without losing dashboards. Snapshot the Grafana PersistentVolume before upgrading if the dashboards matter. +- Your playlists are migrated into Grafana's unified storage on first + boot. This is also one-way, and the same snapshot covers it. Check + after upgrading that the playlists you expect are all present before + you rely on the installation; the migration logs a warning rather + than failing if it rejects an individual playlist. - Support for AngularJS plugins has been removed (disabled by default in Grafana 11, gone entirely in 12). Grafana's built-in panels are unaffected and legacy Graph/Table panels are migrated automatically,