Major security upgrades to 25 release

CHANGELOG.md

@@ -1,3 +1,27 @@
+## Version 25.03.xx
+Security Fixes:
+- [docker] Rebuilt the api and frontend images as multi-stage builds on Debian 13 with Node.js 24, so compilers, development headers and build tooling are no longer present in the shipped images.
+- [core] Updated vulnerable transitive dependencies through npm overrides: tar, form-data, brace-expansion, minimatch, immutable, ip-address, basic-ftp, websocket-driver, js-yaml, body-parser, qs, postcss, protobufjs and ws
+- [hooks] Replaced the unmaintained v8-sandbox custom-code sandbox with isolated-vm, which exposes no network, filesystem or process surface to custom code. Custom code that relies on `setTimeout`, `setInterval` or asynchronous completion now fails with a logged error instead of running, and the `bufferToBase64` and `base64ToBuffer` helpers are no longer available
+- [ab-testing] Replaced pystan 2.19 and its pickled models with cmdstanpy and compiled Stan executables, removing the end-of-life Python 3.8 runtime
+
+Fixes:
+- [hooks] Custom code calling `setResult(undefined)` no longer fails the effect
+- [docker] A plugin whose dependency install or asset build fails now fails the image build instead of being shipped broken without warning
+- [ab-testing] The model runner locates its CmdStan installation automatically when `CMDSTAN` is not set in the service environment
+
+Enterprise Fixes:
+- [ab-testing] `/o/ab-testing/check-models` reports whether the models are actually compiled.
+
+Enterprise Features:
+- [ab-testing] Added `/o/ab-testing/test-models`, a global-admin diagnostic that runs a model against supplied variant data and returns the raw result
+
+Dependencies:
+- [core] Node.js 20 to 22 for package installs and upgrades. The hooks plugin requires Node.js 22 or later, as isolated-vm does not support Node.js 20. Fresh installs on Ubuntu and RHEL now install Node.js 22, and `bin/upgrade/25.03/upgrade.nodejs.22.sh` upgrades an existing 25.03 install. RHEL 8 additionally installs gcc-toolset-11 and Python 3.8, since isolated-vm publishes no prebuilt binary for glibc 2.28 and has to be compiled there
+- [docker] Node.js 20 to 24, Debian 12 to 13, npm 10 to 11.19.0. Custom plugins containing native addons must be rebuilt, and images derived from these must reinstall any build tooling they need
+- [ab-testing] Python 3.8 to 3.12, pystan 2.19.1.1 to cmdstanpy 1.2.5 with CmdStan 2.36.0
+- [hooks] v8-sandbox 3.2.12 to isolated-vm 6.2.0
+
 ## Version 25.03.51
 Fixes:
 - [hooks] Internal event hooks are validated on save: an unknown event type is rejected, and an event that names a cohort, hook or alert must name one belonging to the hook's own apps

Dockerfile-api

@@ -1,12 +1,108 @@
-# node:iron-bookworm-slim -- Node.js 20.20.2 (Iron LTS) on Debian bookworm-slim.
-# Multi-arch manifest list digest, resolved 2026-08-03.
-FROM node:iron-bookworm-slim@sha256:2cf067cfed83d5ea958367df9f966191a942351a2df77d6f0193e162b5febfc0
+# node:24-trixie-slim -- Node.js 24 (Krypton LTS) on Debian trixie-slim.
+# Multi-arch manifest list digest, resolved 2026-08-07.
+FROM node:24-trixie-slim@sha256:0711b541c1c33a8a530ac4f0d391baa9a15b3d804695b1b24a47daa5fb60e74d AS builder-base
+
+ENV NODE_OPTIONS="--max-old-space-size=2048"
+
+# Build dependencies live only in build stages and are never shipped
+RUN apt-get update && \
+	apt-get install -y curl wget git make gcc g++ unzip xz-utils build-essential sudo \
+		python3 python3-setuptools zlib1g-dev libssl-dev libncurses-dev libgdbm-dev libnss3-dev libsqlite3-dev \
+		libreadline-dev libffi-dev libbz2-dev sqlite3 ca-certificates
+
+# Python 3.12 from source for the ab-testing cmdstanpy models
+RUN cd /usr/src && \
+	wget -q https://www.python.org/ftp/python/3.12.12/Python-3.12.12.tar.xz && \
+	tar -xf Python-3.12.12.tar.xz && \
+	cd Python-3.12.12 && \
+	./configure --enable-optimizations --enable-shared && \
+	make -j"$(nproc)" && \
+	make altinstall && \
+	ldconfig && \
+	python3.12 -c "import ssl" && \
+	python3.12 -m pip install --no-cache-dir --upgrade pip setuptools wheel && \
+	rm -rf /usr/src/Python*
+
+
+FROM builder-base AS ab-models
+
+ARG COUNTLY_PLUGINS=mobile,web,desktop,plugins,density,locale,browser,sources,views,logger,systemlogs,populator,reports,crashes,push,star-rating,slipping-away-users,compare,server-stats,dbviewer,times-of-day,compliance-hub,alerts,onboarding,consolidate,remote-config,hooks,dashboards,sdk,data-manager,guides
+# CmdStan pinned to the release cmdstanpy 1.2.5 was fixed against
+ARG CMDSTAN_VERSION=2.36.0
+ENV CMDSTAN=/opt/cmdstan/cmdstan-${CMDSTAN_VERSION}
+WORKDIR /opt/countly
+
+# ab-testing is EE-only. Bracketing every separator makes the path one glob, so
+# these COPYs resolve to nothing instead of failing in a CE-only build context.
+COPY plugins[/]ab-testing[/]api[/]bayesian[/]requirements_docker.txt ./plugins/ab-testing/api/bayesian/
+
+# Install the python stack only when the ab-testing plugin is enabled
+RUN set -eux; \
+	echo "$COUNTLY_PLUGINS" | tr ',' '\n' | grep -qx ab-testing \
+	  || { mkdir -p plugins/ab-testing/api/bayesian; touch plugins/ab-testing/api/bayesian/.ab_disabled; exit 0; }; \
+	python3.12 -m pip install --no-cache-dir -r plugins/ab-testing/api/bayesian/requirements_docker.txt
+
+COPY plugins[/]ab-testing[/]api[/]bayesian[/]*.py ./plugins/ab-testing/api/bayesian/
+COPY plugins[/]ab-testing[/]api[/]bayesian[/]models[/]*.py ./plugins/ab-testing/api/bayesian/models/
+COPY plugins[/]ab-testing[/]api[/]bayesian[/]models[/]stan_models[/]*.stan ./plugins/ab-testing/api/bayesian/models/stan_models/
+
+# Install CmdStan and compile the seven variant models
+RUN set -eux; \
+	if [ -f plugins/ab-testing/api/bayesian/.ab_disabled ]; then exit 0; fi; \
+	CMDSTAN= CMDSTAN_VERSION="${CMDSTAN_VERSION}" CMDSTAN_DIR=/opt/cmdstan CMDSTAN_BUILD_CORES="$(nproc)" \
+	python3.12 plugins/ab-testing/api/bayesian/models/cmdstanpy_model.py; \
+	for i in 2 3 4 5 6 7 8; do test -x "plugins/ab-testing/api/bayesian/models/stan_models/model_${i}_variants"; done
+
+# Prune CmdStan to the runtime minimum, preserving RPATH-relative layout
+RUN set -eux; \
+	D="/opt/cmdstan-min/cmdstan-${CMDSTAN_VERSION}"; mkdir -p "$D/bin"; \
+	if [ ! -d "$CMDSTAN" ]; then exit 0; fi; \
+	cp "$CMDSTAN/bin/stansummary" "$CMDSTAN/bin/diagnose" "$CMDSTAN/bin/stanc" "$D/bin/"; \
+	cp "$CMDSTAN/makefile" "$D/makefile"; \
+	find "$CMDSTAN/stan/lib/stan_math/lib" -name 'libtbb*.so*' -print | while read -r lib; do \
+		rel="${lib#"$CMDSTAN"/}"; mkdir -p "$D/$(dirname "$rel")"; cp -a "$lib" "$D/$rel"; done; \
+	find "$D" -name 'libtbb*.so*' | grep -q .
+
+
+FROM builder-base AS builder
 
 ARG COUNTLY_PLUGINS=mobile,web,desktop,plugins,density,locale,browser,sources,views,logger,systemlogs,populator,reports,crashes,push,star-rating,slipping-away-users,compare,server-stats,dbviewer,times-of-day,compliance-hub,alerts,onboarding,consolidate,remote-config,hooks,dashboards,sdk,data-manager,guides
 # Countly Enterprise:
 #ARG COUNTLY_PLUGINS=mobile,web,desktop,plugins,density,locale,browser,sources,views,license,drill,funnels,retention_segments,flows,cohorts,surveys,remote-config,ab-testing,formulas,activity-map,concurrent_users,revenue,logger,systemlogs,populator,reports,crashes,push,geo,block,users,star-rating,slipping-away-users,compare,server-stats,dbviewer,crash_symbolication,crashes-jira,groups,white-labeling,alerts,times-of-day,compliance-hub,onboarding,active_users,performance-monitoring,config-transfer,consolidate,data-manager,hooks,dashboards,heatmaps,sdk,guides,journey_engine,content
 
 ARG GEOIP=empty
+
+ENV COUNTLY_CONTAINER="api" \
+	COUNTLY_DEFAULT_PLUGINS="${COUNTLY_PLUGINS}" \
+	npm_config_allow_git=root \
+	npm_config_dangerously_allow_all_scripts=true \
+	PUPPETEER_CACHE_DIR=/opt/countly/.cache/puppeteer
+
+WORKDIR /opt/countly
+COPY . .
+COPY --from=ab-models /opt/countly/plugins/ab-testing/api/bayesian ./plugins/ab-testing/api/bayesian
+
+RUN ./bin/docker/modify.sh && \
+	cp -n ./api/config.sample.js ./api/config.js && \
+	cp -n ./frontend/express/config.sample.js ./frontend/express/config.js && \
+	HOME=/tmp npm install -g npm@11.19.0 && \
+	HOME=/tmp npm install && \
+	HOME=/tmp npm_config_build_from_source=true npm install argon2 && \
+	./bin/docker/preinstall.sh && \
+	bash /opt/countly/bin/scripts/detect.init.sh && \
+	npm remove -y --no-save mocha nyc should supertest && \
+	# CE builds carry only a stub ab-testing dir from the ab-models COPY; drop it
+	{ [ -f plugins/ab-testing/package.json ] || rm -rf plugins/ab-testing; } && \
+	rm -rf test /tmp/* /var/tmp/* /root/.npm && \
+	chown -R 1001:0 /opt/countly && \
+	chmod -R g=u /opt/countly
+
+
+FROM node:24-trixie-slim@sha256:0711b541c1c33a8a530ac4f0d391baa9a15b3d804695b1b24a47daa5fb60e74d AS runtime
+
+ARG COUNTLY_PLUGINS=mobile,web,desktop,plugins,density,locale,browser,sources,views,logger,systemlogs,populator,reports,crashes,push,star-rating,slipping-away-users,compare,server-stats,dbviewer,times-of-day,compliance-hub,alerts,onboarding,consolidate,remote-config,hooks,dashboards,sdk,data-manager,guides
+ARG CMDSTAN_VERSION=2.36.0
+
 EXPOSE 3001
 HEALTHCHECK --start-period=400s CMD bash -c '\
   if [ "$COUNTLY_CONFIG_API_API_SSL_ENABLED" = "true" ] || [ "$COUNTLY_CONFIG_API_API_SSL_ENABLED" = "\"true\"" ]; then \
@@ -15,72 +111,42 @@ HEALTHCHECK --start-period=400s CMD bash -c '\
     curl --fail http://localhost:3001/o/ping || exit 1; \
   fi'
 
-USER root
-
-# Core dependencies
-## Tini
 ENV COUNTLY_CONTAINER="api" \
 	COUNTLY_DEFAULT_PLUGINS="${COUNTLY_PLUGINS}" \
 	COUNTLY_CONFIG_API_API_WORKERS="1" \
 	COUNTLY_CONFIG_API_API_HOST="0.0.0.0" \
-	NODE_OPTIONS="--max-old-space-size=2048" \
-	TINI_VERSION="0.18.0"\
+	NODE_OPTIONS="--max-old-space-size=2048 --no-node-snapshot" \
+	HOME=/tmp \
+	CMDSTAN=/opt/cmdstan/cmdstan-${CMDSTAN_VERSION} \
 	PUPPETEER_CACHE_DIR=/opt/countly/.cache/puppeteer
-
-WORKDIR /opt/countly
-COPY . .
 
-# install required dependencies which slim image doesn't have
-RUN apt-get update && \
-	apt-get install -y iputils-ping procps net-tools telnet apt-transport-https curl wget git make gcc g++ unzip xz-utils
+# Runtime-only packages: init, ops tools, Chrome and Python shared libs
+RUN apt-get update && apt-get upgrade -y && \
+	apt-get install -y --no-install-recommends \
+		tini curl procps iputils-ping net-tools ca-certificates \
+		sudo telnet xz-utils apt-transport-https lsb-release \
+		libffi8 libbz2-1.0 liblzma5 libsqlite3-0 libreadline8 libncursesw6 \
+		libgbm1 libasound2 libatk1.0-0 libatk-bridge2.0-0 libcairo2 libcups2 \
+		libdbus-1-3 libexpat1 libfontconfig1 libgdk-pixbuf-2.0-0 libglib2.0-0 \
+		libgtk-3-0 libnspr4 libnss3 libpango-1.0-0 libpangocairo-1.0-0 \
+		libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 \
+		libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 \
+		fonts-liberation xdg-utils && \
+	apt-get clean && rm -rf /var/lib/apt/lists/* && \
+	echo "Etc/UTC" > /etc/timezone
 
-RUN apt-get update && \
-	apt-get upgrade -y && \
-	cd /usr/src && \
-	wget https://www.python.org/ftp/python/3.8.20/Python-3.8.20.tar.xz && \
-	tar -xf Python-3.8.20.tar.xz && \
-	apt-get install -y build-essential sudo zlib1g-dev libssl3 libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libsqlite3-dev libreadline-dev libffi-dev curl libbz2-dev && \
-	cd Python-3.8.20 && \
-	./configure --enable-optimizations --enable-shared && \
-	make && \
-	make altinstall && \
-	ldconfig /usr/src/Python-3.8.20 && \
-	ln -sf /usr/local/bin/python3.8 /usr/bin/python3
-
-RUN curl -s -L -o /tmp/tini.deb "https://github.com/krallin/tini/releases/download/v${TINI_VERSION}/tini_${TINI_VERSION}.deb" && \
-	dpkg -i /tmp/tini.deb && \
-	\
-	# modify standard distribution
-	apt-get update && apt-get install -y sqlite3 && \
-	./bin/docker/modify.sh && \
-	\
-	# preinstall
-	cp -n ./api/config.sample.js ./api/config.js && \
-	cp -n ./frontend/express/config.sample.js ./frontend/express/config.js && \
-	HOME=/tmp npm install -g npm@10 && \
-	HOME=/tmp npm install --unsafe-perm=true --allow-root && \
-	HOME=/tmp npm install argon2 --build-from-source --unsafe-perm=true --allow-root && \
-	./bin/docker/preinstall.sh && \
-	bash /opt/countly/bin/scripts/detect.init.sh && \
-        python3.8 -m pip install --upgrade pip setuptools && \
-	\
-	# cleanup & chown
-	npm remove -y --no-save mocha nyc should supertest && \
-        apt-get purge -y python3 python3-pip python3-distutils python3-setuptools && \
-	apt-get remove -y git gcc g++ make automake autoconf libtool pkg-config unzip sqlite3 wget && \
-	apt-get install -y libgbm-dev libgbm1 gconf-service libasound2 libatk1.0-0 libatk-bridge2.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils && \
-	apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \
-        apt-get autoremove -y && \
-	rm -rf test /tmp/* /tmp/.??* /var/tmp/* /var/tmp/.??* /var/log/* /root/.npm /usr/local/lib/python3.8/ensurepip/_bundled/*.whl /usr/src/Python* /var/lib/dpkg/info/python3* && \
-	\
-	# temporary to remove npm bug message
-	mkdir /.npm && chown -R 1001:0 /.npm && \
-	\
-	# More tests needed
-	# (find / -perm +6000 -type f -exec chmod a-s {} \; || true) && \
-	chown -R 1001:0 /opt/countly && \
-	chmod -R g=u /opt/countly
+WORKDIR /opt/countly
+# ab-models /usr/local carries python 3.12 + cmdstanpy stack; builder adds npm@11
+COPY --from=ab-models /usr/local /usr/local
+COPY --from=builder /usr/local/lib/node_modules /usr/local/lib/node_modules
+COPY --from=ab-models /opt/cmdstan-min /opt/cmdstan
+COPY --from=builder /opt/countly /opt/countly
 
+RUN ldconfig && \
+	ln -sf /usr/local/bin/python3.12 /usr/bin/python3 && \
+	ln -sf /opt/countly/bin/commands/countly.sh /usr/bin/countly && \
+	chown 1001:0 /opt/countly && chmod g=u /opt/countly && \
+	mkdir -p /.npm && chown -R 1001:0 /.npm && chmod -R g=u /.npm
 
 USER 1001:0
 

Dockerfile-centos-api

@@ -25,6 +25,7 @@ ENV COUNTLY_CONTAINER="api" \
 	COUNTLY_CONFIG_API_API_HOST="0.0.0.0" \
 	NODE_OPTIONS="--max-old-space-size=2048" \
 	TINI_VERSION="0.18.0" \
+	CMDSTAN="/opt/cmdstan/cmdstan-2.36.0" \
 	PATH="/opt/rh/rh-nodejs10/root/usr/bin:${PATH}"
 
 WORKDIR /opt/countly
@@ -41,7 +42,7 @@ RUN yum update -y
 RUN curl -s -L -o /tmp/tini.rpm "https://github.com/krallin/tini/releases/download/v${TINI_VERSION}/tini_${TINI_VERSION}.rpm" && \
 	rpm -i /tmp/tini.rpm && \
 	\
-	curl -sL https://rpm.nodesource.com/setup_20.x | bash - && \
+	curl -sL https://rpm.nodesource.com/setup_22.x | bash - && \
 	yum install -y nodejs python3.8 python2 python38-libs python38-devel python38-pip nss libdrm libgbm cyrus-sasl* && \
 	ln -s /usr/bin/node /usr/bin/nodejs && \
 	unlink /usr/bin/python3 && \
@@ -50,12 +51,16 @@ RUN curl -s -L -o /tmp/tini.rpm "https://github.com/krallin/tini/releases/downlo
 	\
 	yum install -y python3-policycoreutils && \
 	yum group install -y "Development Tools" && \
+	yum install -y gcc-toolset-11 && \
 	yum install -y epel-release && \
 	yum install -y pango.x86_64 libXcomposite.x86_64 libXcursor.x86_64 libXdamage.x86_64 libXext.x86_64 libXi.x86_64 libXtst.x86_64 cups-libs.x86_64 libXScrnSaver.x86_64 libXrandr.x86_64 GConf2.x86_64 alsa-lib.x86_64 atk.x86_64 gtk3.x86_64 xorg-x11-fonts-100dpi xorg-x11-fonts-75dpi xorg-x11-utils xorg-x11-fonts-cyrillic xorg-x11-fonts-Type1 xorg-x11-fonts-misc && \
     curl -L -O -J "https://box.tools.count.ly/public.php/dav/files/Wj8opzNdyE5DyDX/?accept=zip" && \
     yum install -y raven-release.el8.noarch.rpm && \
 	yum install -y wget openssl-devel make git libsqlite* sqlite unzip bzip2 && \
 	# modify standard distribution
+	# isolated-vm needs -std=c++20 and node-gyp needs python>=3.8; stock CentOS 8 has neither
+	. /opt/rh/gcc-toolset-11/enable && \
+	export npm_config_python=/usr/bin/python3.8 && \
 	./bin/docker/modify.sh && \
 	\
 	# preinstall

Dockerfile-centos-frontend

@@ -23,6 +23,7 @@ ENV COUNTLY_CONTAINER="frontend" \
 	COUNTLY_CONFIG_FRONTEND_WEB_HOST="0.0.0.0" \
 	NODE_OPTIONS="--max-old-space-size=2048" \
 	TINI_VERSION="0.18.0" \
+	CMDSTAN="/opt/cmdstan/cmdstan-2.36.0" \
 	PATH="/opt/rh/rh-nodejs10/root/usr/bin:${PATH}"
 
 WORKDIR /opt/countly
@@ -39,7 +40,7 @@ RUN yum update -y
 RUN curl -s -L -o /tmp/tini.rpm "https://github.com/krallin/tini/releases/download/v${TINI_VERSION}/tini_${TINI_VERSION}.rpm" && \
         rpm -i /tmp/tini.rpm && \
         \
-        curl -sL https://rpm.nodesource.com/setup_20.x | bash - && \
+        curl -sL https://rpm.nodesource.com/setup_22.x | bash - && \
         yum install -y nodejs python3.8 python2 python38-libs python38-devel python38-pip nss libdrm libgbm cyrus-sasl* && \
         ln -s /usr/bin/node /usr/bin/nodejs && \
         unlink /usr/bin/python3 && \
@@ -48,13 +49,19 @@ RUN curl -s -L -o /tmp/tini.rpm "https://github.com/krallin/tini/releases/downlo
         \
         yum install -y python3-policycoreutils && \
         yum group install -y "Development Tools" && \
+	yum install -y gcc-toolset-11 && \
         yum install -y epel-release && \
         yum install -y pango.x86_64 libXcomposite.x86_64 libXcursor.x86_64 libXdamage.x86_64 libXext.x86_64 libXi.x86_64 libXtst.x86_64 cups-libs.x86_64 libXScrnSaver.x86_64 libXrandr.x86_64 GConf2.x86_64 alsa-lib.x86_64 atk.x86_64 gtk3.x86_64 xorg-x11-fonts-100dpi xorg-x11-fonts-75dpi xorg-x11-utils xorg-x11-fonts-cyrillic xorg-x11-fonts-Type1 xorg-x11-fonts-misc && \
         curl -L -O -J "https://box.tools.count.ly/public.php/dav/files/Wj8opzNdyE5DyDX/?accept=zip" && \
         yum install -y raven-release.el8.noarch.rpm && \
         yum install -y ipa-gothic-fonts && \
         yum install -y wget openssl-devel make git sqlite libsqlite* unzip bzip2 && \
         # modify standard distribution
+        # isolated-vm needs -std=c++20 and node-gyp needs python>=3.8; stock CentOS 8 has neither
+        . /opt/rh/gcc-toolset-11/enable && \
+        export npm_config_python=/usr/bin/python3.8 && \
+        # frontend never runs models; the marker makes ab-testing's setup.sh a no-op
+        { [ ! -d plugins/ab-testing/api/bayesian ] || touch plugins/ab-testing/api/bayesian/.ab_disabled; } && \
         ./bin/docker/modify.sh && \
         \
 	# preinstall

Dockerfile-core

@@ -43,7 +43,7 @@ RUN sed -i 's|http://|https://|g' /etc/apt/sources.list && \
         gcc g++ make binutils autoconf automake autotools-dev libtool pkg-config zlib1g-dev libcunit1-dev libssl-dev libxml2-dev libev-dev \
         libevent-dev libjansson-dev libjemalloc-dev cython python3-dev python-setuptools && \
     # node
-    wget -qO- https://deb.nodesource.com/setup_20.x | bash - && \
+    wget -qO- https://deb.nodesource.com/setup_22.x | bash - && \
     # data_migration (mongo clients)
     wget -qO - https://www.mongodb.org/static/pgp/server-6.0.asc | sudo apt-key add - && \
     echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/6.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-6.0.list && \