Backups in v2.6.42+ exhausting memory for database with lots of tables

[agelwarg]

We have > 10k tables, which we know is a design that is abnormal. However, this has worked for our use case and backups up until version v2.6.42 have been working fine. We run 2 nodes with all tables replicated, logically treating one node (with more cpu/memory) as a read/write node, and the other purely as a reader. We perform alternating backups on both nodes and beginning with clickhouse-backup version v2.6.42, the reader node backup fails 100% of the time due to memory. We are wondering if this could be related to this change: #1194

[CaptTofu]

Could you please provide any logs captured during the backup? Thank you!

[agelwarg]

Here's an example log:

2026-04-14 14:11:39.860 ERR pkg/backup/create.go:153 > backup failed error: CheckSystemPartsColumnsForTables failed: code: 241, message: (total) memory limit exceeded: would use 83.67 GiB (attempt to allocate chunk of 10.21 MiB bytes), current RSS: 87.91 GiB, maximum: 87.92 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: while executing 'FUNCTION and(equals(__table1.database, 'data__XXXXXXXXX'_String) : 240, equals(__table1.table, '__ts_fields'_String) : 233) -> and(equals(__table1.database, 'data__XXXXXXXXX'_String), equals(__table1.table, '__ts_fields'_String... UInt8 : 4759'
2026-04-14 14:11:39.862 INF pkg/clickhouse/clickhouse.go:120 > clickhouse connection success: tcp://prod-clickhouse-default-1:9000
2026-04-14 14:11:39.960 INF pkg/backup/delete.go:171 > remove '/var/lib/clickhouse/backup/backup.20260414.141004'
2026-04-14 14:11:39.960 INF pkg/backup/delete.go:180 > done, operation=delete, location=local, backup=backup.20260414.141004, duration=100ms
2026-04-14 14:11:39.960 INF pkg/clickhouse/clickhouse.go:322 > clickhouse connection closed
2026-04-14 14:11:39.962 INF pkg/clickhouse/clickhouse.go:120 > clickhouse connection success: tcp://prod-clickhouse-default-1:9000
2026-04-14 14:11:39.962 INF pkg/clickhouse/clickhouse.go:1185 > SELECT countIf(name='type') AS is_disk_type_present, countIf(name='object_storage_type') AS is_object_storage_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')
2026-04-14 14:11:39.966 INF pkg/clickhouse/clickhouse.go:1185 > SELECT d.path AS path, any(d.name) AS name, any(lower(if(d.type='ObjectStorage',d.object_storage_type,d.type))) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  LEFT JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path
2026-04-14 14:12:27.987 INF pkg/backup/delete.go:45 > /var/lib/clickhouse/shadow
2026-04-14 14:12:27.987 INF pkg/clickhouse/clickhouse.go:322 > clickhouse connection closed
2026-04-14 14:12:27.987 INF pkg/clickhouse/clickhouse.go:322 > clickhouse connection closed
2026-04-14 14:12:27.988 FTL cmd/clickhouse-backup/main.go:820 > , error=CheckSystemPartsColumnsForTables failed: code: 241, message: (total) memory limit exceeded: would use 83.67 GiB (attempt to allocate chunk of 10.21 MiB bytes), current RSS: 87.91 GiB, maximum: 87.92 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: while executing 'FUNCTION and(equals(__table1.database, 'data__XXXXXXXXX'_String) : 240, equals(__table1.table, '__ts_fields'_String) : 233) -> and(equals(__table1.database, 'data__XXXXXXXXX'_String), equals(__table1.table, '__ts_fields'_String... UInt8 : 4759'
stack:
github.com/Altinity/clickhouse-backup/v2/pkg/backup.(*Backuper).createBackupLocal()
  github.com/Altinity/clickhouse-backup/v2/pkg/backup/create.go:316
github.com/Altinity/clickhouse-backup/v2/pkg/backup.(*Backuper).CreateBackup()
  github.com/Altinity/clickhouse-backup/v2/pkg/backup/create.go:150
main.main.func4()
  github.com/Altinity/clickhouse-backup/v2/cmd/clickhouse-backup/main.go:106
github.com/urfave/cli.HandleAction()
  github.com/urfave/cli@v1.22.17/app.go:524
github.com/urfave/cli.Command.Run()
  github.com/urfave/cli@v1.22.17/command.go:175
github.com/urfave/cli.(*App).Run()
  github.com/urfave/cli@v1.22.17/app.go:277
main.main()
  github.com/Altinity/clickhouse-backup/v2/cmd/clickhouse-backup/main.go:819
runtime.main()
  runtime/proc.go:290
runtime.goexit()
  runtime/asm_amd64.s:1771

[agelwarg]

To clarify, our backups work just fine with v2.6.41, but fail beginning with v2.6.42.

[CaptTofu]

yeah, that sounds like a regression. It would be interesting to know what the memory usage is between the two versions.

[caymard]

@agelwarg not sure if related but I had this error on a fresh new clickhouse cluster, with altinity/clickhouse-backup:2.6.42:

2026-04-22 07:42:17.524 DBG pkg/server/server.go:81 > Wait for ClickHouse
2026-04-22 07:42:17.580 ERR pkg/server/server.go:123 > ResumeOperationsAfterRestart return error: open /var/lib/clickhouse/backup: no such file or directory

With you message, I tried altinity/clickhouse-backup:2.6.41 and no more bugs. Thanks :)