Skip to content

ci: cache the sccache directory across C++ test builds #765

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wgtmac merged 2 commits into from
Jun 23, 2026
Merged

ci: cache the sccache directory across C++ test builds #765

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
73b91d4
ci: cache sccache across remaining C++ build legs
Jun 20, 2026
4a0fff5
ci: cache sccache on Meson build legs
Jun 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 44 additions & 3 deletions .github/workflows/aws_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,8 @@ jobs:
AWS_DEFAULT_REGION: us-east-1
AWS_ENDPOINT_URL: http://127.0.0.1:9000
AWS_EC2_METADATA_DISABLED: "TRUE"
SCCACHE_DIR: ${{ github.workspace }}/.sccache
SCCACHE_CACHE_SIZE: "2G"
steps:
- name: Checkout iceberg-cpp
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
Expand Down Expand Up @@ -113,11 +115,29 @@ jobs:
if: ${{ matrix.s3 == 'ON' }}
shell: bash
run: bash ci/scripts/start_minio.sh
- name: Restore sccache cache
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5

@zhjwpku zhjwpku Jun 20, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like the Windows workflow was using mozilla-actions/sccache-action together with SCCACHE_GHA_ENABLED, while this PR changes it to use explicit actions/cache restore/save steps plus mozilla-actions/sccache-action.

I'm not sure which approach is better, but in case you missed it, mozilla-actions/sccache-action README documents the SCCACHE_GHA_ENABLED setup here:
https://github.com/mozilla-actions/sccache-action#cc-code

@abnobdoss abnobdoss Jun 20, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @zhjwpku, thank you for the review!

With SCCACHE_GHA_ENABLED, sccache uploads each compiled object as its own cache entry, which adds up to hundreds per build. On the larger Linux/macOS builds this hits GitHub's upload rate-limit throttle and surfaces as cache write errors. The build still passes when we hit the throttle, but the affected objects silently don't get cached. Pointing sccache at a local directory with a single actions/cache save/restore turns it into one archive per leg, so there's nothing to throttle. In my fork that brought write errors down to zero, and it was a bit faster overall since it uploads once instead of hundreds of small objects that each take a network round trip.

@zhjwpku zhjwpku Jun 20, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, in that case, you may want to make the same change to the Meson Windows build as well. I believe it was missed in this PR, unless I'm overlooking.

@abnobdoss abnobdoss Jun 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call out - I did leave it out of this PR. I was worried there might be too many changes in one PR for review as it already has a sizeable diff on the CI files. If you think it'd be fine here I can add it, or if you prefer I can raise it as a follow-up, wdyt?

@zhjwpku zhjwpku Jun 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can add that here in a separate commit. It will trigger another CI run, and we'll be able to see how effective the cache is in practice.

@abnobdoss abnobdoss Jun 21, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! I've folded the Meson sccache conversion into this PR - this includes all Meson legs plus the SigV4 build.

As a heads-up, we won't see the speedup on this PR itself. PR cache uploads can only be reused by the branch that created them, so I've gated saves to main to keep PR caches from saturating the repo's cache or evicting the main branch's cache entries.

I validated the full stack warm on my fork: every build finishes in about 4 minutes or less except the Windows build at ~5 min, down from 20 to 27 min cold. Example run: https://github.com/abnobdoss/iceberg-cpp/actions/runs/27910572526

with:
path: ${{ github.workspace }}/.sccache
key: sccache-aws-${{ matrix.runs-on }}-bundle${{ matrix.bundle_awssdk }}-s3${{ matrix.s3 }}-sigv4${{ matrix.sigv4 }}-${{ github.run_id }}
restore-keys: |
sccache-aws-${{ matrix.runs-on }}-bundle${{ matrix.bundle_awssdk }}-s3${{ matrix.s3 }}-sigv4${{ matrix.sigv4 }}-
- name: Setup sccache
uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10
- name: Build and test Iceberg
shell: bash
env:
CMAKE_TOOLCHAIN_FILE: ${{ startsWith(matrix.runs-on, 'ubuntu') && matrix.bundle_awssdk == 'OFF' && '/usr/local/share/vcpkg/scripts/buildsystems/vcpkg.cmake' || '' }}
run: ci/scripts/build_iceberg.sh "$(pwd)" OFF OFF ${{ matrix.s3 }} ${{ matrix.sigv4 }} ${{ matrix.bundle_awssdk }}
run: ci/scripts/build_iceberg.sh "$(pwd)" OFF ON ${{ matrix.s3 }} ${{ matrix.sigv4 }} ${{ matrix.bundle_awssdk }}
- name: Show sccache stats
shell: bash
run: sccache --show-stats
- name: Save sccache cache
if: github.ref == 'refs/heads/main'
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-aws-${{ matrix.runs-on }}-bundle${{ matrix.bundle_awssdk }}-s3${{ matrix.s3 }}-sigv4${{ matrix.sigv4 }}-${{ github.run_id }}

# Exercise the Meson build with SigV4 enabled (resolves aws-cpp-sdk-core via
# its CMake config, not pkg-config whose Cflags force -std=c++11).
Expand All @@ -126,6 +146,9 @@ jobs:
name: Meson SigV4 (AMD64 Ubuntu 24.04)
runs-on: ubuntu-24.04
timeout-minutes: 45
env:
SCCACHE_DIR: ${{ github.workspace }}/.sccache
SCCACHE_CACHE_SIZE: "2G"
steps:
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
Expand Down Expand Up @@ -161,14 +184,32 @@ jobs:
echo "::error::vcpkg install failed after 3 attempts"
exit 1
- name: Set Ubuntu Compilers
# Wrap the compiler with sccache: Meson uses an explicit CC/CXX verbatim,
# so the launcher must be prepended here to route compiles through sccache.
run: |
echo "CC=gcc-14" >> $GITHUB_ENV
echo "CXX=g++-14" >> $GITHUB_ENV
echo "CC=sccache gcc-14" >> $GITHUB_ENV
echo "CXX=sccache g++-14" >> $GITHUB_ENV
- name: Restore sccache cache
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-meson-sigv4-${{ github.run_id }}
restore-keys: |
sccache-meson-sigv4-
- name: Setup sccache
uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10
- name: Build and test Iceberg
shell: bash
env:
CMAKE_PREFIX_PATH: /usr/local/share/vcpkg/installed/x64-linux
run: |
meson setup builddir -Dsigv4=enabled
meson compile -C builddir
sccache --show-stats
meson test -C builddir --timeout-multiplier 0 --print-errorlogs
- name: Save sccache cache
if: github.ref == 'refs/heads/main'
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-meson-sigv4-${{ github.run_id }}
24 changes: 23 additions & 1 deletion .github/workflows/sanitizer_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,9 @@ jobs:
if: ${{ github.event_name != 'pull_request' || github.event.pull_request.draft == false }}
name: "ASAN and UBSAN Tests"
runs-on: ubuntu-24.04
env:
SCCACHE_DIR: ${{ github.workspace }}/.sccache
SCCACHE_CACHE_SIZE: "2G"
steps:
- name: Checkout iceberg-cpp
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
Expand All @@ -47,14 +50,33 @@ jobs:
- name: Install dependencies
shell: bash
run: sudo apt-get update && sudo apt-get install -y libcurl4-openssl-dev
- name: Restore sccache cache
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-sanitizer-ubuntu-${{ github.run_id }}
restore-keys: |
sccache-sanitizer-ubuntu-
- name: Setup sccache
uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10
- name: Configure and Build with ASAN & UBSAN
env:
CC: gcc-14
CXX: g++-14
run: |
mkdir build && cd build
cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Debug -DICEBERG_ENABLE_ASAN=ON -DICEBERG_ENABLE_UBSAN=ON
cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Debug -DICEBERG_ENABLE_ASAN=ON -DICEBERG_ENABLE_UBSAN=ON \
-DCMAKE_C_COMPILER_LAUNCHER=sccache -DCMAKE_CXX_COMPILER_LAUNCHER=sccache
cmake --build . --verbose
- name: Show sccache stats
shell: bash
run: sccache --show-stats
- name: Save sccache cache
if: github.ref == 'refs/heads/main'
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-sanitizer-ubuntu-${{ github.run_id }}
- name: Run Tests
working-directory: build
env:
Expand Down
23 changes: 23 additions & 0 deletions .github/workflows/sql_catalog_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,9 @@ jobs:
runs-on: windows-2025
cmake_build_type: Release
cmake_extra_args: -DCMAKE_TOOLCHAIN_FILE=C:/vcpkg/scripts/buildsystems/vcpkg.cmake
env:
SCCACHE_DIR: ${{ github.workspace }}/.sccache
SCCACHE_CACHE_SIZE: "2G"
steps:
- name: Checkout iceberg-cpp
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
Expand All @@ -85,6 +88,15 @@ jobs:
shell: pwsh
run: |
vcpkg install zlib:x64-windows nlohmann-json:x64-windows nanoarrow:x64-windows roaring:x64-windows sqlite3:x64-windows
- name: Restore sccache cache
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-sqlcatalog-${{ matrix.runs-on }}-${{ github.run_id }}
restore-keys: |
sccache-sqlcatalog-${{ matrix.runs-on }}-
- name: Setup sccache
uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10
- name: Configure Iceberg
shell: bash
run: |
Expand All @@ -96,10 +108,21 @@ jobs:
-DICEBERG_BUILD_REST=OFF \
-DICEBERG_BUILD_SQL_CATALOG=ON \
-DICEBERG_SQL_SQLITE=ON \
-DCMAKE_C_COMPILER_LAUNCHER=sccache \
-DCMAKE_CXX_COMPILER_LAUNCHER=sccache \
${{ matrix.cmake_extra_args }}
- name: Build SQL catalog tests
shell: bash
run: cmake --build build --target sql_catalog_test
- name: Show sccache stats
shell: bash
run: sccache --show-stats
- name: Save sccache cache
if: github.ref == 'refs/heads/main'
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-sqlcatalog-${{ matrix.runs-on }}-${{ github.run_id }}
- name: Run SQL catalog tests
shell: bash
run: ctest --test-dir build -R '^sql_catalog_test$' --output-on-failure -C ${{ matrix.cmake_build_type }}
93 changes: 81 additions & 12 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,9 @@ jobs:
timeout-minutes: 30
strategy:
fail-fast: false
env:
SCCACHE_DIR: ${{ github.workspace }}/.sccache
SCCACHE_CACHE_SIZE: "2G"
steps:
- name: Checkout iceberg-cpp
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
Expand All @@ -53,12 +56,30 @@ jobs:
- name: Install dependencies
shell: bash
run: sudo apt-get update && sudo apt-get install -y libcurl4-openssl-dev
- name: Restore sccache cache
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-test-ubuntu-${{ github.run_id }}
restore-keys: |
sccache-test-ubuntu-
- name: Setup sccache
uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10
- name: Build Iceberg
shell: bash
env:
CC: gcc-14
CXX: g++-14
run: ci/scripts/build_iceberg.sh $(pwd) ON
run: ci/scripts/build_iceberg.sh $(pwd) ON ON
- name: Show sccache stats
shell: bash
run: sccache --show-stats
- name: Save sccache cache
if: github.ref == 'refs/heads/main'
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-test-ubuntu-${{ github.run_id }}
- name: Build Example
shell: bash
env:
Expand All @@ -72,14 +93,35 @@ jobs:
timeout-minutes: 30
strategy:
fail-fast: false
env:
SCCACHE_DIR: ${{ github.workspace }}/.sccache
SCCACHE_CACHE_SIZE: "2G"
steps:
- name: Checkout iceberg-cpp
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
persist-credentials: false
- name: Restore sccache cache
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-test-macos-${{ github.run_id }}
restore-keys: |
sccache-test-macos-
- name: Setup sccache
uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10
- name: Build Iceberg
shell: bash
run: ci/scripts/build_iceberg.sh $(pwd)
run: ci/scripts/build_iceberg.sh $(pwd) OFF ON
- name: Show sccache stats
shell: bash
run: sccache --show-stats
- name: Save sccache cache
if: github.ref == 'refs/heads/main'
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-test-macos-${{ github.run_id }}
- name: Build Example
shell: bash
run: ci/scripts/build_example.sh $(pwd)/example
Expand All @@ -90,6 +132,9 @@ jobs:
timeout-minutes: 60
strategy:
fail-fast: false
env:
SCCACHE_DIR: ${{ github.workspace }}/.sccache
SCCACHE_CACHE_SIZE: "2G"
steps:
- name: Checkout iceberg-cpp
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
Expand All @@ -103,17 +148,28 @@ jobs:
shell: pwsh
run: |
vcpkg install zlib:x64-windows nlohmann-json:x64-windows nanoarrow:x64-windows roaring:x64-windows cpr:x64-windows
- name: Restore sccache cache
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-test-windows-${{ github.run_id }}
restore-keys: |
sccache-test-windows-
- name: Setup sccache
uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10
- name: Build Iceberg
shell: pwsh
env:
SCCACHE_GHA_ENABLED: "true"
run: |
$ErrorActionPreference = "Stop"
bash -lc 'ci/scripts/build_iceberg.sh $(pwd) OFF ON'
if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
sccache --show-stats
- name: Save sccache cache
if: github.ref == 'refs/heads/main'
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-test-windows-${{ github.run_id }}
- name: Build Example
shell: pwsh
run: |
Expand All @@ -124,6 +180,9 @@ jobs:
name: Meson - ${{ matrix.title }}
runs-on: ${{ matrix.runs-on }}
timeout-minutes: 30
env:
SCCACHE_DIR: ${{ github.workspace }}/.sccache
SCCACHE_CACHE_SIZE: "2G"
strategy:
max-parallel: 15
fail-fast: false
Expand Down Expand Up @@ -152,24 +211,34 @@ jobs:
python3 -m pip install --upgrade pip
python3 -m pip install -r requirements.txt
- name: Set Ubuntu Compilers
# Wrap the compiler with sccache: Meson auto-detects sccache for the
# default compiler (macOS/Windows), but uses an explicit CC/CXX verbatim,
# so the launcher must be prepended here to route Ubuntu compiles through it.
if: ${{ startsWith(matrix.runs-on, 'ubuntu') }}
run: |
echo "CC=${{ matrix.CC }}" >> $GITHUB_ENV
echo "CXX=${{ matrix.CXX }}" >> $GITHUB_ENV
echo "CC=sccache ${{ matrix.CC }}" >> $GITHUB_ENV
echo "CXX=sccache ${{ matrix.CXX }}" >> $GITHUB_ENV
- name: Restore sccache cache
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-meson-${{ matrix.runs-on }}-${{ github.run_id }}
restore-keys: |
sccache-meson-${{ matrix.runs-on }}-
- name: Setup sccache
if: ${{ startsWith(matrix.runs-on, 'windows') }}
uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10
- name: Enable sccache
if: ${{ startsWith(matrix.runs-on, 'windows') }}
shell: bash
run: echo "SCCACHE_GHA_ENABLED=true" >> "$GITHUB_ENV"
- name: Build Iceberg
run: |
meson setup builddir ${{ matrix.meson-setup-args || '' }}
meson compile -C builddir
- name: Show sccache stats
if: ${{ startsWith(matrix.runs-on, 'windows') }}
run: sccache --show-stats
- name: Save sccache cache
if: github.ref == 'refs/heads/main'
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-meson-${{ matrix.runs-on }}-${{ github.run_id }}
- name: Test Iceberg
run: |
meson test -C builddir --timeout-multiplier 0 --print-errorlogs
Loading