Skip to content

Conversation

@debarshiray
Copy link
Member

@gemini-code-assist
Copy link

Summary of Changes

Hello @debarshiray, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request is part of the ongoing effort to prepare a stable branch for an upcoming release, potentially addressing security vulnerabilities. It enhances the continuous integration pipeline by introducing system tests for Fedora 40, thereby expanding test coverage and ensuring the project's stability and compatibility with newer operating system versions. This proactive measure helps maintain the quality and reliability of the codebase.

Highlights

  • New CI Job for Fedora 40: A new Zuul job, system-test-fedora-40, has been introduced to run system tests specifically on the Fedora 40 environment.
  • Expanded Test Coverage: The newly added Fedora 40 system tests are now integrated into the periodic, check, and gate job lists, ensuring comprehensive testing for all changes and regular stability checks.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@debarshiray debarshiray force-pushed the wip/rishi/prepare-release-0.0.99.4-part-2 branch from 78c171a to 8003413 Compare January 29, 2026 13:33
debarshiray added a commit to debarshiray/toolbox that referenced this pull request Jan 29, 2026
Copy link

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request successfully introduces a new system test job for Fedora 40 into the Zuul CI/CD configuration. The new job definition is consistent with existing Fedora system tests, and its inclusion in the periodic, check, and gate sections ensures comprehensive testing coverage for the new environment. The changes are well-structured and align with the project's existing CI practices.

@softwarefactory-project-zuul
Copy link

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/8f6a891dd5be4fa1a0204bd3d05c118d

✔️ unit-test SUCCESS in 1m 52s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 14s
✔️ system-test-fedora-rawhide SUCCESS in 12m 44s
system-test-fedora-40 FAILURE in 19m 17s
✔️ system-test-fedora-39 SUCCESS in 18m 52s
system-test-fedora-38 NODE_FAILURE Node request 200-0008149244 failed in 0s

petraglyph and others added 4 commits January 29, 2026 15:14
Any image or container that has APT or systemd may have /etc/kernel.
eg., the arch-toolbox and ubuntu-toolbox images.

containers#1409
containers#1750

Signed-off-by: Penn Bauman <me@pennbauman.com>
(backported from commit 2f3ee0e)
The working directory from which bats(1) is invoked might not be part of
the Toolbx container.  eg., Podman's downstream Fedora CI invokes the
tests as:
  $ cd /path/to/toolbox/test/system
  $ bats .

... and it led to [1]:
  not ok 110 run: Smoke test with true(1)
  # (from function `assert_output' in file
      ./libs/bats-assert/src/assert.bash, line 255,
  #  in test file ./104-run.bats, line 38)
  #   `assert_output ""' failed
  #
  # -- output differs --
  # expected (0 lines):
  #
  # actual (3 lines):
  #   Error: crun: chdir to `/usr/share/toolbox/test/system`: No such
      file or directory: OCI runtime attempted to invoke a command that
      was not found
  #   Error: directory /usr/share/toolbox/test/system not found in
      container fedora-toolbox-41
  #   Using /home/testuser instead.
  # --
  #

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2263968

containers#1457
containers#1740
containers#1750
(cherry picked from commit 9d3ae61)
(backported from commit c0fb8f3)
Fedora 38 reached End of Life on 21st May 2024:
https://docs.fedoraproject.org/en-US/releases/eol/

containers#1527
containers#1741
containers#1750
(cherry picked from commit b684b19)
(cherry picked from commit 4e54402)
@debarshiray debarshiray force-pushed the wip/rishi/prepare-release-0.0.99.4-part-2 branch from 8003413 to acae699 Compare January 29, 2026 14:19
@softwarefactory-project-zuul
Copy link

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/70f488814e3e4df8ba816bb0c1681536

✔️ unit-test SUCCESS in 1m 43s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 07s
✔️ system-test-fedora-rawhide SUCCESS in 10m 34s
✔️ system-test-fedora-39 SUCCESS in 17m 30s
system-test-fedora-38 NODE_FAILURE Node request 200-0008149314 failed in 0s

@debarshiray
Copy link
Member Author

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/8f6a891dd5be4fa1a0204bd3d05c118d

✔️ unit-test SUCCESS in 1m 52s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 14s
✔️ system-test-fedora-rawhide SUCCESS in 12m 44s
system-test-fedora-40 FAILURE in 19m 17s
✔️ system-test-fedora-39 SUCCESS in 18m 52s
system-test-fedora-38 NODE_FAILURE Node request 200-0008149244 failed in 0s

This was due to:

fedora-40 | not ok 54 run: Ensure that /home/zuul-worker is used as a fallback working directory in 2298ms
fedora-40 | # (from function `assert_line' in file test/system/libs/bats-assert/src/assert.bash, line 488,
fedora-40 | #  in test file test/system/104-run.bats, line 175)
fedora-40 | #   `assert_line --index 0 "$HOME"' failed
fedora-40 | # /etc/kernel ~/src/github.com/containers/toolbox
fedora-40 | # ~/src/github.com/containers/toolbox
fedora-40 | #
fedora-40 | # -- line differs --
fedora-40 | # index    : 0
fedora-40 | # expected : /home/zuul-worker
fedora-40 | # actual   : /etc/kernel
fedora-40 | # --
fedora-40 | #

... which was fixed in commit 2f3ee0e in the main branch.

@softwarefactory-project-zuul
Copy link

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/6806cdc8dd8544d086ef674b3c8043ca

✔️ unit-test SUCCESS in 1m 45s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 07s
✔️ system-test-fedora-rawhide SUCCESS in 10m 36s
✔️ system-test-fedora-40 SUCCESS in 16m 54s
✔️ system-test-fedora-39 SUCCESS in 16m 28s
system-test-fedora-38 NODE_FAILURE Node request 200-0008149351 failed in 0s

@softwarefactory-project-zuul
Copy link

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/8feec35d2bb242eba4b1ea60a5c73251

unit-test RETRY_LIMIT in 47s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 02s
system-test-fedora-rawhide RETRY_LIMIT in 47s
✔️ system-test-fedora-40 SUCCESS in 16m 59s
✔️ system-test-fedora-39 SUCCESS in 16m 57s
system-test-fedora-38 NODE_FAILURE Node request 200-0008149416 failed in 0s

@debarshiray
Copy link
Member Author

recheck

@softwarefactory-project-zuul
Copy link

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/05af62d249214097b525bfc2a357f534

unit-test RETRY_LIMIT in 50s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 06s
system-test-fedora-rawhide RETRY_LIMIT in 47s
✔️ system-test-fedora-40 SUCCESS in 16m 55s
✔️ system-test-fedora-39 SUCCESS in 16m 18s
system-test-fedora-38 NODE_FAILURE Node request 200-0008149526 failed in 0s

@debarshiray
Copy link
Member Author

recheck

@softwarefactory-project-zuul
Copy link

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/dad596c77907479d83e59bb563ac2568

✔️ unit-test SUCCESS in 1m 41s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 08s
✔️ system-test-fedora-rawhide SUCCESS in 10m 37s
✔️ system-test-fedora-40 SUCCESS in 16m 50s
✔️ system-test-fedora-39 SUCCESS in 16m 36s
system-test-fedora-38 NODE_FAILURE Node request 200-0008150885 failed in 0s

@debarshiray
Copy link
Member Author

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/8feec35d2bb242eba4b1ea60a5c73251

unit-test RETRY_LIMIT in 47s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 02s
system-test-fedora-rawhide RETRY_LIMIT in 47s
✔️ system-test-fedora-40 SUCCESS in 16m 59s
✔️ system-test-fedora-39 SUCCESS in 16m 57s
system-test-fedora-38 NODE_FAILURE Node request 200-0008149416 failed in 0s

This was due to:

TASK [Install RPM packages]
fedora-rawhide | MODULE FAILURE:
fedora-rawhide | libdnf5::OptionBool& libdnf5::ConfigMain::get_gpgcheck_option() is deprecated: Use get_pkg_gpgcheck_option()
fedora-rawhide | Traceback (most recent call last):
fedora-rawhide |   File "<stdin>", line 107, in <module>
fedora-rawhide |   File "<stdin>", line 99, in _ansiballz_main
fedora-rawhide |   File "<stdin>", line 47, in invoke_module
fedora-rawhide |   File "<frozen runpy>", line 226, in run_module
fedora-rawhide |   File "<frozen runpy>", line 98, in _run_module_code
fedora-rawhide |   File "<frozen runpy>", line 88, in _run_code
fedora-rawhide |   File "/tmp/ansible_ansible.legacy.dnf5_payload_kmez8z38/ansible_ansible.legacy.dnf5_payload.zip/ansible/modules/dnf5.py", line 717, in <module>
fedora-rawhide |   File "/tmp/ansible_ansible.legacy.dnf5_payload_kmez8z38/ansible_ansible.legacy.dnf5_payload.zip/ansible/modules/dnf5.py", line 713, in main
fedora-rawhide |   File "/tmp/ansible_ansible.legacy.dnf5_payload_kmez8z38/ansible_ansible.legacy.dnf5_payload.zip/ansible/modules/dnf5.py", line 680, in run
fedora-rawhide |   File "/usr/lib64/python3.14/site-packages/libdnf5/base.py", line 936, in download
fedora-rawhide |     return _base.Transaction_download(self)
fedora-rawhide |            ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
fedora-rawhide | libdnf5.exception.RepoPackageDownloadErrorNested: Failed to download packages

It looks like a breakage in Fedora Rawhide that eventually got fixed.

It's far more consistent and understandable if all tests start with a
clean state without any containers or images present.  Otherwise, the
subtle side-effects of having some image left behind from a previous
test can lead to surprises, and there's no need to spend time wondering
whether some tests should only clean up the containers or both
containers and images.

This additional work of cleaning up the images for all tests makes it
necessary to increase the timeout for all Fedora nodes to prevent the CI
from timing out.

containers#1526
containers#1741
containers#1750
(backported from commits 67d4002 and
 55c0e63)
(backported from commit 23b4a34)
@softwarefactory-project-zuul
Copy link

Build succeeded.
https://softwarefactory-project.io/zuul/t/local/buildset/0d12a3452e414324bbdffb2a55cc04d5

✔️ unit-test SUCCESS in 1m 48s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 02s
✔️ system-test-fedora-rawhide SUCCESS in 10m 28s
✔️ system-test-fedora-40 SUCCESS in 16m 53s
✔️ system-test-fedora-39 SUCCESS in 16m 22s

Currently, the CI has been frequently timing out on stable Fedora nodes.
So, increase the timeout from 1 hour 50 minutes to 2 hours to avoid
that.

For what it's worth, the timeout for Fedora Rawhide nodes is 2 hours 10
minutes and it seems enough.

containers#1546
containers#1741
containers#1750
(backported from commit f2dc3b8)
(cherry picked from commit 316a3b1)
This keeps the timeout for the Fedora nodes synchronized with the main
branch.

containers#1548
containers#1741
containers#1750
(backported from commit 83f28c5)
(cherry picked from commit c09ef38)
The system tests download several images when setting up the test suite,
and cache them for later use by the tests [1].  This saves time and
avoids hitting rate limits imposed by OCI registries by not downloading
the same images repeatedly for several tests, but at the cost of
increased use of storage space to cache the images.

The images are cached under BATS_TMPDIR.  It defaults to the TMPDIR
environment variable, and if that's not set then to /tmp [2].  Normally,
TMPDIR isn't set, and the images end up getting cached under /tmp.  Now,
/tmp is typically on tmpfs backed by RAM or swap, which means that it
should be used for smaller size-bounded files only, and /var/tmp should
be used for everything else [3].

The images are big enough that a collection of them can't be described
as smaller and size-bounded, and it led to:
  1..306
  # test suite: Set up
  # test suite: Tear down
  not ok 1 setup_suite
  # (from function `setup_suite' in test file ./setup_suite.bash, line
      55)
  #   `_pull_and_cache_distro_image fedora "$((system_version-1))" ||
      false' failed
  # Failed to cache image registry.fedoraproject.org/fedora-toolbox:40
      to /tmp/bats-run-IPz4Cn/image-cache/fedora-toolbox-40
  # time="2024-02-19T11:41:43Z" level=fatal msg="copying system image
      from manifest list: writing blob: write
      /tmp/bats-run-IPz4Cn/image-cache/fedora-toolbox-40/dir-put-blob607392514:
      no space left on device"
  # bats warning: Executed 1 instead of expected 306 tests

So, change the default location of the BATS_TMPDIR environment variable
to /var/tmp by setting TMPDIR.

[1] Commit 50683c9
    containers@50683c9d9a78adc9
    containers#375

[2] https://bats-core.readthedocs.io/en/stable/writing-tests.html

[3] https://systemd.io/TEMPORARY_DIRECTORIES/

containers#1462
containers#1742
containers#1750
(backported from commit 571dc97)
(cherry picked from commit dc8a35d)
@softwarefactory-project-zuul
Copy link

Build succeeded.
https://softwarefactory-project.io/zuul/t/local/buildset/6169270f89cf4e73b49b7b234fe8622b

✔️ unit-test SUCCESS in 1m 54s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 02s
✔️ system-test-fedora-rawhide SUCCESS in 30m 56s
✔️ system-test-fedora-40 SUCCESS in 33m 41s
✔️ system-test-fedora-39 SUCCESS in 33m 21s

@softwarefactory-project-zuul
Copy link

Build succeeded.
https://softwarefactory-project.io/zuul/t/local/buildset/a4cb5f58a2184728a6c7240f0e7137ca

✔️ unit-test SUCCESS in 1m 50s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 04s
✔️ system-test-fedora-rawhide SUCCESS in 28m 17s
✔️ system-test-fedora-40 SUCCESS in 34m 15s
✔️ system-test-fedora-39 SUCCESS in 32m 52s

@softwarefactory-project-zuul
Copy link

Build succeeded.
https://softwarefactory-project.io/zuul/t/local/buildset/bff972ffd3eb4b20a2dac9776a93fa74

✔️ unit-test SUCCESS in 1m 44s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 40s
✔️ system-test-fedora-rawhide SUCCESS in 29m 38s
✔️ system-test-fedora-40 SUCCESS in 33m 22s
✔️ system-test-fedora-39 SUCCESS in 32m 35s

@softwarefactory-project-zuul
Copy link

Build succeeded.
https://softwarefactory-project.io/zuul/t/local/buildset/e8c440ba43b94744a61d2aac96602edf

✔️ unit-test SUCCESS in 1m 44s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 13s
✔️ system-test-fedora-rawhide SUCCESS in 30m 08s
✔️ system-test-fedora-40 SUCCESS in 34m 39s
✔️ system-test-fedora-39 SUCCESS in 33m 46s

@softwarefactory-project-zuul
Copy link

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/6a8d969d34814d9a9c644701d04e18c3

✔️ unit-test SUCCESS in 1m 43s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 04s
✔️ system-test-fedora-rawhide SUCCESS in 28m 20s
system-test-fedora-41 FAILURE in 32m 32s
✔️ system-test-fedora-40 SUCCESS in 34m 09s
✔️ system-test-fedora-39 SUCCESS in 33m 15s

@debarshiray
Copy link
Member Author

recheck

debarshiray added a commit to debarshiray/toolbox that referenced this pull request Feb 2, 2026
The working directory from which bats(1) is invoked might not be part of
the Toolbx container.  eg., the downstream Fedora CI invokes the tests
as:
  $ cd /path/to/toolbox/test/system
  $ bats .

... and it led to:
  not ok 8 help: Try unknown command (forwarded to host)
  # tags: commands-options
  # (from function `assert_line' in file
      ./libs/bats-assert/src/assert.bash, line 488,
  #  in test file ./002-help.bats, line 135)
  #   `assert_line --index 0
        "Error: unknown command \"foo\" for \"toolbox\""' failed
  #
  # -- line differs --
  # index    : 0
  # expected : Error: unknown command "foo" for "toolbox"
  # actual   : Error: crun: chdir to `/usr/share/toolbox/test/system`:
      No such file or directory: OCI runtime attempted to invoke a
      command that was not found
  # --
  #

containers#1560
containers#1745
containers#1750
(backported from commit 1e90c72)
(cherry picked from commit acdafef)
debarshiray added a commit to debarshiray/toolbox that referenced this pull request Feb 2, 2026
Fedora 39 reached End of Life on 26th November 2024:
https://docs.fedoraproject.org/en-US/releases/eol/

containers#1602
containers#1745
containers#1750
(backported from commit 0bb4ff8)
(cherry picked from commit 23a9850)
@softwarefactory-project-zuul
Copy link

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/f22f2f46787649c697db0b711612e77f

✔️ unit-test SUCCESS in 1m 39s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 17s
✔️ system-test-fedora-rawhide SUCCESS in 30m 56s
system-test-fedora-41 FAILURE in 34m 08s
✔️ system-test-fedora-40 SUCCESS in 35m 58s
✔️ system-test-fedora-39 SUCCESS in 35m 05s

@debarshiray
Copy link
Member Author

recheck

@softwarefactory-project-zuul
Copy link

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/824e5fe62d434caaacc3def1026f3200

✔️ unit-test SUCCESS in 1m 41s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 23s
✔️ system-test-fedora-rawhide SUCCESS in 31m 35s
system-test-fedora-41 FAILURE in 35m 24s
✔️ system-test-fedora-40 SUCCESS in 34m 46s
✔️ system-test-fedora-39 SUCCESS in 32m 50s

@debarshiray debarshiray merged commit 488b0ba into containers:release-0.0.99.4 Feb 2, 2026
1 of 2 checks passed
@debarshiray debarshiray deleted the wip/rishi/prepare-release-0.0.99.4-part-2 branch February 2, 2026 22:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants