Post-PR failures September 2026

[traversaro]

ros-lyrical-ur-simulation-gz
ros-lyrical-ur
ros-lyrical-moveit
Traceback (most recent call last):
  File "/home/runner/work/ros-lyrical/ros-lyrical/.pixi/envs/default/bin/vinca-gha", line 10, in <module>
    sys.exit(main())
             ~~~~^^
  File "/home/runner/work/ros-lyrical/ros-lyrical/.pixi/envs/default/lib/python3.14/site-packages/vinca/generate_gha.py", line 589, in main
    build_unix_pipeline(
    ~~~~~~~~~~~~~~~~~~~^
        stages,
        ^^^^^^^
    ...<6 lines>...
        pixi_version=pixi_version,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/runner/work/ros-lyrical/ros-lyrical/.pixi/envs/default/lib/python3.14/site-packages/vinca/generate_gha.py", line 280, in build_unix_pipeline
    dump_for_gha(workflow, outfile)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/ros-lyrical/ros-lyrical/.pixi/envs/default/lib/python3.14/site-packages/vinca/generate_gha.py", line 160, in dump_for_gha
    raise RuntimeError(
    ...<3 lines>...
    )
RuntimeError: Generated workflow linux_aarch64.yml is 504.7 KiB, exceeding the 500 KiB limit. Increase --batch_size to reduce the number of generated jobs.

https://github.com/RoboStack/ros-lyrical/actions/runs/34697410459

[traversaro]

The originally reported issue was fixed by @sea-bass in #49 .

However, we have a much more tricky issue now on linux-64. Due to an error on my side, I halted the first job after the mutex package was produced and uploaded, but now all the re-generated job fail with:

2026-09-13T07:49:53.9726582Z  │ │ Resolving host environment:
2026-09-13T07:49:53.9727400Z  │ │   Platform: linux-64 [__unix=0=0, __linux=6.17.0=0, __glibc=2.39=0, __archspec=1=zen2]
2026-09-13T07:49:53.9728157Z  │ │   Channels: 
2026-09-13T07:49:53.9728659Z  │ │    - file:///home/runner/conda-bld/
2026-09-13T07:49:53.9729875Z  │ │    - https://prefix.dev/robostack-lyrical/
2026-09-13T07:49:53.9730553Z  │ │    - https://prefix.dev/conda-forge/
2026-09-13T07:49:53.9731082Z  │ │   Specs:
2026-09-13T07:49:53.9731479Z  │ │    - pkg-config
2026-09-13T07:49:53.9731894Z  │ │    - numpy 2.*
2026-09-13T07:49:53.9732306Z  │ │    - pip
2026-09-13T07:49:53.9732732Z  │ │    - python 3.14.* *_cp314
2026-09-13T07:49:53.9733339Z  │ │    - ros2-ament-cmake-core
2026-09-13T07:49:53.9733909Z  │ │    - ros2-ament-package
2026-09-13T07:49:53.9734384Z  │ │    - ros2-distro-mutex 0.19.* lyrical_*
2026-09-13T07:49:53.9734720Z  │ │    - libstdcxx >=14
2026-09-13T07:49:53.9735067Z  │ │    - libgcc >=14
2026-09-13T07:49:53.9735795Z  │ │    - libgcc >=14
2026-09-13T07:49:53.9736267Z  │ │    - __glibc >=2.28,<3.0.a0
2026-09-13T07:49:54.5758094Z  │ │
2026-09-13T07:49:54.5758473Z  │ ╰─────────────────── (took 3 seconds)
2026-09-13T07:49:54.5758810Z  │
2026-09-13T07:49:54.5759074Z  ╰─────────────────── (took 5 seconds)
2026-09-13T07:49:54.5858483Z Error:   × Failed to resolve dependencies: Cannot solve the request because of: No
2026-09-13T07:49:54.5859131Z   │ candidates were found for ros2-distro-mutex 0.19.* lyrical_*.
2026-09-13T07:49:54.5859503Z   │ 
2026-09-13T07:49:54.5859878Z   ╰─▶ Cannot solve the request because of: No candidates were found for ros2-
2026-09-13T07:49:54.5860893Z       distro-mutex 0.19.* lyrical_*.
2026-09-13T07:49:54.5861163Z       

See for example https://github.com/RoboStack/ros-lyrical/actions/runs/34724537008 .

That is strange as the package is indeed there: https://prefix.dev/channels/robostack-lyrical/packages/ros2-distro-mutex .

[traversaro]

I even tried restarting the original job (the one that produced ros2-distro-mutex ) at https://github.com/RoboStack/ros-lyrical/actions/runs/34697487039 . But the problem is exactly the same.

[sea-bass]

Is there a way to delete what is on the channel first in case it was corrupted somehow, and then rerunning that generating job?

[traversaro]

Is there a way to delete what is on the channel first in case it was corrupted somehow, and then rerunning that generating job?

I asked the AI to investigate, and I think it found the problem: https://chatgpt.com/share/6aa68489-d93c-83ed-86c0-51fc626cba8f , that I reported in prefix-dev/prefix-dev#97 .

To actually fix the problem, as suggested by ChatGPT it was sufficient to manually reindex the channel:

curl -fSs -X POST \
  -H "Authorization: Bearer ${PREFIX_API_KEY}" \
  https://prefix.dev/api/v1/reindex/robostack-lyrical/linux-64

and then restart the job.

[traversaro]

Ok, now the problem is different:

 + pixi run upload /home/runner/conda-bld//linux-64/ros-lyrical-ament-cmake-core-2.8.8-hb0f4dca_23.conda --skip-existing
✨ Pixi task (upload): rattler-build upload prefix -c robostack-lyrical --generate-attestation /home/runner/conda-bld//linux-64/ros-lyrical-ament-cmake-core-2.8.8-hb0f4dca_23.conda --skip-existing
+ for conda_package in "${conda_packages[@]}"
+ pixi run upload /home/runner/conda-bld//linux-64/ros2-ros-environment-4.5.1-np2py314hb48eaa3_23.conda --skip-existing
✨ Pixi task (upload): rattler-build upload prefix -c robostack-lyrical --generate-attestation /home/runner/conda-bld//linux-64/ros2-ros-environment-4.5.1-np2py314hb48eaa3_23.conda --skip-existing
Error:   × Failed to generate attestation for ros2-ros-environment-4.5.1-
  │ np2py314hb48eaa3_23.conda: Failed to sign attestation with Sigstore:
  │ Signing error: Failed to get timestamp: HTTP error: TSA returned status
  │ 400 Bad Request
  │ Upload aborted because attestation generation was requested but failed.
  │ 
  │ Troubleshooting:
  │ 1. Check that you're running in a supported CI environment (GitHub
  │ Actions, GitLab CI, etc.)
  │ 2. For GitHub Actions, ensure you have 'id-token: write' permission
  │ 3. Verify OIDC token is available and valid

[sea-bass]

Oh, I got a few of those too. Seems you can just rerun those jobs and it will work next time. (Which it did; I just restarted)

But yes, I have encountered lots of little flakes like that.

[sea-bass]

Just restarted the linux64 workflow again; the other flake that happens is sometimes the file that was just uploaded in a previous job doesn't yet get found by the next ones.

https://github.com/RoboStack/ros-lyrical/actions/runs/34754003472/job/103721571724

EDIT: Had to do this 7 more times for this build