Stop linking the Python bindings library into C++ consumers of interface packages (fixes dyld _PyExc_RuntimeError / ros-kilted#76)

[mini-1235]

Problem

Every ROS 2 interface package exports its Python C bindings library, lib<pkg>__rosidl_generator_py, in <pkg>_TARGETS (via ament_export_targets in rosidl_generator_py). So a plain C++ node that does target_link_libraries(node ${std_msgs_TARGETS}) links it. That library calls CPython functions but is only meant to be loaded by a Python interpreter:

• macOS (RoboStack links it with Python3::Module, i.e. -undefined dynamic_lookup): the C++ node aborts at startup:

  dyld[...]: symbol not found in flat namespace '_PyExc_RuntimeError'
  

  This used to be hidden by -Wl,-dead_strip_dylibs from the compiler activation. compilers 2.x (clang 21) no longer sets LDFLAGS, so downstream projects now hit it.
• Linux with Python3::Module: undefined references at link time, or symbol lookup error at load time. This is Static linking issue with _rosidl_generator_py.so in build _17 ros-kilted#76. Linking Python3::Python on Linux hides it, but then every C++ node loads libpython.

Linking Python3::Python on macOS isn't a fix: it loads a second interpreter into conda-forge's statically linked python (see ros2/rosidl_python#253). Upstream #253 is stuck on the Linux side for the same reason: ROS CI links shared libraries with --no-undefined.

Fix

This removes the shared library instead of patching around it. It replaces ros-rolling-rosidl-generator-py.osx.patch with a cross-platform ros-rolling-rosidl-generator-py.patch:

• The generated conversion code (_*_s.c) is compiled as an OBJECT library into each <pkg>_s__rosidl_typesupport_* Python extension module, which already links Python3::Module. Nothing is installed or exported, so <pkg>_TARGETS only contains C/C++ libraries. No library is left with unresolved CPython symbols, and no linker flags are needed.
• Nested types from other packages (e.g. builtin_interfaces/Time in std_msgs/Header): the conversion functions are read (and cached) from the dependency's Python message class. They come from the _CONVERT_FROM_PY / _CONVERT_TO_PY PyCapsules, which rclpy already uses and which the parent's __import_type_support__ already loads. This replaces linking lib<dep>__rosidl_generator_py. Same-package types still call the functions directly.

Commits

1. Add regression test (tests/ros-rolling-std-msgs.yaml). It builds and runs a C++ executable against ${std_msgs_TARGETS} (unix only), with LDFLAGS cleared so --as-needed / -dead_strip_dylibs can't hide the problem. It also checks that the Python bindings still work (rclpy serialize round-trip). std_msgs → build 26 so CI runs the test. Expected to fail on macOS. On Linux the library currently links libpython (Python3::Python), which hides the leak: the test passes, but every C++ consumer loads libpython.
2. The patch, plus rosidl_generator_py, rosidl_core_generators and rosidl_default_generators → build 26. std_msgs only depends on the generator through the latter two, so they have to be rebuilt too; otherwise rattler-build may build std_msgs first, against the old generator from the channel. The test is expected to pass.
3. [DO NOT MERGE] A temporary testpr.yml line that drops the std_msgs build 26 cached by this PR's earlier CI runs, which was built with the old generator. Please drop this commit before merging.

Local results (rattler-build, this repo)

	osx-arm64	linux-64
Commit 1 (current patch)	❌ dyld: symbol not found in flat namespace '_PyExc_RuntimeError'	✅ runs, but links libstd_msgs__rosidl_generator_py → libpython
Linux, Python3::Module only (kilted #76 setup)	–	❌ undefined reference to PyObject_GetAttrString (and friends)
Commit 2	✅ both tests	✅ both tests
Python round-trip of every msg/srv/action type in std_msgs, geometry_msgs, test_msgs, builtin_interfaces, service_msgs	✅	✅ 124/124

Things to be aware of

• The Python bindings of all interface packages need rebuilding together (e.g. in the next full rebuild). Packages built by the old generator link lib<dep>__rosidl_generator_py, which a dependency rebuilt with this patch no longer ships. I reproduced this locally: an old service_msgs failed to import against a rebuilt builtin_interfaces, and worked after rebuilding it. The other direction (new packages on top of old dependencies) works. This PR only bumps the two packages needed to exercise the test. Pure C++ consumers are unaffected.
• Not tested yet: Windows, the upstream rosidl_python / rclpy test suites, and code outside the message packages that links <pkg>__rosidl_generator_py directly (none found in the rolling env).
• If this approach looks right, I'd propose the same change upstream in ros2/rosidl_python.

---

This PR description was AI-generated with Claude Opus 5.5 (claude-opus-5-5).

🤖 Generated with Claude Code

[mini-1235]

Macos failing with:

 -- Build files have been written to: $SRC_DIR/build
 │ [1/2] Building CXX object CMakeFiles/std_msgs_cpp_consumer.dir/main.cpp.o
 │ [2/2] Linking CXX executable std_msgs_cpp_consumer
 │ ld: warning: dylib ($PREFIX/lib/libstd_msgs__rosidl_typesupport_fastrtps_cpp.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/libstd_msgs__rosidl_typesupport_introspection_c.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/libstd_msgs__rosidl_typesupport_introspection_cpp.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/libstd_msgs__rosidl_typesupport_cpp.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/libstd_msgs__rosidl_typesupport_fastrtps_c.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/libstd_msgs__rosidl_generator_py.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/libstd_msgs__rosidl_typesupport_c.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/libstd_msgs__rosidl_generator_c.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/libbuiltin_interfaces__rosidl_typesupport_fastrtps_cpp.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/libbuiltin_interfaces__rosidl_typesupport_fastrtps_c.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/librosidl_typesupport_fastrtps_cpp.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/librosidl_typesupport_fastrtps_c.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/libfastcdr.2.3.6.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/librmw.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/librosidl_dynamic_typesupport.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/libbuiltin_interfaces__rosidl_typesupport_introspection_cpp.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/librosidl_typesupport_introspection_cpp.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/libbuiltin_interfaces__rosidl_typesupport_introspection_c.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/librosidl_typesupport_introspection_c.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/libbuiltin_interfaces__rosidl_generator_py.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/libbuiltin_interfaces__rosidl_typesupport_cpp.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/libbuiltin_interfaces__rosidl_typesupport_c.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/libbuiltin_interfaces__rosidl_generator_c.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/librosidl_runtime_c.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/librcutils.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ ld: warning: dylib ($PREFIX/lib/librosidl_buffer.dylib) was built for newer macOS version (14.0) than being linked (11.0)
 │ dyld[5227]: symbol not found in flat namespace '_PyExc_RuntimeError'
 │ $SRC_DIR/conda_build.sh: line 11:  5227 Abort trap: 6           ./build/std_msgs_cpp_consumer
 │ × error Script failed with status 134
 │ × error 
 │ × error Script execution failed.
 │ × error 
 │ × error   Work directory: /Users/runner/work/ros-rolling/ros-rolling/output/test/test_ros2-std-msgsdiVrKB/test_run_env/etc/conda/test-files/ros2-std-msgs/0
 │ × error   Prefix: /Users/runner/work/ros-rolling/ros-rolling/output/test/test_ros2-std-msgsdiVrKB/test_run_env
 │ × error   Build prefix: /Users/runner/work/ros-rolling/ros-rolling/output/test/test_ros2-std-msgsdiVrKB/test_build_env
 │ × error 
 │ × error To run the script manually, use the following command:
 │ × error 
 │ × error   cd "/Users/runner/work/ros-rolling/ros-rolling/output/test/test_ros2-std-msgsdiVrKB/test_run_env/etc/conda/test-files/ros2-std-msgs/0" && ./conda_build.sh
 │ × error 
 │ × error To run commands interactively in the build environment:
 │ × error 
 │ × error   cd "/Users/runner/work/ros-rolling/ros-rolling/output/test/test_ros2-std-msgsdiVrKB/test_run_env/etc/conda/test-files/ros2-std-msgs/0" && source build_env.sh
 │
 ╰─────────────────── (took 8 seconds)
Error:   × Test failed: failed to run test: IO Error: Script failed

After adding the tests (1st commit)

[mini-1235]

All green after adding the patch (2nd commit)

[mini-1235]

@traversaro I added a few tests related to RoboStack/ros-kilted#76, ros2/rosidl_python#253, and a longstanding issue I have encountered when developing ROS 2 on macOS.

Let me first describe the macOS issue.

So far I have been using ROS Humble, Lyrical, and Rolling on macOS for ROS 2 development, for example compiling Nav2 from source and working on personal projects. Across all of these distros, whenever I start a new project I eventually run into errors involving symbols such as _PyExc_RuntimeError, similar to what can be seen in CI after the first commit in this PR.

My workaround has usually been to add something like this where needed:

component_container_env = {}
if sys.platform == 'darwin':
    python_library = f'libpython{sysconfig.get_python_version()}.dylib'
    component_container_env['DYLD_INSERT_LIBRARIES'] = os.path.join(
        sys.prefix, 'lib', python_library
    )

I believe something similar could also be added through the Pixi activation environment, but either way this feels more like a workaround than a good development experience.

Based on my investigation, this problem does not normally show up in the RoboStack prebuilt binaries. The reason seems to be that those binaries are built in an environment where the linker drops unused libraries, so the Python bindings library does not remain as a runtime dependency.

A more detailed explanation from my agent is:

Why the prebuilt binaries are clean:

CMake does add the Python library to the link. When nav2 was built, target_link_libraries(... ${geometry_msgs_TARGETS}) included libgeometry_msgs__rosidl_generator_py, just like in your own build.

The linker then drops it. RoboStack pins clang 19 (c_compiler_version: 19 in conda_build_config.yaml), and that compiler's activation exports LDFLAGS containing:

macOS: -Wl,-dead_strip_dylibs, which removes linked dylibs nothing uses. A C++ node never calls the Python conversion functions, so the library is dropped.

Linux: -Wl,--as-needed, which does the same for .so files.

So the published binary has no reference to the Python library. dyld never loads it, and the unresolved _PyExc_RuntimeError never matters.

Why your own build crashed: your project uses compilers >=2 (clang 21). The newer activation no longer exports LDFLAGS or CMAKE_ARGS, so nothing strips the unused library. Your executable keeps a load command for libstd_msgs__rosidl_generator_py.dylib, and dyld aborts. We showed this earlier: with compilers 1.x LDFLAGS contained -dead_strip_dylibs and the executable linked 2 dylibs. With 2.x LDFLAGS was empty, the executable linked 28 dylibs including the Python one, and adding -Wl,-dead_strip_dylibs back fixed it.

So you'll see the error when you:

• build C++ code against ROS messages yourself, and

• your toolchain doesn't strip unused libraries: compilers 2.x, plain Xcode clang, or Linux without --as-needed.

On Linux with Python3::Module that gives kilted #76's link errors.

After investigating this, I ended up with two possible fixes.

The larger fix is the one currently proposed in this PR. It makes the generated conversion code part of the Python extension directly and performs cross-package converter lookup at runtime through the Python message classes. This also means everything uses Python3::Module.

A smaller alternative would be:

diff --git a/rosidl_generator_py/cmake/rosidl_generator_py_generate_interfaces.cmake b/rosidl_generator_py/cmake/rosidl_generator_py_generate_interfaces.cmake
index 2810984..f815eab 100644
--- a/rosidl_generator_py/cmake/rosidl_generator_py_generate_interfaces.cmake
+++ b/rosidl_generator_py/cmake/rosidl_generator_py_generate_interfaces.cmake
@@ -38,6 +38,11 @@ if(NOT TARGET Python3::Module OR NOT TARGET Python3::NumPy)
   find_package(Python3 REQUIRED COMPONENTS Interpreter Development NumPy)
 endif()
 
+# The Python C bindings library of an interface package is exported in
+# <pkg>_TARGETS__rosidl_generator_py instead of <pkg>_TARGETS, so that C/C++
+# consumers of the interface package don't link it.
+set(rosidl_generator_py_suffix "__rosidl_generator_py")
+
 # Get a list of typesupport implementations from valid rmw implementations.
 rosidl_generator_py_get_typesupports(_typesupport_impls)
 
@@ -165,10 +170,18 @@ add_dependencies(
   ${rosidl_generate_interfaces_TARGET}__rosidl_typesupport_c
 )
 
+# On macOS, link Python3::Module (-undefined dynamic_lookup): linking libpython
+# would load a second interpreter into a python executable that links it
+# statically (e.g. conda-forge), which crashes.
+if(APPLE)
+  set(_python_target Python3::Module)
+else()
+  set(_python_target Python3::Python)
+endif()
 target_link_libraries(
   ${_target_name_lib} PRIVATE
   Python3::NumPy
-  Python3::Python
+  ${_python_target}
 )
 target_include_directories(${_target_name_lib}
   PRIVATE
@@ -260,9 +273,20 @@ if(NOT rosidl_generate_interfaces_SKIP_INSTALL)
     LIBRARY DESTINATION lib
     RUNTIME DESTINATION bin)
 
-  # Export this target so downstream interface packages can depend on it
-  rosidl_export_typesupport_targets("${rosidl_generator_py_suffix}" "${_target_name_lib}")
-  ament_export_targets(export_${_target_name_lib})
+  # Export this target so downstream interface packages can depend on it.
+  # Not with ament_export_targets(), which would add it to <pkg>_TARGETS.
+  install(
+    EXPORT export_${_target_name_lib}
+    DESTINATION share/${PROJECT_NAME}/cmake
+    NAMESPACE "${PROJECT_NAME}::"
+    FILE "export_${_target_name_lib}Export.cmake")
+  set(_py_extras_file
+    "${CMAKE_CURRENT_BINARY_DIR}/rosidl_generator_py/${_target_name_lib}-extras.cmake")
+  file(WRITE "${_py_extras_file}"
+    "include(\"\${${PROJECT_NAME}_DIR}/export_${_target_name_lib}Export.cmake\")\n"
+    "list(APPEND ${PROJECT_NAME}_TARGETS${rosidl_generator_py_suffix}\n"
+    "  \"${PROJECT_NAME}::${_target_name_lib}\")\n")
+  list(APPEND ${PROJECT_NAME}_CONFIG_EXTRAS "${_py_extras_file}")
 endif()
 
 if(BUILD_TESTING AND rosidl_generate_interfaces_ADD_LINTER_TESTS)

I tested this smaller version as well, and it appears to work.

The main difference is:

• In the larger fix currently proposed in this PR, the generated Python conversion code is treated as part of the Python extension and uses Python3::Module everywhere.
• In the smaller fix, the existing shared-library design is kept, but the Python bindings target is separated from the normal _TARGETS list so C/C++ consumers do not link it accidentally. It uses Python3::Module on macOS and keeps Python3::Python on other platforms.

Since I do not have much experience with Empy and the code-generation side of rosidl_generator_py, I would appreciate it if you could take a look before I spend more time validating them. In particular, I would be interested to know which of these two directions you would prefer

[traversaro]

Thanks a lot for the clear tests and PR description, finally I fully understand the problem behind ros2/rosidl_python#253 . It is a bit late now here in Europe, I will post my thought on this tomorrow. Interestingly, I think the problem is quite similar to PixarAnimationStudios/OpenUSD#3577 .