Linux port pr

[raydelto]

Description

Nabla does not currently build or run on Linux. Most of the CMake plumbing and the POSIX system layer exist, but in practice the DXC sub-build fails, IApplicationFramework::createSystem() returns nullptr, the HLSL compiler is compiled out, the shared library does not link on ELF, and there is no window manager or Vulkan surface. This PR makes the engine, DXC, nsc, the headless examples and a first windowed example build and run on Ubuntu 24.04 with Clang 20.

Toolchain: Clang 20.1.2 + LLD, Ninja, CMake ≥ 3.31 (tested with 4.3.5), NBL_PCH=ON, NBL_NSC_MODE=SOURCE. Clang 18.1.3 crashes in the front end (Sema::tryCaptureVariable) on CAssetConverter.cpp, so Clang 20 is the minimum tested version. GCC is not supported yet: there is no CXX_GNU flags profile.

Windows/MSVC: Linux-specific additions are behind _NBL_PLATFORM_LINUX_, $<PLATFORM_ID:Linux>, if(NOT MSVC) or if(UNIX AND NOT APPLE AND NOT ANDROID). Several shared changes apply to every platform, Windows included: the conformance, API and initialization fixes in core headers, the libjpeg-turbo target, the absolute submodule URLs, NBL_DXC_PARALLEL_LINK_JOBS for Ninja builds, and CSmoothResizeSurface::handleOutOfDate. Each is called out below. Nothing in this PR has been built with MSVC yet (see TODO).

Build system and DXC sub-build

• 3rdparty/dxc: the separate DXC configure now receives the parent's C/C++ compiler, CMAKE_LINKER_TYPE, ccache launcher and LLVM_PARALLEL_LINK_JOBS. The compiler, linker and ccache forwarding is non-MSVC only. LLVM_PARALLEL_LINK_JOBS comes from a new cache var, NBL_DXC_PARALLEL_LINK_JOBS (default 1, because LLVM links are the build's memory peak). It is forwarded for any Ninja generator, so it also applies to Ninja builds on Windows.
• Two workarounds for the DXC fork, non-Windows only, each marked TODO: fix in the DXC fork:
  • lib/DxcSupport/dxcmem.cpp calls the Win32 CoGetMalloc instead of DxcCoGetMalloc. A generated CMAKE_PROJECT_INCLUDE adds CoGetMalloc=DxcCoGetMalloc as a compile definition.
  • The fork bumped external/DirectX-Headers past upstream DXC's pin. The newer wsl/stubs/basetsd.h redefines types from WinAdapter.h. DXC is built against DirectX-Headers 980971e (upstream DXC's pin, NBL_DXC_DIRECTX_HEADERS_REF), exported with git archive into the build tree.
• dxcompiler custom target: the import library and the shared object are the same file on ELF, so duplicate BYPRODUCTS are removed.
• Shared builds set CMAKE_POSITION_INDEPENDENT_CODE ON (the static 3rdparty archives are linked into libNabla.so; this is a no-op on PE/COFF). It is also forwarded to the libjpeg-turbo child build.
• libjpeg-turbo: correct archive name (libjpeg.a for non-MSVC) and single-config output directory. The custom command is now an always-run custom target with BYPRODUCTS, so the child build's own dependency tracking picks up source and flag changes. This one changes Windows behaviour too.
• volk is compiled with -fcommon (non-MSVC): vulkaninfo and volk define the same vkXxx globals, and GCC ≥ 10 and Clang ≥ 11 default to -fno-common.
• libstdc++ runs the parallel std::execution policies on oneTBB when <tbb/tbb.h> exists. A configure check (_PSTL_PAR_BACKEND_TBB) links TBB::tbb when needed, or falls back to _GLIBCXX_USE_TBB_PAR_BACKEND=0 with a warning when the headers are present without TBBConfig.cmake.
• Executables that reuse a -fPIC PCH from a static library (the examples API) are built -fPIC instead of -fPIE; Clang rejects mixing the two in a PCH.
• Volk and Nabla define VK_USE_PLATFORM_XCB_KHR on Linux, and Nabla links X11::xcb (configure fails with a clear message without libxcb1-dev).

nsc and install

• $<TARGET_PDB_FILE> / $<TARGET_PDB_FILE_NAME> install rules and the nsc PDB test are guarded with if(MSVC); those generator expressions are errors on ELF linkers.
• ELF has no delay loading. Installed executables get $ORIGIN-relative INSTALL_RPATH entries to the Nabla and DXC runtime directories, and the installed libNabla.so gets $ORIGIN/<rel>/3rdparty/dxc.
• ELF shared objects are installed with install(TARGETS … LIBRARY) instead of install(PROGRAMS), so the build RPATH (absolute build-tree paths) is replaced by INSTALL_RPATH.
• nsc must be built with NBL_NSC_MODE=SOURCE on Linux: the PACKAGE mode only has a nsc-windows-x64-release channel.

POSIX core

• IApplicationFramework::createSystem() returns CSystemLinux on Linux. Runtime module lookup uses dlopen(RTLD_NOLOAD) and falls back to the search paths.
• CHLSLCompiler / CCompilerSet are enabled on Linux. DXC's WinAdapter.h provides the COM types, plus a small ComPtr shim over CComPtr.
• ISystemPOSIX::createFile used creat(name, 0444), which truncates and leaves the file read-only, so a second run could not reopen its own output. It now uses open(name, flags, 0644) without truncation, matching Win32 OPEN_ALWAYS.
• ISystem::findFileInArchive: canonical("") throws in libstdc++ for a relative file name with no parent. The path is made absolute first.
• Include finder: header-root classification strips a leading /, and that stripped string was also used as the loader search path, so absolute include dirs and #include "/abs/path.hlsl" became relative. The loader path now keeps the separator. The absolute-path existence check uses the non-throwing std::filesystem::exists overload.
• ISystem::isDebuggerAttached: handles read() errors and an off-by-one. The duplicate, non-compiling isDebuggerAttached in CSystemLinux.cpp is removed.
• IAPIConnection::loadRenderdoc on Linux and Android declared a local RENDERDOC_GetAPI that shadowed the outer one, so RenderDoc was never hooked.

Clang conformance and shared fixes (all platforms)

MSVC accepts code in templates that are never instantiated, and some of it does not compile with Clang. Most of the following are syntax-level changes, but a few change public API, initialization or logic. Those are listed separately below.

• missing template/typename keywords, access-specifier mismatches between declarations (IGPUCommandPool), stale code from the BLAS tracking refactor (pushTrackedBLASes calls in ILogicalDevice.h), incomplete-type uses moved out of line (ILogicalDevice::supportsAccelerationStructureVertexFormat), missing includes, ExternalHandleNull instead of nullptr for external_handle_t;
• default member initializers of nested aggregates used while the enclosing class is incomplete (CHashCache::key_t, SConvReqAccelerationStructure, addSearchPath, …). Clang silently marks the dependent declarations invalid and never emits them; they are replaced by constructors or delegating overloads;
• GLM only enables swizzle members when it detects MS extensions. cpp_compat/vector.hlsl defines _MSC_EXTENSIONS around GLM's one-time setup on GCC/Clang only, and errors out if glm/glm.hpp was included first;
• promoted_mul in fast_affine.hlsl: the matrix-matrix overload now deduces on Clang;
• the include IMath/halfLimits.h is now Imath/halfLimits.h, because Linux file systems are case-sensitive.

Shared API, initialization and logic changes:

• IGeometry view encodeElement (public): the return type changes from void to bool. It already did return false; on unformatted views. The proxy overload now forwards the result and returns false when it has no target.
• CAssetConverter::SConvReqAccelerationStructure: the new constructor also zero-initializes compact and buildSize, which had no initializer before.
• IGPUTopLevelAccelerationStructure: instance conversion std::gets the ICPUTopLevelAccelerationStructure instance types, which is the type the variant actually holds. BLAS tracking goes through a new DynamicUpCastingSpanIterator, and the pushTrackedBLASes callers in ILogicalDevice.h pass a begin iterator and a count.
• CTrueIR::emplace: a parameter pack named args shadowed the args member that the body dereferences (args.dst). It is renamed to funcArgs.

XCB windowing and the Vulkan XCB surface

• New IWindowManagerXcb / IWindowXcb (public) and CWindowManagerXcb / CWindowXcb, using only libxcb (the few ICCCM/EWMH/Motif properties are written by hand). Supported: create, show/hide, move/resize, minimize/maximize/restore, title, close (WM_DELETE_WINDOW → onWindowClosed, which unmaps the window instead of destroying it, because a VkSurface may still use it).
• Threading and lifetime, also documented in IWindowManagerXcb.h: one xcb_connection_t and one event thread per manager (the same IAsyncQueueDispatcher pattern as Win32). All IEventCallbacks run on that thread. Callbacks may create windows and drop other windows; dropping the last reference to the window whose callback is running is forbidden and asserted in debug.
• CSurfaceVulkanXcb (+ Native), and VK_KHR_xcb_surface is requested on the instance. The native handle is a {connection, window} pair; CSurface::getNativeWindowHandle() returns its address for non-pointer handles (no change for HWND/ANativeWindow).
• CSmoothResizeSurface::handleOutOfDate() said it recreated the swapchain but only retried the acquire, which then failed on the invalidated resources and made the surface irrecoverable. It now recreates and then acquires, like CSimpleResizeSurface. On X11 the driver can return VK_ERROR_OUT_OF_DATE_KHR before the event thread delivers ConfigureNotify; the same race could in principle happen on Win32.
• Not implemented yet: keyboard/mouse input channels, cursor control, clipboard, RandR (getPrimaryDisplayInfo returns the whole X screen), native Wayland. Examples that wait for a mouse (MonoWindowApplication, e.g. 09, 12, 73) do not start on Linux yet.
• New CTest NBL_UI_XCB_WINDOW_MANAGER (src/nbl/ui/test, no GPU, exit 77 = skipped without DISPLAY).

Build hygiene and docs

• docs/linux-build.md (prerequisites, compiler selection, configure flags, build order with memory limits, running the tests) and a pointer in docs/build/LINUX.md.

Testing

Machine: Ubuntu 24.04.5, kernel 6.8, X11 session (GNOME/mutter). Clang 20.1.2 + LLD 20, CMake 4.3.5, Ninja 1.11.1, Release, shared build, NBL_PCH=ON, NBL_NSC_MODE=SOURCE. LunarG Vulkan SDK 1.4.363.0 (loader and Khronos validation layer from the SDK), with the GPU selected through VK_DRIVER_FILES.

GPU	Driver	Vulkan API
NVIDIA GeForce RTX 3050 Laptop	580.178.04	1.4.312
Intel UHD Graphics (TGL GT1)	Mesa 25.2.8 (ANV)	1.4.318

The results below come from the recorded runs of the merged PRs and their independent reviews. They were not all re-run together at the final head e82edb2ff.

Build: libdxcompiler.so, libNabla.so and nsc build from an empty build directory, and examples 01, 02, 08, 21 and 23 build on top. ldd -r libNabla.so reports no undefined symbols. The RUNPATH of the packaged runtime/nbl/libNabla.so is exactly $ORIGIN/3rdparty/dxc. Embedded builtin resources were disabled in this configuration (NBL_EMBED_BUILTIN_RESOURCES=OFF, the default), so nsc ran with -no-nbl-builtins.

nsc: CTest 6/6 pass, including the installed nsc compiling from a different working directory. The output passes spirv-val --target-env vulkan1.3.

Examples (validation layer on; Release logs warnings and errors only):

01 and 21 are CPU/VFS tests. They passed with each vendor's ICD selected, but that doesn't show anything ran on the GPU.

Example	RTX 3050	Intel UHD
01_HelloCoreSystemAsset	pass (exit 0)	pass (exit 0)
02_HelloCompute	pass	pass
21_LRUCacheUnitTest	pass	pass
23_Arithmetic2UnitTest	pass (~400 s)	crash (exit 139) in Mesa ANV vkCreateComputePipelines, see below
08_HelloSwapchain (X11)	renders; resize, minimize, maximize, close; 0 validation messages	same

Intel and test 23: with native subgroup arithmetic, requiredSubgroupSize = 32, workgroup 64, 4 items per invocation and the workgroup inclusive scan, ANV's shader compiler segfaults. Diagnostic probes passed for the other permutations: the reduction with the same parameters, workgroups 128–1024 with 4 items, subgroup sizes 8 and 16, and the emulated path. Those probes used a local skip of the crashing case, so they are not a successful run of the unmodified test on Intel. The extracted SPIR-V passes spirv-val, and a standalone C reproducer crashes on that pipeline at subgroup size 32 and succeeds at 16 and 8. The crash is the same on Mesa 25.2.8-0ubuntu0.24.04.2 and .4. We treat it as a driver bug; a Mesa report has not been filed yet.

XCB: NBL_UI_XCB_WINDOW_MANAGER passes on the real X11 display, on bare Xvfb and on Xvfb + Openbox, and is skipped without DISPLAY. It covers creating a window from a callback, dropping it from another callback, continued event delivery and teardown from the main thread. 60 resizes 15 ms apart on Xvfb + Openbox (Intel, MESA_VK_WSI_DEBUG=sw) all recreate the swapchain with no validation messages.

Not tested or known not working:

• Windows/MSVC (and clang-cl) builds of this branch. No Windows CI ran on these changes.
• GCC (no flags profile yet).
• NBL_PCH=OFF: known failing. It was attempted, and about 126 translation units fail on missing includes. There is no successful PCH-OFF build.
• NBL_EMBED_BUILTIN_RESOURCES=ON.
• Debug and RelWithDebInfo configurations, and static (NBL_STATIC_BUILD) builds.
• Wayland and XWayland, other X window managers on real hardware, AMD GPUs, other distributions.
• Any example not listed above.

TODO list:

• Windows: build this branch with MSVC (and clang-cl) and run the existing Windows CI. The conformance fixes and the libjpeg-turbo target change affect Windows.
• Merge the companion PR to Devsh-Graphics-Programming/Nabla-Examples-and-Tests, then point examples_tests back at the Devsh URL and pin it to the upstream merge commit.
• Decide whether the DXC workarounds stay here or get fixed in Devsh-Graphics-Programming/DirectXShaderCompiler (dxcmem.cpp CoGetMalloc → DxcCoGetMalloc; DirectX-Headers pin). If they are fixed there, remove NBL_DXC_DIRECTX_HEADERS_REF and the CMAKE_PROJECT_INCLUDE.
• File the Mesa ANV bug for the test 23 crash (SPIR-V and reproducer are ready).
• Optional, could be follow-up PRs instead:
  • Self-contained headers so NBL_PCH=OFF builds.
  • GCC 13 support (CXX_GNU flags profile and conformance pass).
  • XCB keyboard (xkbcommon) and mouse input, cursor, clipboard, RandR; then examples 09, 12 and 73.
  • Linux CI job (build-only on a hosted Ubuntu runner).
  • Linux nsc PACKAGE channel, if PACKAGE mode should work on Linux.
  • The build-tree RUNPATH of libNabla.so still ends with an empty entry.
  • With NBL_ENABLE_CONFIG_INSTALL=ON, the extra lib/ copy of libNabla.so gets a DXC RUNPATH that doesn't resolve from that directory. The packaged runtime/nbl/ copy is correct.

[devshgraphicsprogramming]

Awesome contribution, we have 3 small blockers to integrate this:

1. @AnastaZIuk and @keptsecret are working on porting everything to Conan and getting rid of submodules
2. due to extensive use of Claude, can you clearly label every bit of code/documentation that you haven't read, 100% understand and are willing to present as your own? This lets us know where the dragons are - we're not going to approve the CI pipeline until this audit is complete
3. on our end we need to construct a CI pipeline for this to test everything on Linux