Skip to content

Fix reflection serialization for D3D11_SHADER_TYPE_DESC#1

Open
killerdevildog wants to merge 4 commits intosamhocevar:masterfrom
killerdevildog:master
Open

Fix reflection serialization for D3D11_SHADER_TYPE_DESC#1
killerdevildog wants to merge 4 commits intosamhocevar:masterfrom
killerdevildog:master

Conversation

@killerdevildog
Copy link

Fix reflection serialization for D3D11_SHADER_TYPE_DESC

Summary

This PR fixes a critical bug in the shader reflection serialization that prevented bgfx (and potentially other users of d3d4linux) from extracting uniform/constant buffer information from compiled HLSL shaders.

The Problem

When bgfx compiles shaders using d3d4linux, it needs to extract reflection data to determine uniform names, types, and sizes. The reflection API provides this through:

ID3D11ShaderReflectionVariable* var = cbuffer->GetVariableByIndex(i);
ID3D11ShaderReflectionType* type = var->GetType();
D3D11_SHADER_TYPE_DESC type_desc;
type->GetDesc(&type_desc);

However, d3d4linux was not serializing the D3D11_SHADER_TYPE_DESC data across the Wine IPC boundary. This meant that while variables were being enumerated, their type information was garbage, causing:

  • Uniforms to have incorrect types (e.g., mat4 appearing as unknown)
  • Uniform sizes to be wrong or zero
  • Shaders to fail at runtime due to incorrect constant buffer layouts

The Fix

Wine side (d3d4linux.cpp)

Added serialization of the type descriptor after each variable:

// After writing variable descriptor...
ID3D11ShaderReflectionType *type = variable->GetType();
D3D11_SHADER_TYPE_DESC type_desc;
type->GetDesc(&type_desc);
p.write_raw(&type_desc, sizeof(type_desc));
p.write_string(type_desc.Name ? type_desc.Name : "");

Client side (d3d4linux_impl.h)

Added deserialization to populate the m_type member:

// After reading variable descriptor...
p.read_raw(&var.m_type.m_desc, sizeof(var.m_type.m_desc));
var.m_type.m_name = p.read_string();

Min Precision Support (d3d4linux.h)

Added the D3D_MIN_PRECISION enum which was missing from the header. This enum is required by the D3D11_SHADER_TYPE_DESC structure to specify minimum precision requirements for shader variables (e.g., D3D_MIN_PRECISION_DEFAULT, D3D_MIN_PRECISION_FLOAT_16, etc.):

enum D3D_MIN_PRECISION
{
    D3D_MIN_PRECISION_DEFAULT   = 0,
    D3D_MIN_PRECISION_FLOAT_16  = 1,
    D3D_MIN_PRECISION_FLOAT_2_8 = 2,
    D3D_MIN_PRECISION_RESERVED  = 3,
    D3D_MIN_PRECISION_SINT_16   = 4,
    D3D_MIN_PRECISION_UINT_16   = 5,
    D3D_MIN_PRECISION_ANY_16    = 0xf0,
    D3D_MIN_PRECISION_ANY_10    = 0xf1,
};

Without this enum, the MinPrecision field in D3D11_SHADER_TYPE_DESC would cause compilation errors or undefined behavior.

Testing

Tested with bgfx shader compiler (shaderc) on Linux:

  1. Build: Compiled shaderc with d3d4linux integration
  2. Compile: Built cubes example shaders for dx11 profile
    shaderc -f vs_cubes.sc -o vs_cubes.bin --type vertex --platform windows -p vs_5_0
    shaderc -f fs_cubes.sc -o fs_cubes.bin --type fragment --platform windows -p ps_5_0
    
  3. Verify: Confirmed u_modelViewProj uniform extracted correctly:
    • Type: D3D_SVT_FLOAT (matrix)
    • Rows: 4, Columns: 4
    • Size: 64 bytes
  4. Runtime: Ran cubes example in Wine+DXVK - renders correctly with proper transforms

Before/After

Before: Uniforms had garbage type data, causing runtime failures

Uniform: u_modelViewProj
  Type: 0 (unknown)
  Size: 0

After: Uniforms correctly identified

Uniform: u_modelViewProj  
  Type: D3D_SVT_FLOAT (matrix 4x4)
  Size: 64

Compatibility

This change maintains backward compatibility with the IPC protocol - it only adds data that was previously missing. The ID3D11ShaderReflectionType stub class already had m_desc and m_name members; they just weren't being populated.

Related to bkaradzic/bgfx#3578

This fixes a critical bug where the type descriptor for shader variables
was not being serialized/deserialized. bgfx's shaderc uses this type info
to determine uniform types (via findUniformType) and array element counts.

Without this fix, shaders compile but uniforms are not properly extracted,
causing rendering issues at runtime.

Changes:
- d3d4linux.cpp: Serialize D3D11_SHADER_TYPE_DESC after each variable
- d3d4linux_impl.h: Deserialize type descriptor into var.m_type

Also includes Wine 11+ fallback support in d3d4linux_impl.h
- Add ID3D11ShaderReflectionType struct and GetType() method
- Add m_type member to ID3D11ShaderReflectionVariable
- Fix struct alignment with explicit padding for D3D11_SIGNATURE_PARAMETER_DESC
- Add Wine 11+ support notes

These changes enable proper shader reflection for bgfx's shaderc.
killerdevildog added a commit to killerdevildog/bgfx that referenced this pull request Feb 4, 2026
Include d3d4linux.cpp and Makefile with fixes for D3D11_SHADER_TYPE_DESC
serialization. This allows building shaderc with proper uniform extraction
while upstream PR is pending: samhocevar/d3d4linux#1
This forces all uniform variables to be marked as 'used' during reflection
deserialization, ensuring bgfx extracts type info for all uniforms.

Note: This deviates from native D3D compiler behavior where unused uniforms
do not have the D3D_SVF_USED flag set. Native behavior would skip extraction
for uniforms not actually referenced in shader code.
Only write comment string when szComments is non-null to match
server-side read behavior. Fixes 'bad message received' error.
bkaradzic added a commit to bkaradzic/bgfx that referenced this pull request Feb 6, 2026
* Add d3d4linux to 3rdparty for Linux HLSL shader compilation

- Added d3d4linux wrapper library for D3DCompiler access on Linux
- Includes Microsoft D3DCompiler DLLs (versions 43 and 47)
- Source from https://github.com/samhocevar/d3d4linux
- Commit: c9b1ca9 (Handle D3DReflect() regardless of DLL version)
- Enables HLSL shader cross-compilation on Linux via Wine

This is the first step toward implementing issue #1869:
Enable DirectX shader compilation on Linux platforms without
requiring Windows machines in the build pipeline.

Components included:
- d3d4linux.cpp - Wine-based D3DCompiler wrapper
- d3dcompiler_43.dll & d3dcompiler_47.dll - MS compiler DLLs
- Header files for D3D API compatibility
- Makefile for building the wrapper library
- Test files for validation

* Fix d3d4linux struct alignment for D3DReflect compatibility

- Add D3D_MIN_PRECISION enum to d3d4linux_enums.h
- Add MinPrecision field and padding to D3D11_SIGNATURE_PARAMETER_DESC
- Fixes struct size mismatch (32 bytes -> 40 bytes) with d3dcompiler_47.dll
- All 4 D3D functions now work: D3DCompile, D3DReflect, D3DStripShader, D3DDisassemble
- Add implementation_log.md documenting the integration process

Part of issue #1869: Enable HLSL shader compilation on Linux via Wine

* Integrate d3d4linux with bgfx shaderc for Linux HLSL compilation

Add support for HLSL shader compilation on Linux via d3d4linux (Wine-based).
This enables building D3D11/D3D12 shaders on Linux without Windows.

Changes:
- shaderc.h: Add SHADERC_CONFIG_HLSL_D3D4LINUX option to enable HLSL on Linux
- shaderc_hlsl.cpp: Add d3d4linux code path with conditional compilation
  - Uses d3d4linux inline functions instead of Windows DLL loading
  - load() returns d3d4linux compiler info on Linux
  - unload() is no-op on d3d4linux (no DLL to close)
- shaderc.lua: Add d3d4linux include path for Linux/macOS builds

Usage:
  Build with -DSHADERC_CONFIG_HLSL_D3D4LINUX=1
  Set D3D4LINUX_WINE, D3D4LINUX_EXE, D3D4LINUX_DLL environment variables

Requires: Wine, d3d4linux.exe, d3dcompiler_47.dll

Part of issue #1869: Enable HLSL shader compilation on Linux

* d3d4linux: Complete shaderc HLSL integration with Wine

Full working HLSL shader compilation on Linux via d3dcompiler_47.dll and Wine.

d3d4linux fixes:
- Add D3D_MIN_PRECISION enum and MinPrecision field to D3D11_SIGNATURE_PARAMETER_DESC
  (fixes struct size mismatch: 32 bytes -> 40 bytes to match Windows ABI)
- Add missing D3DCOMPILE_* flags used by shaderc
- Add ID3D11ShaderReflectionType struct with GetType() support
- Wine 11+ compatibility: default path changed from wine64 to wine with fallback
- Fix D3DDisassemble IPC protocol bug (only write comment string when present)

shaderc integration:
- Add SHADERC_CONFIG_HLSL_D3D4LINUX=1 define for Linux/macOS builds
- Restructure shaderc_hlsl.cpp with conditional compilation for d3d4linux vs Windows
- Fix null pointer crash when D3DCompile returns error without message
- Initialize ID3DBlob pointers to NULL

Usage:
  D3D4LINUX_EXE=/path/to/d3d4linux.exe shaderc -f shader.sc -o out.bin \
    --type vertex --platform windows -p s_5_0

Tested with vs_cubes.sc, fs_cubes.sc, vs_bump.sc - all compile successfully.
D3DCompile, D3DReflect, D3DStripShader, D3DDisassemble all working.

Implements: #1869

* docs: Add d3d4linux documentation

- docs/d3d4linux-shaderc-support.md: User guide explaining prerequisites,
  installation, usage, environment variables, and troubleshooting for
  compiling HLSL shaders on Linux/macOS via Wine

- implementation_log.md: Updated with shaderc integration completion details,
  including all code changes made, test results, and commit references

* docs: Add EditorConfig/eclint notes to implementation log

Documents that eclint reports some style issues in modified files, but these
are pre-existing in the original codebase (copyright header comments using
spaces, line length in shaderc files). Our changes follow the existing code
style and do not introduce new violations.

* Fix d3d4linux integration after rebase onto upstream master

This commit fixes build issues that arose after rebasing onto the updated
upstream master which includes the TINT library and directx-headers.

Changes in this commit:

## shaderc.h
- Disable DXIL (DXC compiler) when d3d4linux is enabled
  - DXIL uses DirectX Compiler (DXC) which requires directx-headers
  - d3d4linux uses D3DCompiler via Wine which is a different compilation path
  - These two approaches are mutually exclusive on Linux

## shaderc.lua
- Removed directx-headers include paths for Linux builds
  - d3d4linux provides its own D3D type definitions
  - directx-headers conflicts with d3d4linux headers (duplicate typedefs)

## shaderc_hlsl.cpp
- Made all D3D-related includes conditional on SHADERC_CONFIG_HLSL_D3D4LINUX
  - Use <d3d4linux.h> when d3d4linux is enabled
  - Use <d3dcompiler.h> and <d3d11shader.h> on Windows
  - PFN_D3D_* function pointer typedefs (use WINAPI calling convention)
  - D3DCompiler struct with IID_ID3D11ShaderReflection member
  - s_d3dcompiler[] array of DLL versions
  - s_d3dcompilerdll handle
- Added simplified D3DCompiler struct for d3d4linux path (fileName only)
- Fixed D3DReflect call to use d3d4linux's integer IID macro

## Testing
- Successfully compiled shaderc with SHADERC_CONFIG_HLSL_D3D4LINUX=1
- Successfully compiled vs_cubes.sc to D3D11 bytecode (DXBC format)
- Output verified with xxd showing VSH header and DXBC signature

* Cleanup d3d4linux integration per maintainer feedback

Simplified changes:
- Remove SHADERC_CONFIG_HLSL_D3D4LINUX define, use BX_PLATFORM_* directly
- SHADERC_CONFIG_HLSL now enabled on Windows/Linux/macOS
- SHADERC_CONFIG_DXIL now Windows-only (DXC needs native D3D)

Removed files:
- d3d4linux test files (compile-hlsl.cpp, ps_sample.hlsl)
- d3d4linux extras (UE4 patches)
- d3d4linux.Build.cs (UE4 build file)
- d3dcompiler_43.dll (only 47 needed)
- implementation_log.md
- docs/d3d4linux-shaderc-support.md

Build and test verified on Linux with Wine 11.0.

* Enable DXIL on Linux/macOS and configure d3d4linux paths

Changes:
- shaderc.h: Enable SHADERC_CONFIG_DXIL on Linux and macOS (previously Windows-only)
  This allows Shader Model 6.0+ compilation via DXC on non-Windows platforms

- shaderc.lua: Configure d3d4linux and directx-headers for Linux/macOS
  - Add directx-headers include paths for DXIL support
  - Add WSL stubs include path for COM compatibility types
  - Define D3D4LINUX_EXE path pointing to 3rdparty/d3d4linux/d3d4linux.exe
  - Define D3D4LINUX_DLL path for d3dcompiler_47.dll (Wine Z: drive prefix)

Prerequisites for HLSL compilation on Linux:
1. Wine installed (/usr/bin/wine or /usr/bin/wine64)
2. Build d3d4linux.exe: cd 3rdparty/d3d4linux && make
   (requires mingw-w64 cross-compiler: x86_64-w64-mingw32-c++)
3. d3dcompiler_47.dll in 3rdparty/d3d4linux/

Tested: SM 5.0 vertex shader compilation works via Wine/d3d4linux
DXIL (SM 6.0+) requires native DXC library installation

* Remove macOS from DXIL config (no DXC library available)

Microsoft's DXC releases only include:
- Windows: dxcompiler.dll
- Linux: libdxcompiler.so

There is no libdxcompiler.dylib for macOS. Updated SHADERC_CONFIG_DXIL
to only enable DXIL on Windows and Linux.

macOS still supports legacy HLSL (SM 5.0) via d3d4linux.

* Remove d3d4linux build files from bgfx repo

Users should obtain d3d4linux.exe and d3dcompiler_47.dll from:
https://github.com/killerdevildog/d3d4linux

Only headers are needed for bgfx compilation.

* Fix d3d4linux reflection: deserialize D3D11_SHADER_TYPE_DESC

Fixes uniform extraction bug where type info was not being read from
the IPC stream. This is required for bgfx to properly determine uniform
types and array element counts via findUniformType().

* Add d3d4linux source with reflection serialization fix

Include d3d4linux.cpp and Makefile with fixes for D3D11_SHADER_TYPE_DESC
serialization. This allows building shaderc with proper uniform extraction
while upstream PR is pending: samhocevar/d3d4linux#1

* Apply maintainer cleanup: unified code paths and header guards

* Force D3D_SVF_USED flag on all uniform variables

This forces all uniform variables to be marked as 'used' during reflection
deserialization, ensuring bgfx extracts type info for all uniforms.

Note: This deviates from native D3D compiler behavior where unused uniforms
do not have the D3D_SVF_USED flag set. Native behavior would skip extraction
for uniforms not actually referenced in shader code.

* Fix D3DDisassemble serialization mismatch

Only write comment string when szComments is non-null to match
server-side read behavior. Fixes 'bad message received' error.

* shaderc: Improve d3d4linux integration and error handling

This commit enhances the d3d4linux integration for HLSL shader compilation
on Linux and macOS platforms with several key improvements:

Key Changes:
- Dynamic path detection for d3d4linux binaries instead of hardcoded paths
- Relocate d3dcompiler_47.dll from 3rdparty/d3d4linux/ to tools/bin/windows/
- Remove d3d4linux.exe from 3rdparty (now managed separately)
- Improved error handling with proper null checks and user feedback
- Enhanced logging using BX_TRACE for consistency
- Remove --verbose flag from shader compilation make rules for cleaner output
- Automatic environment variable setup (D3D4LINUX_EXE, D3D4LINUX_DLL)

Technical Details:
- shaderc_hlsl.cpp now dynamically locates d3d4linux.exe and d3dcompiler_47.dll
  relative to the executable directory
- Proper error messages when required files are not found
- Simplified Makefile in 3rdparty/d3d4linux/
- Removed hardcoded D3D4LINUX_EXE and D3D4LINUX_DLL defines from shaderc.lua

Files Modified:
- 3rdparty/d3d4linux/Makefile: Simplified build rules
- scripts/shader.mk: Removed --verbose flags from compilation commands
- scripts/shaderc.lua: Removed hardcoded path defines
- tools/shaderc/shaderc_hlsl.cpp: Enhanced d3d4linux integration logic

Files Deleted:
- 3rdparty/d3d4linux/d3d4linux.exe
- 3rdparty/d3d4linux/d3dcompiler_47.dll

This improves portability and makes the build system more maintainable
by removing hardcoded paths and improving runtime binary discovery.

Credit: bkaradzic provided patch

* Delete d3d4linux.md

* Delete examples/runtime/shaders/dx11/vs_cubes.bin

* Remove macOS.

---------

Co-authored-by: Branimir Karadžić <branimirkaradzic@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant