tools/d3d12: refuse non-Microsoft fxc output in the regen script

Post-mortem follow-up to cd971cd / da20f5b. The pre-bake attempt
failed because Wine's bundled d3dcompiler_47.dll (which is the
vkd3d-shader reimplementation, currently at version 1.10) produces
DXBC that's syntactically valid as a container but trips Microsoft
D3D12's CreateGraphicsPipelineState with E_FAIL for non-trivial
shaders. The vram_copy.ps.hlsl case crossed that threshold; the
simpler fullscreen_quad.vs.hlsl + copy.ps.hlsl cases happen to
work and are still in the repo, but they were also vkd3d-shader-
compiled and should be regenerated with Microsoft fxc when the
opportunity arises.

The fix is twofold: improve the SDK auto-detection so contributors
who have Microsoft fxc installed don't have to fight the script,
and add an output validator that refuses to write .inc files
produced by any compiler that isn't Microsoft fxc.

Auto-detection improvements (find_fxc):

* Wine prefix discovery widened. Previously the script only
  checked $WINEPREFIX/drive_c if WINEPREFIX was set. Now it also
  checks ~/.wine (the Wine default when WINEPREFIX is unset),
  and any prefix directory under ~/.local/share/wineprefixes/
  that has a drive_c subtree (PlayOnLinux / lutris-style layouts).
  Both /Program Files/ and /Program Files (x86)/ are searched in
  each prefix.
* Error message now points at the README's installation steps
  rather than telling the user to just install the SDK.
* Error message explicitly calls out that Wine's bundled
  d3dcompiler_47.dll is NOT a substitute and references
  da20f5b for context.

Output validator (extract_rdef_creator + check_microsoft_creator):

* extract_rdef_creator walks the DXBC container chunk index, finds
  the RDEF chunk, follows the creator_offset field at chunk body
  offset 24, and reads back the null-terminated ASCII creator
  string. The DXBC container layout (16-byte hash + version +
  total size + chunk count + chunk offsets) and the RDEF body
  layout (constant_buffer_count, constant_buffer_offset,
  resource_binding_count, resource_binding_offset, shader_version,
  compiler_flags, creator_offset) are documented in MSDN's old
  D3D11 reflection notes and in the wikitalk pages around DXBC;
  the implementation here uses fixed offsets that match what
  fxc.exe and vkd3d-shader both produce.
* check_microsoft_creator hooks into compile_one after the magic
  check. It refuses with a clear error message if the creator
  string doesn't contain "Microsoft". Microsoft fxc stamps
  something like "Microsoft (R) HLSL Shader Compiler 10.1";
  vkd3d-shader stamps "vkd3d-shader 1.10 (Wine bundled)". One
  substring check disambiguates.

What the validator does NOT do:

* It does not retroactively reject existing .inc files in the
  repo. The validator only fires on freshly-produced DXBC during
  a regen run. The vkd3d-shader-built fullscreen_quad_vs.inc and
  copy_ps.inc are unchanged by this commit; they'll get refreshed
  the next time a contributor with Microsoft fxc runs the regen
  script. Until then, they keep working because they're already
  in the repo and the build just compiles them as plain C++
  byte arrays - they never go through PSO validation (they're
  already valid DXBC).
* It does not lock the script to a specific fxc version. Any
  Microsoft fxc.exe (10.1, 10.5, the version that ships with the
  Windows 11 SDK, etc.) is accepted - they all stamp "Microsoft"
  in the creator string.
* It does not validate the binding layout vs the runtime root
  signature. That's a future enhancement - the runtime side
  expects specific (b0, t0, s0) layouts and the regen script
  could cross-check, but that's a much bigger feature.

README changes:

* New "Microsoft fxc.exe is required" section spells out why and
  cites the cd971cd / da20f5b failure as the worked example.
* "Installing Microsoft fxc.exe" subsection with separate
  Windows / Linux instructions. Linux gets two paths: install
  the SDK under a Wine prefix (auto-detected), or extract
  fxc.exe + d3dcompiler_47.dll directly from the MSI with
  msitools and pass --fxc. Both must live in the same dir so
  Wine's loader prefers Microsoft's d3dcompiler_47.dll over its
  bundled vkd3d-shader version.
* The auto-detection scope (WINEPREFIX, ~/.wine, ~/.local/share/
  wineprefixes/*) is now documented so contributors know where
  the script is looking.

This commit does NOT re-add the vram_copy.ps.hlsl source removed
by da20f5b. That comes back in a follow-up patch once a Microsoft
fxc-built .inc is available - same patch lands the .hlsl, the
.inc, and the call-site flip together (the dd54a73 lesson).

Test:
  - python3 -c 'import ast; ast.parse(open("tools/regen_d3d12_dxbc.py").read())'
    passes.
  - Loading the script as a module and feeding the EXISTING
    vkd3d-built copy_ps.inc into extract_rdef_creator returns
    'vkd3d-shader 1.10 (Wine bundled)'; the Microsoft substring
    check returns False. Confirms the validator would catch the
    cd971cd mistake if anyone tries to reproduce it.
  - find_fxc's broader Wine prefix scan is exercised by the
    standard test of running the script in an environment without
    Microsoft fxc; it now fails with the new pointed-at-README
    message instead of the old generic SDK message.
This commit is contained in:
LibretroAdmin 2026-05-18 20:04:30 +00:00 • committed by U-DESKTOP-SPFP6AQ\twistedtechre
commit ebd2300cde
2 changed files with 180 additions and 19 deletions

View file

@ -51,11 +51,53 @@ file and rewrites the matching `.inc` under `src/common/d3d12/embedded_dxbc/`.
Both the `.hlsl` and the regenerated `.inc` must be committed together so the
two stay in sync.
`fxc.exe` ships with the Windows 10 SDK and is **not** invoked by the core
build. On Linux, run the script under Wine - it will look for `fxc.exe` in
the standard SDK install path under `$WINEPREFIX/drive_c/Program Files (x86)/Windows Kits/10/bin/<version>/x64/fxc.exe`,
or you can pass `--fxc PATH` explicitly. The `--wine PATH` flag lets you
point at a specific wine binary if `wine` isn't on `PATH`.
### Microsoft `fxc.exe` is required
The regen script will refuse to write `.inc` files produced by any compiler
other than Microsoft's `fxc.exe`. The check parses the DXBC `RDEF` chunk and
matches the creator string against a Microsoft-prefix allowlist.
The reason is empirical: Wine ships its own `d3dcompiler_47.dll`
implementation (vkd3d-shader), and that compiler's output, while a valid
DXBC container, doesn't pass the Microsoft D3D12 runtime's
`CreateGraphicsPipelineState` validation for non-trivial shaders. The
failure mode is `E_FAIL` (0x80004005) at PSO-creation time with no
useful diagnostic from `Log_ErrorPrintf`. Commit cd971cd shipped a
vkd3d-shader-compiled `vram_copy_ps.inc` and broke the entire D3D12
backend on launch; da20f5b reverts it. The script's creator-string
validator exists to prevent that mistake from recurring.
### Installing Microsoft `fxc.exe`
**Windows**: Install the Windows 10 or 11 SDK from Microsoft. `fxc.exe`
ends up at
`C:\Program Files (x86)\Windows Kits\10\bin\<version>\x64\fxc.exe` (or
the same path under `\Program Files\` if the 64-bit SDK is installed).
The regen script auto-detects it there.
**Linux**: Microsoft's `fxc.exe` is a Windows x86 PE binary and runs
fine under Wine - the trick is just to get the SDK installed inside a
Wine prefix. Two options:
* Run the Windows SDK installer (`winsdksetup.exe` from Microsoft) under
`wine` and select the Windows SDK Signing Tools / SDK component that
ships `fxc.exe`. The default install puts it at
`<prefix>/drive_c/Program Files (x86)/Windows Kits/10/bin/<version>/x64/fxc.exe`,
which the regen script picks up automatically.
* Or extract `fxc.exe` + `d3dcompiler_47.dll` directly from the SDK MSI
using `msiextract` from `msitools`, drop both in any directory, and
pass `--fxc PATH/TO/fxc.exe` to the regen script. Both files must
live in the same directory so Wine's loader finds Microsoft's
`d3dcompiler_47.dll` before falling back to its bundled
vkd3d-shader version.
The script searches for `fxc.exe` in: `$WINEPREFIX` (if set), `~/.wine`,
and any subdirectory under `~/.local/share/wineprefixes/` that has a
`drive_c` tree. Set `WINEPREFIX` to your prefix path if it lives
elsewhere.
The `--wine PATH` flag lets you point at a specific wine binary if
`wine` isn't on `$PATH`.
## Filename convention

View file

@ -60,9 +60,21 @@ TEMPLATE_VARIANTS = {}
def find_fxc(explicit_fxc, explicit_wine):
# On Windows, look for fxc.exe in the Windows 10 SDK install location
# first, then PATH. On Linux, require Wine + explicit --fxc pointing at
# the Windows fxc.exe install.
# On Windows, look for fxc.exe in the Windows 10 / 11 SDK install
# location first, then PATH. On Linux, look in any Wine prefix
# we can find that has the SDK installed; only fall back to the
# --fxc override when the user explicitly points us at one.
#
# Microsoft's actual fxc.exe is required - Wine's bundled
# d3dcompiler_47.dll (which is the vkd3d-shader reimplementation)
# produces DXBC that doesn't pass Microsoft D3D12 PSO validation
# for non-trivial shaders. cd971cd / da20f5b chronicles a
# vram_copy_ps pre-bake attempt that failed with
# CreateGraphicsPipelineState E_FAIL because the .inc was built
# with vkd3d-shader. The check_microsoft_creator() validator
# below rejects vkd3d-compiled output post-hoc, but it's better
# to find Microsoft's fxc.exe up-front and skip the wasted
# compile time entirely. See data/shaders/d3d12/README.md.
if explicit_fxc:
fxc = Path(explicit_fxc)
if not fxc.exists():
@ -77,17 +89,32 @@ def find_fxc(explicit_fxc, explicit_wine):
if path:
return [path]
# Try the common Windows 10 SDK install location even when not on
# Windows (in case the user has the SDK mounted under Wine prefix).
# SDK auto-detection. The Windows 10 / 11 SDK installs fxc.exe at
# <SDK_root>/<version>/x64/fxc.exe. Try the standard install
# locations on Windows AND under every plausible Wine prefix on
# Linux. The /Program Files/ and /Program Files (x86)/ split
# depends on whether the SDK was installed as 32-bit or 64-bit;
# check both. Wine prefix discovery covers $WINEPREFIX (if set),
# the default ~/.wine, and any prefixes under ~/.local/share/
# wineprefixes/ that PlayOnLinux and lutris-style installs use.
sdk_roots = [
Path("C:/Program Files (x86)/Windows Kits/10/bin"),
Path("C:/Program Files/Windows Kits/10/bin"),
]
# Add Wine-prefix style path if we have one.
wine_prefix = os.environ.get("WINEPREFIX")
if wine_prefix:
sdk_roots.append(Path(wine_prefix) / "drive_c" / "Program Files (x86)" / "Windows Kits" / "10" / "bin")
sdk_roots.append(Path(wine_prefix) / "drive_c" / "Program Files" / "Windows Kits" / "10" / "bin")
wine_prefixes = []
if "WINEPREFIX" in os.environ:
wine_prefixes.append(Path(os.environ["WINEPREFIX"]))
default_prefix = Path.home() / ".wine"
if default_prefix.exists() and default_prefix not in wine_prefixes:
wine_prefixes.append(default_prefix)
extra_prefix_dir = Path.home() / ".local" / "share" / "wineprefixes"
if extra_prefix_dir.exists():
for child in extra_prefix_dir.iterdir():
if child.is_dir() and (child / "drive_c").exists():
wine_prefixes.append(child)
for prefix in wine_prefixes:
sdk_roots.append(prefix / "drive_c" / "Program Files (x86)" / "Windows Kits" / "10" / "bin")
sdk_roots.append(prefix / "drive_c" / "Program Files" / "Windows Kits" / "10" / "bin")
for root in sdk_roots:
if not root.exists():
@ -102,13 +129,101 @@ def find_fxc(explicit_fxc, explicit_wine):
[explicit_wine or "wine", str(candidate)])
sys.stderr.write(
"error: fxc.exe not found. Install the Windows 10 SDK (or a newer\n"
" Windows SDK that still ships fxc.exe), or pass --fxc PATH\n"
" pointing at a working fxc.exe binary. On Linux pass --wine\n"
" PATH if `wine` isn't on PATH.\n")
"error: fxc.exe not found. The Microsoft Windows 10/11 SDK ships\n"
" it under <SDK>/bin/<version>/x64/fxc.exe. See\n"
" data/shaders/d3d12/README.md for installation steps,\n"
" including how to set up the SDK under a Wine prefix\n"
" for Linux contributors. If you have fxc.exe elsewhere,\n"
" pass --fxc PATH. On Linux pass --wine PATH if `wine`\n"
" isn't on PATH.\n"
"\n"
" Wine's bundled d3dcompiler_47.dll (vkd3d-shader\n"
" reimplementation) is NOT a substitute - its DXBC output\n"
" fails Microsoft D3D12 PSO validation. See da20f5b for\n"
" a worked example.\n")
sys.exit(1)
def extract_rdef_creator(data):
# Inspect the DXBC RDEF chunk and pull out the compiler-creator
# string. The DXBC container is:
# [0:4] magic "DXBC"
# [4:20] 16-byte content hash
# [20:24] container version
# [24:28] total size
# [28:32] chunk count
# [32:..] uint32 offsets, one per chunk
# Each chunk header is 8 bytes (4-byte magic + 4-byte size) and
# is followed by chunk-type-specific payload. The RDEF chunk
# body lays out as:
# [0:4] constant_buffer_count
# [4:8] constant_buffer_offset
# [8:12] resource_binding_count
# [12:16] resource_binding_offset
# [16:20] shader_version (program_type | major | minor)
# [20:24] compiler_flags
# [24:28] creator_offset (relative to body_start)
# Offsets in the RDEF body are relative to the START of the
# chunk body, NOT the start of the file - that's the same
# convention DXBC uses internally for SGN tables.
chunk_count_bytes = data[28:32]
if len(chunk_count_bytes) != 4:
return None
chunk_count = struct.unpack("<I", chunk_count_bytes)[0]
chunk_offsets_bytes = data[32:32 + 4 * chunk_count]
if len(chunk_offsets_bytes) != 4 * chunk_count:
return None
chunk_offsets = struct.unpack(f"<{chunk_count}I", chunk_offsets_bytes)
for off in chunk_offsets:
if off + 8 > len(data):
continue
magic = data[off:off + 4]
if magic != b"RDEF":
continue
body_start = off + 8
if body_start + 28 > len(data):
return None
creator_offset = struct.unpack("<I", data[body_start + 24:body_start + 28])[0]
creator_addr = body_start + creator_offset
# Creator string is null-terminated ASCII.
end = data.find(b"\0", creator_addr)
if end == -1:
return None
return data[creator_addr:end].decode("ascii", errors="replace")
return None
def check_microsoft_creator(data, hlsl_path):
# Reject anything that isn't from Microsoft's fxc.exe. The
# creator string lives in the RDEF chunk and is what the
# compiler stamps as its identifier:
#
# Microsoft fxc: "Microsoft (R) HLSL Shader Compiler 10.1"
# (or similar - version trails the prefix)
# Wine vkd3d-shader: "vkd3d-shader 1.10 (Wine bundled)"
#
# Microsoft fxc.exe is the only compiler whose DXBC output is
# known to pass D3D12 PSO validation for the shaders this
# backend uses. Other compilers may produce bytecode that's
# technically valid DXBC but trips PSO creation with E_FAIL.
# cd971cd / da20f5b is the worked example.
creator = extract_rdef_creator(data)
if creator is None:
sys.stderr.write(
f"error: could not parse RDEF / creator string from output\n"
f" for {hlsl_path.name}. The DXBC structure may be\n"
f" malformed; refusing to write .inc.\n")
sys.exit(1)
if "Microsoft" not in creator:
sys.stderr.write(
f"error: {hlsl_path.name} compiled with non-Microsoft fxc.\n"
f" Creator string: {creator!r}\n"
f" Microsoft's fxc.exe is required - see\n"
f" data/shaders/d3d12/README.md. da20f5b chronicles\n"
f" the failure mode when other compilers are used.\n")
sys.exit(1)
def sanitize_identifier(stem):
# Map "vram_read.ps" -> "vram_read_ps"; only [A-Za-z0-9_] survive.
name = re.sub(r"[^A-Za-z0-9_]", "_", stem)
@ -152,6 +267,10 @@ def compile_one(fxc_cmd, hlsl_path, target, defines=None):
f"error: {hlsl_path.name} produced output with bad magic "
f"(0x{magic:08x}, expected 0x43425844)\n")
sys.exit(1)
# Reject non-Microsoft compilers. The check parses the RDEF chunk
# for the creator string; Wine's bundled d3dcompiler_47.dll
# (vkd3d-shader) gets refused here.
check_microsoft_creator(data, hlsl_path)
return data