TsundereStation/data/shaders/d3d12
LibretroAdmin 80d0495145 core/d3d12: pre-bake the display pixel shader
The prize set. display_ps is the per-frame hot shader - every
frame goes through this on its way to the host display, so cold-
flip cost matters here more than anywhere else in the GPU
pipeline. This commit pre-bakes 54 variants covering the full
runtime configuration matrix and replaces the shadergen +
m_shader_cache.GetPixelShader path in GetDisplayPipeline with
indexed-lookup into two static arrays of D3D12_SHADER_BYTECODE.

Variant axes (4 axes, 54 unique blobs):

  * DEPTH_24BIT (2 values): 16-bit framebuffer (the common case;
    every game's standard scanout) vs 24-bit (FMV cutscenes,
    games that opt into true-colour framebuffers).

  * interlace_mode (3 values): InterlacedRenderMode enum from
    gpu_hw.h. Mapped to a (INTERLACED, INTERLEAVED) bool pair
    in the variant /D defines:
      i=0 (None):              (INTERLACED=0, INTERLEAVED=0)
      i=1 (InterleavedFields): (INTERLACED=1, INTERLEAVED=1)
      i=2 (SeparateFields):    (INTERLACED=1, INTERLEAVED=0)

  * SMOOTH_CHROMA (2 values): chroma-plane smoothing in the
    24-bit path. Heavy when active - SampleVRAM24Smoothed runs
    9 SampleVRAM24 calls per output pixel (1 centre + 4 x 2x2
    averages), each of which is 2 LoadVRAM calls. Dead code
    when DEPTH_24BIT=0; the variant table omits the c=1 entry
    for d=0 (54 = 18 + 36, not 72 = 2 x 3 x 2 x 6).

  * MULTISAMPLES (6 values: 1, 2, 4, 8, 16, 32): MSAA sample
    count. Same cardinality axis as vram_read_ps (21f4623).
    Composed multiplicatively with SMOOTH_CHROMA: at
    /DSMOOTH_CHROMA=1 /DMULTISAMPLES=32 the inner loop unrolls
    to 9 * 2 * 32 = 576 texture loads per pixel - the d1i?c1m32
    blobs are correspondingly the largest in the set (99 kB
    each).

Variant suffix convention: d{0,1}i{0,1,2}c{0,1}m{01,02,04,08,16,
32}. MULTISAMPLES padded to 2 digits so alphabetical .inc
filename ordering matches the natural nested-loop iteration
order. The d=0 set has 18 entries (one chroma value); d=1 has
36 (both chroma values).

Blob size distribution:
  smallest:  1276 B  d0i0c0m01  (16-bit, no interlace, no MSAA)
  median:   ~3500 B
  largest: 99152 B  d1i2c1m32  (24-bit, SeparateFields, smooth
                                chroma, 32x MSAA - 576 loads/px)
  total:    942 kB of DXBC across 54 .inc files
  on-disk: ~5.6 MB of .inc text (the hex-encoding overhead is
           ~6x vs the binary - "0xXX, " per byte - same shape
           as the vram_read note in 21f4623)

Runtime selection in GetDisplayPipeline:

  if (depth_24) -> k_display_d1[interlace_mode][chroma][ms_idx]
  else          -> k_display_d0[interlace_mode][ms_idx]

Two arrays rather than one ragged 4D, because the smooth_chroma
dimension is structurally absent on the d=0 path (SMOOTH_CHROMA
is dead code there; fxc /O3 would have produced identical DXBC
for c=0 and c=1 anyway, so we save 18 blobs / ~30 kB DXBC by
not baking them). ms_idx is log2(m_multisamples) computed by a
function-scope kMultisamplesIndex lambda - same predicate /
fallback shape as the vram_read pre-bake. The arrays are
function-scope static const so they initialise on first call
and the cost is amortised across the per-frame display PSO
stream for the rest of the session.

Files added:

* data/shaders/d3d12/display.ps.hlsl: standalone HLSL matching
  GenerateDisplayFragmentShader's D3D12 output post-e64fc28
  cbuffer routing. 24-byte cbuffer (u_vram_offset.xy,
  u_crop_left, u_field_offset, u_resolution_scale, u_pad0)
  layout-identical to the GPU_HW_D3D12::UpdateDisplay
  SetGraphicsRoot32BitConstants push. RGBA8ToRGBA5551,
  RGBToYUV, YUVToRGB, fixYCoord helpers all inlined.
  SampleVRAM24, SampleVRAMAverage2x2, SampleVRAM24Smoothed
  match the shadergen body byte-for-byte.

* src/common/d3d12/embedded_dxbc/display_ps_d{0,1}i{0,1,2}c{0,1}
  m{01,02,04,08,16,32}.inc: 54 DXBC blobs (with the
  smooth-chroma collapse on the d=0 path). All built with
  Microsoft fxc; full RDEF/ISGN/OSGN/SHEX/STAT chunk set; all
  54 stamped "Microsoft (R) HLSL Shader Compiler 10.1". Validator
  (check_microsoft_creator in tools/regen_d3d12_dxbc.py)
  passes all 54.

Files modified:

* tools/regen_d3d12_dxbc.py: TEMPLATE_VARIANTS gains the
  display.ps.hlsl entry with 54 rows. Comment explains the
  interlace_mode -> (INTERLACED, INTERLEAVED) mapping and the
  d=0 chroma collapse.

* src/common/d3d12/embedded_shaders.h: 108 extern declarations
  (54 array + 54 size_bytes). Header docblock explains the
  variant axis breakdown, the suffix convention, and the
  two-array runtime selection.

* src/common/d3d12/embedded_shaders.cpp: 54 #include lines for
  the new .inc files, inside namespace D3D12::EmbeddedShaders.

* src/core/gpu_hw_d3d12.cpp GetDisplayPipeline: shadergen call
  + m_shader_cache.GetPixelShader path replaced with the
  kMultisamplesIndex lambda + two static arrays + ternary
  indexed-lookup. The m_shadergen-not-yet-built check is
  dropped for this path since the embedded blobs are
  statically linked. Pipeline build downstream is unchanged
  (same root signature, RT format, blend/depth state).

Cost matrix delta (D3D12) for display_ps:
  Cold-flip multisamples toggle: ~50 ms x 6 D3DCompile +
                                 ~10 ms x 6 PSO
                                 -> ~10 ms x 6 PSO only.
  (Multisamples rebuild touches the full 2x3 display PSO
  matrix at the new sample count.)
  Cold-flip smooth_chroma toggle: ~50 ms x 2 D3DCompile +
                                  ~10 ms x 2 PSO
                                  -> ~10 ms x 2 PSO only.
  (Only the 2 d=1 PSOs are affected; d=0 PSOs are
  chroma-independent.)
  Disk-cache-warm: ~1 ms each -> 0 ms.
  Resolution scale toggle: 0 ms (cbuffer-routed via e64fc28).
  Per-frame PSO selection: previously a shader_cache hash
  lookup; now an array index. ~ns vs ~us delta - negligible.

The shadergen GenerateDisplayFragmentShader function is left
intact in gpu_hw_shadergen.cpp - the other three backends still
call it through their runtime shader-cache paths. Only D3D12
bypasses it now. This is the final shader on the non-batch
pre-bake roadmap.

Test:
  - Build: D3D12 backend links without unresolved symbols
    across all 54 new externs. Specific risk: a typo in any of
    the 108 extern names or 54 includes would surface here.
  - Runtime, default config (MSAA off, no chroma smoothing):
    every game must continue to scanout correctly. Failure
    modes if the d=0 path is wrong:
      * Garbled framebuffer / wrong colours on the per-frame
        display.
      * Interlace fields rendered wrong - test signal: any PSX
        game in interlaced output mode (most TV-target games).
        Wrong interlace mapping would manifest as field
        comb-tearing or half-height output.
  - Runtime, 24-bit colour: trigger any FMV. Standard test
    cases:
      * Sony intro bookend FMV - leaves no doubt the 24-bit
        path is hit.
      * In-game cutscene FMVs (FF7, MGS, Resident Evil).
    Failure mode if d=1 c=0 path wrong: FMV plays but colours
    are wrong (off hue, mismatched chroma).
  - Runtime, 24-bit + chroma smoothing: enable the "24-bit
    Chroma Smoothing" core option, then play FMV. Failure mode
    if d=1 c=1 path wrong: FMV plays with sharp chroma edges
    (the smoothing isn't happening) or with garbled output
    (variant table miswired).
  - MSAA at each level (2x/4x/8x/16x/32x): same screenshot
    tests at each. Failure mode for any individual blob:
    visible MSAA averaging artefacts in the displayed image.
    32x specifically validates that kMultisamplesIndex picks
    ms_idx=5 - wrong-side switch would index m16 (4) instead
    of m32 (5).
  - All combinations: display_ps is selected freshly each
    UpdateDisplay call, so the array indexing is exercised
    every frame. Any mis-indexed combo would be visually
    obvious.
  - Chroma toggle: flip "24-bit Chroma Smoothing" mid-FMV. The
    next display PSO selection picks blobs from k_display_d1
    [interlace_mode][1] instead of [0]. Failure mode: chroma
    smoothing doesn't toggle, or wrong PSO selected leading to
    garbled output.

If everything works, the only remaining pre-bake target is the
batch matrix - the ~1 min cold-flip prize, hundreds of variants.
That's the final arc of this whole sequence.
2026-05-19 00:22:35 +02:00
..
copy.ps.hlsl core/d3d12: pre-bake the copy/blit pixel shader 2026-05-18 20:22:17 +02:00
display.ps.hlsl core/d3d12: pre-bake the display pixel shader 2026-05-19 00:22:35 +02:00
fullscreen_quad.vs.hlsl core/d3d12: pre-bake the fullscreen-quad vertex shader 2026-05-18 19:40:13 +02:00
README.md tools/d3d12: refuse non-Microsoft fxc output in the regen script 2026-05-18 22:16:18 +02:00
vram_copy.ps.hlsl core/d3d12: pre-bake the VRAM copy pixel shader (Microsoft fxc) 2026-05-18 22:57:57 +02:00
vram_fill.ps.hlsl core/d3d12: pre-bake the VRAM fill pixel shader 2026-05-18 23:16:38 +02:00
vram_read.ps.hlsl core/d3d12: pre-bake the VRAM read pixel shader 2026-05-18 23:42:30 +02:00
vram_update_depth.ps.hlsl core/d3d12: pre-bake the VRAM update-depth pixel shader 2026-05-18 23:27:45 +02:00
vram_write.ps.hlsl core/d3d12: pre-bake the VRAM write pixel shader 2026-05-18 23:04:29 +02:00

D3D12 shader sources (pre-baked DXBC)

This directory holds the HLSL sources for shaders used by the D3D12 backend that have been moved off the runtime ShaderGen path. Each .hlsl file here has a matching pre-compiled DXBC blob checked in to src/common/d3d12/embedded_dxbc/.

Why this exists

Historically the D3D12 backend generated HLSL at runtime via src/core/shadergen.cpp and src/core/gpu_hw_shadergen.cpp, then handed the text to D3DCompile (the in-process HLSL -> DXBC compiler from d3dcompiler_47.dll) to produce a DXBC blob for each PSO. On a cold shader cache, walking the 1164-cell batch pipeline matrix in precompile_sync mode takes on the order of one minute, which is the worst entry in the post-dim-cache cost matrix (see the cost-matrix comments in 10c53b8, 57ac62e, and the patches that followed). The runtime cache hides this on warm boots, but cold boots and shader cache invalidations re-pay the full per-cell D3DCompile cost every time the user changes a setting that hits the full-flush branch.

Pre-baking the DXBC removes:

  • per-launch HLSL compilation stall on shader cache misses,
  • a class of "stale cache after d3dcompiler_47.dll version drift" failures,
  • the implicit dependency on D3DCompile being available at runtime for the cells that have been pre-baked.

The OpenGL / OpenGL ES / D3D11 / Vulkan backends still use the runtime ShaderGen path; only the D3D12 backend reads from the embedded DXBC. (The Vulkan equivalent of this infrastructure lives at data/shaders/vulkan/ / src/common/vulkan/embedded_spirv/ / tools/regen_vulkan_spirv.py; this directory is the mirror.)

Build-time constraint

The core build (Makefile.libretro and src/common/CMakeLists.txt) must never invoke fxc.exe or any other external shader compiler. The .inc files under src/common/d3d12/embedded_dxbc/ are checked into the repository and compiled like any other C++ source. The tools/regen_d3d12_dxbc.py helper exists for contributors who edit the .hlsl sources in this directory; it is not part of the build.

Regenerating the DXBC blobs

After editing a .hlsl file here, run:

python3 tools/regen_d3d12_dxbc.py

This invokes fxc.exe (Shader Model 5.0, /O3, /E main) on each .hlsl file and rewrites the matching .inc under src/common/d3d12/embedded_dxbc/. Both the .hlsl and the regenerated .inc must be committed together so the two stay in sync.

Microsoft fxc.exe is required

The regen script will refuse to write .inc files produced by any compiler other than Microsoft's fxc.exe. The check parses the DXBC RDEF chunk and matches the creator string against a Microsoft-prefix allowlist.

The reason is empirical: Wine ships its own d3dcompiler_47.dll implementation (vkd3d-shader), and that compiler's output, while a valid DXBC container, doesn't pass the Microsoft D3D12 runtime's CreateGraphicsPipelineState validation for non-trivial shaders. The failure mode is E_FAIL (0x80004005) at PSO-creation time with no useful diagnostic from Log_ErrorPrintf. Commit cd971cd shipped a vkd3d-shader-compiled vram_copy_ps.inc and broke the entire D3D12 backend on launch; da20f5b reverts it. The script's creator-string validator exists to prevent that mistake from recurring.

Installing Microsoft fxc.exe

Windows: Install the Windows 10 or 11 SDK from Microsoft. fxc.exe ends up at C:\Program Files (x86)\Windows Kits\10\bin\<version>\x64\fxc.exe (or the same path under \Program Files\ if the 64-bit SDK is installed). The regen script auto-detects it there.

Linux: Microsoft's fxc.exe is a Windows x86 PE binary and runs fine under Wine - the trick is just to get the SDK installed inside a Wine prefix. Two options:

  • Run the Windows SDK installer (winsdksetup.exe from Microsoft) under wine and select the Windows SDK Signing Tools / SDK component that ships fxc.exe. The default install puts it at <prefix>/drive_c/Program Files (x86)/Windows Kits/10/bin/<version>/x64/fxc.exe, which the regen script picks up automatically.
  • Or extract fxc.exe + d3dcompiler_47.dll directly from the SDK MSI using msiextract from msitools, drop both in any directory, and pass --fxc PATH/TO/fxc.exe to the regen script. Both files must live in the same directory so Wine's loader finds Microsoft's d3dcompiler_47.dll before falling back to its bundled vkd3d-shader version.

The script searches for fxc.exe in: $WINEPREFIX (if set), ~/.wine, and any subdirectory under ~/.local/share/wineprefixes/ that has a drive_c tree. Set WINEPREFIX to your prefix path if it lives elsewhere.

The --wine PATH flag lets you point at a specific wine binary if wine isn't on $PATH.

Filename convention

Stage is encoded in the suffix:

  • .vs.hlsl -> vertex shader (compiled as vs_5_0)
  • .ps.hlsl -> pixel shader (compiled as ps_5_0)
  • .cs.hlsl -> compute shader (compiled as cs_5_0)
  • .gs.hlsl -> geometry shader (compiled as gs_5_0)

The _5_0 shader model targets D3D_FEATURE_LEVEL_11_0, matching the runtime baseline of the D3D12 backend (see the shared compile path in src/common/d3d11/shader_compiler.cpp which both D3D11 and D3D12 used prior to the pre-bake migration).

Template variants

Shaders that need multiple pre-baked variants (e.g. one per BatchRenderMode or per GPUTextureFilter) are listed in the TEMPLATE_VARIANTS dict at the top of tools/regen_d3d12_dxbc.py. Each entry maps an .hlsl filename to the list of (variant_suffix, [/D defines]) tuples that the regen script should produce.

If you add a template variant to the dict, add the matching extern declaration to src/common/d3d12/embedded_shaders.h AND the matching slot lookup to src/common/d3d12/embedded_shaders.cpp - the index ordering at the runtime call site must match the order in the dict so the correct blob gets bound at PSO-creation time.