Broken Shader Using Loops When Optimizations ON

[jmweever]

It seems like any shader with a loop comes out of respv::Optimizer::run broken, as long as it was compiled with optimizations ON. NVIDIA's driver then segfaults inside vkCreateComputePipelines, and Mesa lavapipe returns VK_ERROR_UNKNOWN.

With optimizations on, a loop counter is not kept in memory. Instead the top of the loop has an OpPhi meaning "the counter is 0 on the first pass, or counter + 1 on every later pass". Nothing except that OpPhi reads the instruction computing counter + 1. re-spirv does not count an OpPhi as a reader, so it decides that instruction is unused and deletes it. The OpPhi is left pointing at an instruction that no longer exists.

This looks like #17 and #20, but the cause is different. No constant conditions or branch pruning are involved, just dead code elimination miscounting.

Reproduction

min_repro.zip

#version 460
layout( local_size_x = 64 ) in;
layout( binding = 0, std430 ) buffer Out { float vals[]; };
layout( push_constant, std430 ) uniform Params { int n; };
void main()
{
	for ( int x = 0; x < n; x++ ) vals[ x ] = 1.0;
}

glslc -fshader-stage=compute --target-env=vulkan1.2 -O min_loop.glsl -o min_loop.spv

The min_loop.spv passes spirv-val. After respv::Optimizer::run, it fails with The following forward referenced IDs have not been defined: '35[%35]', because %35 is gone while %41 still names it:

 %10 = OpLabel
 %41 = OpPhi %int %int_0 %5 %35 %11
       OpLoopMerge %12 %11 None
       OpBranchConditional %23 %11 %12
 %11 = OpLabel
       OpStore %32 %float_1
-%35 = OpIAdd %int %41 %int_1
       OpBranch %10

Loop shape does not matter. Constant trip counts, while with break, and accumulate-only bodies all reproduce it. A phi from ?: does not, since it has no back edge.

Potential Cause

It does seem that Shader::parseData skips the back edge at re-spirv.cpp:1974:

// Make sure this label doesn't come from the loop continue.
if (labelId == continueLabelId) {
    continue;
}

Skipping the adjacency edge is correct, since instructionAdjacentListIndices feeds the topological sort in Shader::sort and the back edge would make the graph cyclic.

But that same list also seeds instructionOutDegrees (re-spirv.cpp:2131), which is what dead code elimination reads. The optimizerReduceResultDegrees deletes an instruction once its out degree hits zero, so an increment that only feeds the loop header phi starts at zero and is removed.

It seems like the count is already reduced in the other direction: the teardown at re-spirv.cpp:2332 passes pIncludePhi = true, so deleting an OpPhi reduces the degree of every operand it names, back edge included. It is subtracted but never added.

Possible Fix

I have attached a patch. It keeps the back edge out of the adjacency list so the sort still works, and counts it into the out degrees separately. Deletion already subtracts these, so the count balances.

I tested this with eight compute shaders from a real project, each at -O0, -O and -Os, plus the case above.

input	before	after
5 shaders with loops, at -O and -Os	invalid, no pipeline	valid, pipeline works
3 shaders without loops, all levels	valid	valid, output byte for byte the same
all 8 shaders at -O0	valid	valid, output byte for byte the same

I should note that unoptimized input is untouched, so this is a no-op for callers that never optimize first. A dead loop still gets removed, with identical output before and after. I also confirmed end-to-end in a real game on Godot 4.8.dev (5ec4857b3), NVIDIA 610.57.04, RTX 2070 SUPER. Same release export, only re-spirv swapped: stock segfaults in libnvidia-glvkspirv.so via compute_pipeline_create, patched runs correctly.

How I Ran Into This

Godot Engine uses re-spirv and runs it on every shader before vkCreateShaderModule on Vulkan. Godot's own shaders are unaffected because it compiles them with the optimizer off:
In godot/modules/glslang/register_types.cpp:126 it default-constructs glslang::SpvOptions, and glslang defaults disableOptimizer to true (SPIRV/GlslangToSpv.h:50, godot/thirdparty/glslang/SPIRV/GlslangToSpv.h:50).
It seems this is probably reachable from anything directly handing optimized SPIR-V to shader_create_from_spirv.