The DC coefficient (zigzag[0]) and subsequent AC coefficient
computations in MDEC::DecodeRLE_New use signed left shifts that
can overflow into the sign bit:
(val << 5) // when val is negative
(val * qt[0]) << 4 // when val * qt[0] is negative
((val * scq) >> 3) << 4 // when val * scq is negative
val comes from sign-extending the low 10 bits of an MDEC RLE
stream halfword to s32; its range is [-512, 511]. qt[i] is u8
[0, 255]. scq is the product m_current_q_scale * qt[i] which
also routinely lands negative or large depending on the input
block.
Signed left shift of a negative value (or a positive value whose
shifted result would not fit in the signed type) is C++<20
undefined behaviour. The C++20 standard redefined signed left
shift in terms of unsigned 2's-complement arithmetic, making it
well-defined.
Same idiom as the BitField::GetValue() and TruncateVertexPosition
fixes that just landed: do the shift in the unsigned domain, then
cast back to s32. The arithmetic right shifts inside the existing
sign-extension chain (val computed from 'static_cast<s32>(u32 << 22) >> 22')
stay unchanged - those are arithmetic right shifts of a signed
value, which is well-defined since C++11.
Codegen and observable behaviour: zero change.
A test harness running 1159 cases covering the full val range
[-512, 511], qt values {0, 1, 127, 255} and q_scale values
{0, 1, 63} plus various scq values reports zero differences
between the old and new forms. -O2 g++ x86_64 emits byte-
identical assembly for both: the testb-jne-sall pattern for the
q_scale-zero path, and the movzbl-imull-sall-leal pattern for
the non-zero path are unchanged.
The fix removes the UB so the code stays well-defined under
future compiler changes (and across compilers - GCC happens to
make the UB resolve to the same value as C++20 modular shift,
but that's not guaranteed).