Skip to content

fix(#1948): read float buffers as bytes in the CIccIO writers, and record the corpus verdicts (#1946) - #1947

Merged
xsscx merged 2 commits into
masterfrom
test/1946-qa-profile-manifest
Aug 2, 2026
Merged

fix(#1948): read float buffers as bytes in the CIccIO writers, and record the corpus verdicts (#1946)#1947
xsscx merged 2 commits into
masterfrom
test/1946-qa-profile-manifest

Conversation

@colourbill-ctrl

@colourbill-ctrl colourbill-ctrl commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Fixes #1946.
Fixes #1948.

#1809 carried two halves: the empty-processing-element policy, which landed as #1887, and
the QA manifest, which @xsscx authorized on 2026-07-28 in
#1809 (comment)
("Go with PoLR paths based on your preference & resolution"), taking both resolutions I
proposed: sha256 scoped to what git tracks, and noncompliant recorded together with
expected_exit: 0 rather than assuming the two correlate. #1809 closed when #1887 merged,
so the manifest half is filed separately as #1946 and this PR delivers it.

The problem

The corpus mixes three kinds of profile that a "did iccDumpProfile exit non-zero" sweep
cannot tell apart: conformant examples, profiles carrying a known baselined warning, and
fixtures that are malformed on purpose. 78 of the 213 profiles are deliberate negatives.
Without a manifest they either read as unexplained failures, or — if a sweep tolerates
them — they mask a conformant profile that has newly turned critical.

What the corpus actually does

Measured in a clean worktree, rebased onto c95a8343. That matters: a working tree that
has run the suite carries 232 .icc under Testing/ where a clean checkout has 213, so measuring in
place produces the wrong baseline.

suite expected status exit count
positive valid 0 108
compatibility warning 0 27
negative critical 255 78

Suite is derived from the verdict rather than hand-assigned, so the manifest stays
regenerable and nobody adjudicates 213 rows by hand.

Four things that came out of the measurement

1. All 27 warnings are #1811. Every one is the
spectralViewingConditions illuminantXYZ/surroundXYZ "appears to be normalized"
diagnostic. That gives #1811 a measured scope it did not have: fixing it moves all 27
profiles from warning to valid, leaving zero compatibility rows, and is a deliberate
re-baseline of this manifest. Corrected on the issue in
#1811 (comment)
my first census said 27 of 28, with Calc/RGBWProjector.icc as an unrelated zero-range
sampled curve. That profile was never a warning; it was the #1948 miscompilation described
below.

2. No profile validates as overall noncompliant. The asymmetry #1809 asked me to
encode has no live instance. NonCompliant! - appears as a per-tag diagnostic line in
quantity, but never as the profile verdict. The status-to-exit mapping is recorded in the
manifest header as the explicit contract, and the header says plainly that no row exercises
that pairing today, rather than implying coverage that does not exist.

3. calcUnderStack_fLab.icc does not report a stack underflow. It reports
"function has invalid operations" — the operation is rejected before stack depth is ever
assessed — unlike its 73 siblings. Recorded as its own reason so the manifest does not
imply it exercises the same validator path. Flagging as an observation, not a defect claim.

4. The issue's repro passes ALL; it makes no difference here. I measured both
iccDumpProfile -v 26 <f> and iccDumpProfile --diag -v 26 <f> ALL across all 213
profiles: identical status line and identical exit code in every case. The test uses the
cheaper form and runs in ~5s.

sha256 scoping

80 tracked profiles carry a digest; the 133 generated ones carry -. 178 of the 191
tracked Testing XML fixtures set <CreationDateTime>now</CreationDateTime>, which
icGetDateTimeValue resolves through localtime_r at conversion time and which then feeds
the recomputed profile ID — so those bytes are neither clock- nor timezone-stable.
Re-verified for this PR: two conversions of Calc/CameraModel.xml two seconds apart differ.
The digest is read from the git object rather than from disk, so regenerating the corpus in
place cannot silently re-baseline it.

Scope

Covers Testing/ only. The 24 profiles tracked under .github/ci/regression/ and
.github/ci/test-data/ are deliberately excluded and the manifest header says so: each is
already the input to a dedicated regression test whose assertion is narrower than a
whole-profile verdict. Testing/hybrid/ICC and Testing/hybrid/Results are excluded
because they are iccdev.hybrid-pipeline output rather than corpus —
Testing/hybrid/.gitignore declares exactly those two paths as generated, and it is the
only .gitignore under Testing/ that declares .icc output. Without that exclusion the
test is ordering-dependent, seeing 213 profiles alone and 232 after the hybrid pipeline has
run.

Verification

  • Clean worktree at c95a8343, both toolchains: 0 build warnings and 146/147 ctest
    under gcc strict ASAN+UBSan Release (CI's own configuration) and under clang Release with
    LTO. The single failure in both is spectral-tiff-preview, which fails because this local
    environment lacks the python imagecodecs module - unrelated, and failing before this
    change.
  • Re-run under an ASan+UBSan build: 213/213 match, so expected_sanitizer: none is
    verified rather than asserted.
  • Red-tested five failure modes plus a control: wrong status/exit, a corpus profile
    missing from the manifest, a manifest row whose profile is gone, tracked-fixture digest
    drift, and sanitizer detection. All five fail as intended; the unmodified control passes.
  • generate is idempotent: regenerating produces a byte-identical manifest.
  • The manifest survives a full corpus regeneration, which is the property that makes
    scoping sha256 workable.
  • shellcheck clean under CI's own invocation. Avoids declare -A: macOS ships bash 3.2
    and nothing else under .github/scripts requires bash 4.

The second half: #1948, and why this PR grew

The manifest above was red in CI on exactly one row, and the row was right about my machine
rather than about the corpus. Chasing that produced a real IccProfLib defect.

What it is

CIccIO::Write16/Write32/Write64 loaded the caller's buffer through the wide integer
type before byte-swapping it:

// IccProfLib/IccIO.cpp, CIccIO::Write32
icUInt32Number *ptr = (icUInt32Number*)pBuf32;
tmp = *ptr;

WriteFloat32Float forwards straight into Write32 whenever icFloatNumber and
icFloat32Number are the same width, which is the normal build. So the object being read is
a float, read through an icUInt32Number lvalue. Those are not similar types, so the
compiler may assume a store through one is not observable by a load through the other.

ENABLE_LTO defaults to ON, which puts the store in CIccSegmentedCurve::Write and this
load into one optimization scope. clang at -O3 then drops the store as dead. That function
reuses a single stack slot for every breakpoint, so the slot kept its initial zero and every
breakpoint reached the file as 0.0.

The read path was never affected: icSwab32Array walks the buffer as icUInt8Number, which
may alias any object. That asymmetry is why readers agreed across toolchains while writers
did not.

Why it matters

Not cosmetic. A flattened trailing breakpoint gives the sampled segment an end point equal to
its start point, and CIccSampledCurveSegment::Begin (IccMpeBasic.cpp:1187) refuses a zero
range — so the curve set cannot be applied at all. Any profile written by an optimized clang
build carries this, including the default macOS toolchain.

gcc is unaffected today, but by the optimizer's choice rather than by the standard.

Evidence

Same source, same commit, unpatched tree, only the build changed:

build trailing breakpoint
clang -O3, LTO on (project default) 0x00000000
clang -O3 -fno-strict-aliasing 0x3F800000
clang -O3 + ASAN/UBSan 0x3F800000
clang Debug 0x3F800000
gcc -O3 0x3F800000

-fno-strict-aliasing alone flipping it at the same -O3 is the discriminator. Note that
neither ASAN nor UBSan catches this, so a green sanitizer lane is not evidence of absence.

The fix

The writers now walk the buffer as unsigned char and reorder bytes into a small scratch
buffer, matching what icSwab32Array has always done on the read side.

The bytes are shuffled directly rather than copied out with memcpy. Both forms are well
defined, but memcpy is intercepted under ASAN, and a call per element cost about 20% on
iccdev.issue-1781-applytolink-qa-matrix — enough to push it past its timeout under parallel
load, since these writers carry every CLUT a profile contains:

Write32 form that test, under ASAN
original (undefined) 26.3s
memcpy 31.5s
byte shuffle (this PR) 24.7s

Verification

  • .github/ci/regression/iccio-write-float-aliasing.cpp drives CIccSegmentedCurve::Write
    and asserts the emitted breakpoint bytes, then reads them back and confirms the curve still
    begins. Both the store and the load live inside IccProfLib, so it reproduces wherever the
    library is miscompiled, independently of how the test translation unit is built.
  • Red-tested: on the unfixed tree it fails with exactly
    second breakpoint wrote 0x00000000, expected 0x3F800000.
  • All 191 tracked Testing/*.xml now convert to byte-identical profiles under clang -O3
    and gcc -O3, modulo the creation date (24-35) and profile ID (84-99) recorded per
    conversion. hybrid/CMYK-STop_Overprint_Profile also differs at 2685017-2685083, which is
    the same date/ID pair of an embedded profile at offset 2684984, not a divergence.
    Calc/calcImport.xml, Calc/calcVars.xml and SpecRef/RefEstimationImport.xml are not
    standalone-convertible under either toolchain.

The one manifest row

Calc/RGBWProjector.icc moves from compatibility/warning to positive/valid. It is
generated from XML at test time, so the baseline had captured this miscompilation as expected
behaviour. Regenerating the manifest instead of fixing the writers would have written my
host's miscompilation into the repository — the row count that had to change was derived from
CI's own 212 matched, 1 mismatched, not from a wholesale re-baseline.

…very corpus profile

The Testing corpus mixes three kinds of profile that a "did iccDumpProfile exit
non-zero" sweep cannot tell apart: 107 conformant examples, 28 carrying a known
baselined warning, and 78 that are malformed on purpose. A sweep that fails on a
non-zero exit reports the 78 deliberate negatives as unexplained failures; one that
tolerates them cannot see a conformant profile that has newly turned critical,
because "some profiles are critical" is already the normal state.

Testing/qa-profile-manifest.tsv records the expected verdict per profile and a CTest
enforces it. Suite is derived from the verdict rather than hand-assigned, so the
manifest stays regenerable and nobody adjudicates 213 rows by hand.

Per the resolutions agreed on #1809: sha256 is populated only for the 80 git-tracked
profiles, because 178 of the 191 tracked Testing XML fixtures set
<CreationDateTime>now</CreationDateTime>, which resolves through localtime_r at
conversion time and feeds the recomputed profile ID - so the generated corpus is
neither clock- nor timezone-stable. expected_status and expected_exit are recorded as
independent fields because iccDumpProfile maps noncompliant to exit 0.
@github-actions github-actions Bot added Testing CTest, regression, or test coverage Scripts Shell, PowerShell, or repository automation scripts Configuration Repository, CMake, YAML, JSON, or tool configuration Build Build system, CMake, compiler, or packaging Unix Linux, macOS, Bash, or POSIX shell scope pending CI checks still running labels Aug 1, 2026
@xsscx xsscx self-assigned this Aug 1, 2026
@xsscx
xsscx enabled auto-merge (squash) August 1, 2026 23:57
@xsscx

xsscx commented Aug 2, 2026

Copy link
Copy Markdown
Member

CI Error Log

2026-08-02 00:14:21 UTC

 67/142 Test  #67: iccdev.qa-profile-manifest .....................................***Failed   24.81 sec
[FAIL] corpus validation verdicts do not match the manifest
       If this is an intended change, re-baseline with:
       .github/scripts/iccdev-qa-profile-manifest.sh generate
  [STATUS] Calc/RGBWProjector.icc -- expected warning, got valid (suite compatibility)

qa-profile-manifest: 213 profiles, 212 matched, 1 mismatched, 0 unlisted, 0 missing (213 manifest rows)

…iters

CIccIO::Write16/Write32/Write64 loaded the caller's buffer through the wide
integer type before byte-swapping it:

    icUInt32Number *ptr = (icUInt32Number*)pBuf32;
    tmp = *ptr;

WriteFloat32Float forwards straight into Write32 whenever icFloatNumber and
icFloat32Number are the same width, which is the normal build, so the object
being read is a float. A float and an icUInt32Number are not similar types, so
that load is undefined: the compiler may assume a store through one cannot be
observed by a load through the other. ENABLE_LTO defaults to ON, which puts the
store in CIccSegmentedCurve::Write and this load in one optimization scope, and
clang at -O3 drops the store as dead. Every breakpoint then reached the file as
the stack slot's initial 0.0.

The read path was never affected because icSwab32Array walks the buffer as
icUInt8Number, which may alias any object. That asymmetry is why readers agreed
across toolchains while writers did not. The writers now walk the buffer the
same way, reordering bytes into a small scratch buffer.

The bytes are shuffled directly rather than copied out with memcpy. Both forms
are well defined, but memcpy is intercepted under ASAN, and a call per element
cost about 20% on iccdev.issue-1781-applytolink-qa-matrix - enough to push it
past its timeout under parallel load, since these writers carry every CLUT a
profile contains. The byte-shuffle form measures slightly faster than the
original undefined code (24.7s against 26.3s on that test).

The damage is not cosmetic. A segmented curve whose trailing breakpoint is
flattened to 0.0 gives its sampled segment an end point equal to its start
point, and CIccSampledCurveSegment::Begin refuses a zero range, so the curve set
cannot be applied at all. Any profile written by an optimized clang build -
including the default macOS toolchain - carries the damage.

gcc is unaffected today but is not immune: nothing in the standard obliges it to
keep honouring a load it is entitled to reorder.

Testing/qa-profile-manifest.tsv records Calc/RGBWProjector.icc as valid rather
than carrying a baselined zero-range warning. That row was measured on a
clang+LTO host, where the defect bites; the corpus is generated from XML at test
time, so the manifest had captured this miscompilation as expected behaviour.
With the writers fixed, all 191 tracked Testing XML fixtures convert to
byte-identical profiles under clang and gcc, modulo the creation date and
profile ID that the header records per conversion.

.github/ci/regression/iccio-write-float-aliasing.cpp drives
CIccSegmentedCurve::Write and asserts the emitted breakpoint bytes, then reads
them back and confirms the curve still begins. Both the store and the load live
inside IccProfLib, so the test reproduces wherever the library itself is
miscompiled, independently of how the test translation unit is built. It fails
on the unfixed tree with the exact 0x00000000-for-0x3F800000 substitution.
@github-actions github-actions Bot added failed One or more CI checks failed and removed pending CI checks still running labels Aug 2, 2026
@colourbill-ctrl colourbill-ctrl changed the title test(#1946): record and enforce the expected validation verdict for every corpus profile fix(#1948): read float buffers as bytes in the CIccIO writers, and record the corpus verdicts (#1946) Aug 2, 2026
@github-actions github-actions Bot added pending CI checks still running Source C or C++ source code changes and removed failed One or more CI checks failed labels Aug 2, 2026
@xsscx
xsscx merged commit c7883bd into master Aug 2, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Build Build system, CMake, compiler, or packaging Configuration Repository, CMake, YAML, JSON, or tool configuration pending CI checks still running Scripts Shell, PowerShell, or repository automation scripts Source C or C++ source code changes Testing CTest, regression, or test coverage Unix Linux, macOS, Bash, or POSIX shell scope

Projects

None yet

2 participants