opt: preserve NULL partition boundaries in scan constraints

[Alignyx]

Summary and scope

Fix missing rows in partition-derived index scans when a LIST partition boundary
contains NULL. The optimizer must cover both defined partition keys and every gap
between them. Ordinary SQL tuple comparisons against NULL do not express that
physical key coverage, so the existing optional filters can omit entire leading
prefixes before filtering, ordering, limiting, or updating rows.

This PR corrects that metadata-to-filter translation in columnComparison, using
null-safe equal prefixes and lexicographic comparison for NULL-containing
boundaries. It preserves the existing non-NULL boundary path, ordinary SQL
comparison semantics, partition definitions, key encoding, and index layout.

The fix addresses #137994's NULL-boundary failure, including unhinted plans. It
does not disable partition-derived scans or reject existing legal partitions.

Problem and correct-result oracle

The investigation retained the report's types, predicate, ordering and limit:

CREATE TABLE tab (
  col1_0 NAME PRIMARY KEY,
  col1_1 INT8,
  col1_3 INT8,
  col1_5 VARCHAR,
  UNIQUE (col1_1, col1_3) PARTITION BY LIST (col1_1, col1_3) (
    PARTITION p0 VALUES IN ((1, NULL)),
    PARTITION p1 VALUES IN ((1000000000, NULL)),
    PARTITION p2 VALUES IN ((2000000000, NULL))
  )
);
INSERT INTO tab VALUES
  ('a', 1, NULL, 'foo1'), ('b', 1000000000, NULL, 'foo2'),
  ('c', 2000000000, NULL, 'foo3'), ('d', 0, NULL, 'foo4'),
  ('aa', 1, 1, 'foo1'), ('bb', 1000000000, 1, 'foo2'),
  ('cc', 2000000000, 1, 'foo3'), ('dd', 0, 1, 'foo4');

SELECT * FROM tab@{NO_FULL_SCAN}
WHERE col1_0 ILIKE col1_0
ORDER BY col1_3 LIMIT 84;

All eight explicit names satisfy this predicate, and the limit exceeds the row
count. The correct result bag therefore contains all eight rows, independently
of which index is chosen. Likewise, the corresponding UPDATE must affect all
eight rows. The baseline instead selects/updates only d and dd.

The omitted rows include both defined partition tuples with a NULL second column
and rows with the same leading values but a non-NULL second column. These are
valid stored rows; partition metadata must not be treated as an exhaustive
restriction on possible data.

Metamorphic relations used in the investigation

MR1: access-path invariance

Keep the schema, data, SQL types, predicate, projection, ordering and limit fixed;
change only the access path. The equivalent control uses tab@tab_pkey in place
of tab@{NO_FULL_SCAN}:

SELECT * FROM tab@tab_pkey
WHERE col1_0 ILIKE col1_0
ORDER BY col1_3 LIMIT 84;

The primary-index control returns the independently expected eight rows. The
hint is not itself a correctness oracle; the explicit input set establishes the
expectation and the different plan helps locate the failure.

Importantly, simply removing NO_FULL_SCAN is not a correct control on the
tested baseline. Its unhinted SELECT and UPDATE also return or update only d
and dd. Those were classified as additional failing W cases, not used to define
the expected result. This also agrees with the issue's
unhinted reduced reproduction.

MR2: materialize the complete input before applying the same query

WITH input AS MATERIALIZED (SELECT * FROM tab@tab_pkey)
SELECT * FROM input
WHERE col1_0 ILIKE col1_0
ORDER BY col1_3 LIMIT 84;

For this pure query, introducing an optimization boundary around the complete
primary-index input preserves the result. It changes where partition-derived
constraints can participate, providing another eight-row control without changing
the predicate or widening its types. This relation is not an unconditional claim
about volatile expressions or evaluation-order-dependent errors.

MR3: selected target rows and rollback-isolated UPDATE effects agree

The target set selected with the same predicate, ordering and limit must match
the rows changed by the UPDATE. Each mutation probe ran in its own transaction:

BEGIN;
UPDATE tab@{NO_FULL_SCAN} SET col1_5 = 'bar'
WHERE col1_0 ILIKE col1_0
ORDER BY col1_3 LIMIT 84;
SELECT col1_0 FROM tab@tab_pkey
WHERE col1_5 = 'bar' ORDER BY col1_0;
ROLLBACK;

The same probe was repeated unhinted and with a forced-primary UPDATE. Rolling
back restores the initial data for each comparison; subsequent checks
require eight rows and zero bar markers. The primary scan used to inspect the
effects avoids reusing the faulty access path as the observer.

The main LIMIT 84 cannot truncate eight rows, so ties in ORDER BY col1_3 do
not make the expected target bag ambiguous. Held-out tests that actually truncate
results use a unique ID tie-breaker and explicit NULL ordering to define a
deterministic selected set. The investigation does not assume arbitrary tied
LIMIT queries must choose the same rows under different plans.

Root cause and how it was localized

GenerateConstrainedScans uses optional filters derived from both partition
values and the intervals between them. inPartitionFilters and
inBetweenFilters share columnComparison. Their resulting constraints must
jointly cover every possible key before the query's actual predicates are applied.

For a boundary (1,NULL), ordinary SQL comparisons are unsuitable for that job:

• (a,b) = (1,NULL) cannot be TRUE, including for the boundary tuple itself.
• (a,b) > (1,NULL) admits a > 1, but not rows with a = 1 and a non-NULL b.
• Combining the partition and complement constraints can therefore omit the
  entire a = 1 prefix rather than just an impossible SQL predicate result.

idxconstraint is correctly respecting SQL three-valued comparison semantics.
The invalid step is expressing partition-key boundaries as those SQL tuple
predicates. The correction belongs at that translation boundary, not in the
general SQL comparator or constraint engine.

EXPLAIN (OPT, VERBOSE) exposed the missing prefixes, while EXPLAIN ANALYZE
showed that rows were lost at the scan, before filtering or mutation. The main
baseline/candidate comparison is:

Observation	Baseline	Candidate
Hinted and unhinted SELECT	d, dd	All eight rows
Hinted and unhinted UPDATE effects	d, dd	All eight rows
Row-engine / distsql=always SELECT controls	d, dd	All eight rows
Primary/materialized SELECT and primary UPDATE	All eight rows	All eight rows
W scan / filter / top-k / index-join actual row counts	2 / 2 / 2 / 2	8 / 8 / 8 / 8
W total KV rows decoded, including primary lookups	4	16

Three complete baseline and three complete candidate matrices agree. In the
candidate, W retains the secondary-index scan, filter, top-k and index join.
The important lower span bounds change as follows:

Baseline lower bound	Corrected lower bound
/2	/1/NULL
/1000000001	/1000000000/NULL
/2000000001	/2000000000/NULL

The candidate now reads eight secondary entries and performs eight primary
lookups; the full-primary control reads eight KV rows. This is corrected scan
coverage, not a fallback that avoids the original partitioned path.

The main plan observations are local, single-node execution. Its
distsql = always arm is a distribution-setting control, not proof of remote-node
execution. These observations plus source analysis are not instrumented
branch-coverage traces.

How the repair works

Only a boundary containing NULL takes the new path. Comparisons use the existing
logical datum ordering, where NULL is below non-NULL values. Index ASC/DESC
directions are still interpreted by idxconstraint; this does not assert
that NULL has the smallest encoded byte position in every descending index.

For equality, the expansion requires null-safe equality at every boundary
position, using ConstructIs. For strict comparisons, it builds the expression
from the last position toward the first:

compare_at_i = strict_at_i OR (null_safe_equal_at_i AND compare_suffix)

The strict-comparison suffix starts as FALSE; the equality suffix starts as TRUE.
The per-position rules are:

Boundary value	Less-than branch	Greater-than branch
NULL	FALSE	column IS NOT NULL
Non-NULL value v	column < v OR column IS NULL	column > v

For (a,b) versus (1,NULL), this retains equality at (1,NULL) and admits
a = 1, b IS NOT NULL into the greater-than side. Thus both the defined partition
and the gap immediately after it remain represented.

The required invariant concerns accepted TRUE membership. An expression can
still evaluate to UNKNOWN outside that set; the patch does not redefine SQL's
three-valued logic. Generated index constraints may safely overapproximate the
predicate, but must not omit a matching key. Residual query filters remain in
place to enforce the original query.

This shared helper repairs both partition and in-between filters. Boundaries
without NULL keep the existing tuple path, and the historical first-column
IS NULL compensation is retained. No tuple elements are reordered, partition
definitions rejected, or storage structures changed.

Discussion, historical precedent, and existing fixtures

The root-cause discussion
identified fragile NULL tuple span generation and considered restricting NULL in
partition tuples. This PR takes the narrower correctness approach of representing
the existing legal boundaries correctly. It does not claim the entire partition
constraint framework has been redesigned.

Historical commit a15bda7,
from PR #63932 fixing #63733, added first-column NULL coverage and regressions.
That compensation and its positive controls are preserved here.

The full optimizer suite exposed three old expected plans whose spans needed
audited corrections, not wholesale snapshot acceptance:

• Two limit plans now start at the missing /key/NULL boundaries, including
  the original report's leading values and another set of 1, 5, 5000000.
  Their predicates, ordering, LIMIT 84, and positive/negative
  SplitLimitedSelectIntoUnionSelects assertions are unchanged.
• The historical t63733_nulls plan now has valid constrained spans where it
  previously scanned without a constraint. Its required c = 1 filter remains;
  coverage review confirms that only nonqualifying keys are excluded.

These are ten span-content changes across two files and three plans. A positive
rule assertion establishes application during exploration, not that the final
chosen plan must contain a Union operator.

Completed local validation

Results refer to existing commit f7581d42fd0129f315fa5e4ed3b9918bd99887cf, based
on 8812064a015d2faf99d3fc7e15880f94042954b0. They describe completed local
validation, not upstream PR CI status.

• Frozen native unit regression: TestPartitionBoundary137994 has 600 direct
  comparison cases: 25 NULL-containing boundary tuples, three comparisons and
  eight ASC/DESC combinations. Each enumerates 27 keys. An independent
  Datums.Compare oracle checks accepted TRUE membership and matching-key span
  coverage. Another 48 cases check the combined partition/complement coverage
  for duplicates, prefixes, nested NULLs, gaps, leading NULL and non-NULL controls.
  The frozen test fails semantically on baseline and passes on the candidate.
• Frozen native SQL regression: eight rows cover defined boundaries, gaps,
  non-NULL suffixes and leading NULL values outside the defined partitions.
  Checks include the limited scan, a later-column predicate, UPDATE effects and
  rollback preservation. The baseline stops at the first SELECT, returning four
  rows instead of eight; later statements are not claimed to have independent
  native-red observations. The complete unchanged regression passes with the fix.
• Repeated metamorphic replay: all six complete matrices pass exact
  baseline/candidate outcome checks, including baseline span bounds, observed
  scan/root counts and rollback.
  Fifteen deliberately corrupted or mislabeled evidence variants are rejected.
• Independent held-outs: all 11 W/C pairs match the literal-input oracle on
  the candidate, correcting eight baseline W deviations. Cases cover NULL in all
  three positions, shared/DEFAULT prefixes, descending indexes, deterministic
  limited selections and UPDATE effects. Six tables retain all 94 rows with zero
  rollback markers. Three hand-computed LIMIT checks and eleven verifier rejection
  self-tests pass.
• Broader regression coverage: full constraint, idxconstraint, partition,
  norm, xform, props and execbuilder unit targets pass. On final verification,
  six unchanged successful targets are cached; full xform reruns after the audited
  expected-span updates. Eleven named partition/index logic executions pass across
  supported local, fakedist and 5node configurations. Six complete execution-plan
  suites (select, select_index, update, delete, limit, topk) and all
  16 SQL test shards also pass.

The committed scope is six files (+291/-9): the 42-line production diff, a
172-line unit regression, a 68-line SQL regression, two Bazel entries and the
audited changes to two historical plan fixtures. Supplemental replay and held-out
artifacts are retained in the local investigation record, not included as extra
PR files.

Current limitations

• Narrow boundary correction: the NULL-free tuple path and its existing
  dependence on current span-generation behavior remain. This is not a complete
  rewrite or proof of every partition-derived optimization.
• Constraint quality: safety means no matching-key omission, not the tightest
  possible spans. Existing mixed-direction limitations tracked by opt: support constrained in between filters with multiple partitioning columns #81456 remain;
  eight-direction unit coverage does not establish that all mixed ASC/DESC
  partition scans now produce useful or optimal constraints.
• Execution coverage: the main distribution control is single-node. A forced
  index alone is not a rule-execution trace, and enabling vectorization does not
  prove UPDATE operators themselves ran in a vectorized engine. Result checks
  establish bags or deterministic selected sets, not every original output's
  natural row order.
• Performance and versions: no benchmark, workload-wide plan-quality study,
  mixed-version upgrade matrix or all-release backport certification is supplied.
  NULL-containing boundaries now build linear-size lexicographic expressions;
  their wider-workload planning costs have not been quantified.
• Past mutation effects: the patch prevents the covered omissions in future
  executions; it cannot retroactively complete application UPDATE statements
  that already missed rows. Any reconciliation of historical application effects
  is separate from this optimizer change.

Resolves: #137994
Epic: none

Release note (bug fix): Fixed an optimizer bug that could omit qualifying
rows from SELECT and UPDATE statements on tables with list-partitioned
indexes whose partition boundaries contain NULL values. This could occur
with or without a NO_FULL_SCAN hint.

[blathers-crl]

Thank you for contributing to CockroachDB. Please ensure you have followed the guidelines for creating a PR.

My owl senses detect your PR is good for review. Please keep an eye out for any test failures in CI.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}