Skip to content

fix(inputs): stop baking DISTINCT into data-driven option queries - #3328

Open
rinkeraven-helloprint wants to merge 3 commits into
evidence-dev:mainfrom
rinkeraven-helloprint:fix/no-distinct-in-option-query-group-by
Open

fix(inputs): stop baking DISTINCT into data-driven option queries#3328
rinkeraven-helloprint wants to merge 3 commits into
evidence-dev:mainfrom
rinkeraven-helloprint:fix/no-distinct-in-option-query-group-by

Conversation

@rinkeraven-helloprint

Copy link
Copy Markdown

Personal Message

Hi @hughess posting this PR as a potential fix for an issue I also reported in Slack. When testing the new OSS Evidence I ran into the problem that dropdown filters do not work with the Cube SQL API because the generated SQL adds DISTINCT in the GROUP BY clause which Cube SQL API does not accept. Dropdown values do not populate because of this. Code in this PR is AI-driven but I did confirm locally that dropdowns work correctly after this fix. Let me know if creating PRs this way is desired or if you prefer me to only post issues if I run into future problems. Thanks!

Everything below AI-generated:


Problem

Every data-driven input is broken against Cube — dropdowns render "No options found"
and the options query 500s:

SELECT DISTINCT store AS "value" FROM ga4_sessions
WHERE (store IS NOT NULL)
GROUP BY DISTINCT store          -- SQL Parser Error: Expected: joined table, found: store
ORDER BY store LIMIT 10000

Root cause

The inputs build the value column with the quantifier baked into the string:

processColumnExpression({ value: `DISTINCT ${valueColumn} as value` }, dialect)

processColumnExpression keeps the remainder as sqlWithoutAlias (DISTINCT store);
hasAgg only matches an aggregate name followed by (, so the column is classified as a
plain dimension; and generateSQLQuery copies every non-aggregate column's
sqlWithoutAlias into the grouping list (sql-options.ts:583) — yielding
GROUP BY DISTINCT store.

That form is valid in PostgreSQL 14+ (it de-duplicates grouping sets), so most engines
tolerated the redundant clause. Cube's DataFusion parser rejects it. It's also invalid
T-SQL, so Fabric was affected too.

Fix

-  value: `DISTINCT ${valueColumn} as value`
+  value: `${valueColumn} as value`

The GROUP BY was already doing the de-duplication, so the keyword was pure redundancy.
It's emitted unconditionally — willUseGroupBy = !config.skipGroupBy
(sql-options.ts:532), and only image/build-image-sql.ts sets skipGroupBy — and every
option config keeps a non-aggregate column, so the clause is never empty.

Applied to all five call sites: dropdown, button_group, input_tabs, table_filter
(StringValueSelector), and repeat (build-repeat-query-config.ts). dimension_grid
hand-writes its own SQL and is unaffected.

Behaviour is otherwise unchanged: the alias stays value, hasAgg stays false, and
ORDER BY handling is unaffected. On GROUP BY ALL dialects (ClickHouse, Snowflake,
BigQuery, Databricks, MotherDuck) only the redundant SELECT DISTINCT goes; on explicit
ones (Postgres, Cube, Fabric) the grouping list loses the quantifier.

SQLQueryConfig.shouldAddDistinct (declared :146, honoured :679) would add a
query-level DISTINCT, but it's dead code and SELECT DISTINCT … GROUP BY … is redundant
by construction, so I left it unused. Happy to wire it up or delete it if maintainers
prefer.

Testing

New core/src/user-components/common/option-query-group-by.test.ts asserts across all
eight dialects that the options query emits a non-empty GROUP BY with no set quantifier
in it. Scoped to that clause on purpose: DISTINCT is legal elsewhere (shouldAddDistinct,
and IS NOT DISTINCT FROM from nullSafeEqual). A second guard scans the call sites, since
the expression is a template string no SQL assertion can see into; it matches the
construction shape and covers ALL too, and is mutation-tested against four evasions.

Expectations updated in build-repeat-query-config.test.ts and
string-value-selector-order.test.ts.

pnpm test    → core 232 files / 3652 passed, 3 skipped; cli 18 files / 340 passed
pnpm check   → core 0 errors

Known follow-up (not in this PR)

hasAgg still doesn't recognise a bare quantifier, so an author-written
<BarChart x="distinct category" /> still produces GROUP BY distinct category and the
same Cube error. Closing that class means stripping or rejecting the quantifier at
sql-options.ts:583 — wider blast radius, so I kept it separate. Glad to follow up.

rinkeraven-helloprint and others added 3 commits August 28, 2026 20:27
The five data-driven inputs built their value column as the string
`DISTINCT <col> as value`. processColumnExpression keeps that whole string
as `sqlWithoutAlias`, and hasAgg only matches an aggregate name followed by
`(`, so the column was classified as a plain dimension and the quantifier
was copied into the GROUP BY:

    SELECT DISTINCT store AS "value" FROM ga4_sessions
    WHERE (store IS NOT NULL) GROUP BY DISTINCT store ORDER BY store

`GROUP BY DISTINCT <expr>` is legal Postgres 14+ (it de-duplicates grouping
sets), so most warehouses tolerated the redundant clause. Cube's
DataFusion-based parser rejects it — "Expected: joined table, found: store" —
so every data-driven dropdown rendered "No options found".

The GROUP BY that generateSQLQuery already emits for non-aggregate columns
is what makes the options distinct, so the keyword was redundant everywhere:
drop it and let the GROUP BY do the de-duplication.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A blanket `not.toMatch(/\bDISTINCT\b/)` over the whole query asserted more
than the fix. DISTINCT is legal elsewhere: `shouldAddDistinct` exists to put
one on the SELECT list, and `IS NOT DISTINCT FROM` is the codebase's own
null-safe equality, which an author's `where` could contain. Assert the
invariant that actually broke Cube — no quantifier in the GROUP BY.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…fier guard

Two review findings.

StringValueSelector's comment claimed the COUNT(*) forces the GROUP BY on
the value column. It doesn't: willUseGroupBy is unconditional
(sql-options.ts:532), so the GROUP BY is emitted whether or not an aggregate
is present — as the other four call sites show. The COUNT(*) is there for
minimum_records. Reworded to match its siblings.

The call-site guard only matched a quantifier inline in the argument to
processColumnExpression, so assigning the string to a variable first slipped
past, as did the ALL quantifier (which fails in DataFusion identically).
Match the construction shape instead, and strip comments first since the
call sites legitimately discuss DISTINCT in prose.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@vercel

vercel Bot commented Aug 29, 2026

Copy link
Copy Markdown

@rinkeraven-helloprint is attempting to deploy a commit to the Evidence Team on Vercel.

A member of the Team first needs to authorize it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant