feat: expose lineage uncertainty and expand dialects - #16
Draft
eitsupi wants to merge 18 commits into
Draft
Conversation
The order of `ColumnLineage.mappings` varied between runs of the same
binary whenever a query had duplicate output column names:
SELECT a.id, b.id FROM a JOIN b ON a.id = b.bid
sometimes [ id <- a.id , id <- b.id ]
sometimes [ id <- b.id , id <- a.id ]
With three duplicates, four distinct orderings showed up across six runs.
Callers that cache or diff results see the same input produce different
output.
`resolve` collected the output nodes into a `HashSet<NodeId>` and built
the mappings by iterating it, so construction order followed hash order
with a per-process random seed. The sort afterwards could not undo this:
it keyed on a `HashMap<String, usize>` of output names, so duplicate
names collided and one index won, leaving same-named mappings in
whatever order the set had produced.
`ordered_cols` already holds the projection order, so build the mappings
straight from it. That makes the sort redundant — it can only reproduce
the order the loop now has, and it is the reason duplicates were
reordered in the first place — so it goes too.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
The resolver returned ColumnOrigin::Concrete with invented table names
("?unknown?", "?cte?") when a column could not be resolved, so callers
could not tell fabricated lineage from real lineage.
Both sites now return ColumnOrigin::Ambiguous with an empty candidate
list, which already means "catalog needed to resolve" and keeps the
0.2.x public API unchanged. Returning None instead would drop the source
from the mapping entirely, and an empty source list already has a valid
meaning (constant expressions).
apply_catalog now skips empty candidate lists. Without that guard a
CatalogProvider resolving a column by name alone would turn an
unresolved origin straight back into a fabricated Concrete one.
determine_edge_kind correctly identifies any aggregate function call as EdgeKind::ViaAggregation regardless of its arguments, but that kind was only ever attached to the graph as an edge to an ancestor column. COUNT(*) has no column ancestor (`*` is a FunctionArgExpr::Wildcard, not an Expr, so collect_ancestors never visits it), so the correctly computed kind was silently discarded and the output's transform classification fell back to Direct — indistinguishable from a literal constant. Store the defining expression's intrinsic edge kind on the Output node itself, and use it as a fallback in derive_transform whenever no ancestor edge exists to classify from. This generalizes to any zero-ancestor aggregate/conditional/expression, not just COUNT(*).
This was referenced Aug 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This draft exposes uncertainty that downstream lineage consumers need to distinguish from proven physical sources:
The branch also contains the patch-compatible resolver fixes currently submitted in #14, including scope-first structured-field resolution and conservative range-variable row-value handling.
Structured-field contract
Plain dotted references use the longest visible relation binding as their prefix. The next component is the physical top-level column and remaining components are nested fields. Thus
agg.event.qualified_fielddepends onagg.event; it does not create a synthetic quoted relation named"agg.event".BigQuery and PostgreSQL whole-row range variables remain explicitly ambiguous when no catalog, CTE, or derived-table column proves an ordinary same-name column. This avoids fabricating a physical column from
ARRAY_AGG(t ...)while retaining the dependency uncertainty.When an expression is analyzed without visible bindings, ordinary qualified references such as
orders.id,raw.orders.id, andwarehouse.raw.orders.idare preserved as table references instead of being reinterpreted as nested fields.Public API changes
ColumnLineagegainshas_unresolved_starsColumnOrigingainsNamedWildcardandSourceFree, and becomes non-exhaustiveDialectgains DuckDB, Redshift, Trino, Spark, ClickHouse, SQLite, and Microsoft SQL Server variants, and becomes non-exhaustiveThese changes are intended for a minor release.
Stack
This is an integration draft stacked on #14. The patch-compatible resolver work can be reviewed and merged independently before the public API portion is split or rebased.
Current stack head:
b010d8083c60aee977055ea36f8a6807c1d51c62.Verification
cargo fmt --all -- --checkcargo test --workspacecargo clippy --workspace --all-targets --all-features -- -D warnings