Skip to content

feat(python/sedonadb-geopandas): add sjoin, dissolve, column assignment, and arithmetic - #1142

Draft
jiayuasu wants to merge 10 commits into
apache:mainfrom
jiayuasu:feature/geopandas-sjoin-dissolve
Draft

feat(python/sedonadb-geopandas): add sjoin, dissolve, column assignment, and arithmetic#1142
jiayuasu wants to merge 10 commits into
apache:mainfrom
jiayuasu:feature/geopandas-sjoin-dissolve

Conversation

@jiayuasu

Copy link
Copy Markdown
Member

Extends the experimental sedonadb-geopandas package (#1052) with the operations a real pipeline needs: a spatial join, a dissolve, column assignment, and arithmetic on columns.

gdf = sgpd.from_geopandas(points)
gdf["density"] = gdf["pop"] / gdf["area"]          # assign a computed column
joined = gdf.sjoin(regions, predicate="within")    # spatial join
zones = joined.dissolve(by="region")               # group and union geometry

What is added

  • sjoin(other, how, predicate, lsuffix, rsuffix, distance)how of inner, left, or right, over the intersects, within, contains, touches, crosses, overlaps, covers, covered_by, and dwithin predicates. The predicate reads left-relative-to-right as in GeoPandas. The result carries exactly one geometry column — the left frame's, or the right frame's for how="right", matching which side GeoPandas keeps — and column names occurring on both sides get the lsuffix/rsuffix treatment.
  • dissolve(by, aggfunc="first") — unions each group's geometry via ST_Union_Agg and carries the remaining columns. by=None dissolves everything into one row.
  • __setitem__gdf["x"] = ... from a Series of the same frame, a SedonaDB expression, or a scalar to broadcast.
  • Arithmetic on Series+, -, *, /, unary -, and the reflected forms.
  • Series.expr — the underlying expression, as an escape hatch for anything the wrapper does not cover (series.expr.funcs.st_point(other.expr)), assignable back onto a frame.

Two semantics worth review attention

/ follows pandas, not SQL. The engine truncates when dividing integers, so v / 2 over an integer column would return floored values where pandas returns 0.5, 1.5, and so on. Since the point of this package is that GeoPandas code keeps working, the numerator is cast to double first. // is deliberately not implemented rather than mapped onto SQL division, which truncates toward zero where Python floors — a subtly wrong operator seemed worse than a missing one.

Assignment rebinds the frame. The underlying frame is immutable, so gdf["x"] = ... replaces the frame this object points at. A Series captured before that assignment therefore belongs to the previous frame, and combining it with a later read raises rather than silently mixing two frames. The error message says so explicitly.

Deviations from GeoPandas, all documented in the README

  • sjoin produces no index_left/index_right column, and dissolve leaves the group keys as ordinary columns rather than moving them into the index. Both follow from there being no row index.
  • Columns cannot be combined across two frames; without row alignment that would be a silent wrong answer, so it raises and suggests joining first.

Verification

Checked against GeoPandas rather than only for self-consistency:

  • sjoin column shapes match exactly for all three how values (after dropping the index_* columns), as do the active geometry column and the row count.
  • sjoin row pairings match.
  • dissolve areas and first-aggregated column values match ([7.0, 4.0] and [1, 3]).
  • Arithmetic matches, including the reflected forms.

46 tests pass locally and in a clean virtual environment installing sedonadb from the nightly index — the environment shape that caught the missing dependency in #1134. ruff format and ruff check are clean.

Deferred

The GeoSeries accessor batch (geom_type, x/y, bounds, envelope, boundary, to_wkt, and friends) is left for a follow-up. One compatibility wrinkle to settle there: ST_GeometryType returns ST_Polygon where GeoPandas' geom_type returns Polygon, so that mapping needs a decision rather than a quick pass-through.

@github-actions
github-actions Bot requested a review from zhangfengcdt August 10, 2026 06:19
…nt, and arithmetic

Extends the experimental GeoPandas-compatible API with the operations a real
pipeline needs, matching GeoPandas semantics where they are well defined:

- `sjoin(other, how, predicate, lsuffix, rsuffix, distance)` for inner/left/right
  joins on `intersects`, `within`, `contains`, `touches`, `crosses`, `overlaps`,
  `covers`, `covered_by`, and `dwithin`. Keeps one geometry column — the left
  frame's, or the right frame's for `how="right"`, matching which side GeoPandas
  keeps — and suffixes column names that occur on both sides.
- `dissolve(by, aggfunc="first")`, unioning each group's geometry and carrying the
  remaining columns.
- `__setitem__`, so a computed column can be assigned back onto a frame.
- Arithmetic on `Series`, including the reflected forms.
- `Series.expr`, exposing the underlying expression as an escape hatch.

Two semantics worth calling out. `/` is true division as in pandas: the engine
follows SQL, where dividing integers truncates, so the numerator is cast to
double first rather than silently returning floored results; `//` is left
unimplemented because SQL division truncates toward zero where Python floors.
And because assignment rebinds the frame, a `Series` captured beforehand is stale
and raises if combined with a later read instead of mixing two frames.

Verified against GeoPandas: identical column shapes for all three `how` values
(minus the `index_*` columns, which need a row index), identical row pairings,
and identical dissolve areas and first-aggregated values.
`DataFrame.mutate` is not present in tagged 0.4.0, but the dependency floor
allowed it, so `to_crs()` and column assignment would fail with an
AttributeError in an otherwise valid installation. mutate first ships in 0.5.0.

The floor is written as the prerelease `0.5.0a0` rather than `0.5.0` so the
nightly wheels this is developed against (`0.5.0aN`, which PEP 440 orders below
`0.5.0`) still satisfy it.

Note the exposure predates column assignment: `to_crs()` has used mutate since
the package was added.
…missing keys

ST_Union_Agg only initializes for polygonal input, so dissolving a group of
points or linestrings returned NULL geometry and a mixed group could silently
discard geometry (apache#1093). Collect the group and union it
afterwards instead, which is geometry-general.

The union is a separate projection rather than part of the aggregate, since a
scalar function wrapped around an aggregate is not a valid aggregate expression.

Also adds `dropna=True`, matching GeoPandas, which drops rows whose group key is
missing.

Verified against GeoPandas: unioned geometry is now equal, and of the same type,
for point, linestring, and polygon groups, and the previously covered areas and
first-aggregated values are unchanged.

Two differences are documented rather than fixed, both consequences of lazy
aggregation: a NaN loaded from pandas is an ordinary value to the engine where
GeoPandas' "first" skips it, and dissolving an empty frame without a key yields
one all-null row because that is what a grouping-free SQL aggregate returns.
Three problems in assignment, all of which produced wrong data or wrong state
rather than an error.

Bare expressions are no longer accepted, and `Series.expr` is withdrawn. An
expression records no origin, so `left["copied"] = right["v"].expr` resolved the
column reference against the destination frame and silently wrote the
destination's own values. There is no way to validate the origin of a bare
expression after the fact, so the safe surface is a Series (whose frame is
checked) or a literal (which holds a value, not a reference).

Scalars are now classified properly. Testing for `__array__` was wrong in both
directions: a list has no `__array__` and was broadcast whole into every row,
while a NumPy scalar has one and was rejected. Sequences are now rejected
explicitly, and array-likes are judged by dimensionality.

The active geometry column is revalidated after assignment. Replacing it with a
number left the name pointing at an integer column, so `.geometry` still returned
a GeoSeries and `.area` failed later with a kernel error.
…me collisions

Two problems in sjoin.

`predicate="dwithin"` missed pairs GeoPandas matches. ST_Distance reports the
endpoint gap rather than zero for geometries that properly cross without sharing
a vertex, so two crossing linestrings looked 1.41 apart and failed a
distance-0 bound. Since anything that crosses also intersects, and intersecting
means a distance of zero, the predicate is now the union of ST_DWithin and
ST_Intersects, which covers that case and leaves the others unchanged. The
underlying distance behaviour is an engine issue, not something this layer can
fix.

Name collisions involving the retained geometry column raised a planning error.
Collisions were computed before deciding which side's geometry is discarded, so a
left geometry named `geom` joined against a frame with an ordinary `geom` column
projected two columns called `geom`. Collisions are now computed over the columns
actually emitted, and — following GeoPandas — the retained geometry keeps its name
while only the opposite side's column is suffixed; ordinary collisions still
suffix both sides.

Adds regression tests for every finding in this pass, including the previously
failing dissolve geometry types, the sequence and NumPy-scalar classification, the
geometry-state revalidation, and both sjoin cases. Verified against GeoPandas for
column shapes and row counts.
…rface

Removes the reference to the withdrawn expression escape hatch, and records the
dissolve aggregation differences (pandas NaN, and empty input without a key).
… provenance gaps

Nine issues from review, the first of which reverses an earlier decision.

The `dwithin` workaround is withdrawn. Composing `ST_DWithin OR ST_Intersects`
returned the right rows but is not recognized by the spatial-join optimizer:
`explain()` shows NestedLoopJoinExec where the single predicate produces
SpatialJoinExec, making every dwithin join quadratic. That costs far more than the
one case it fixed — crossing linestrings, which ST_Distance mis-measures
(apache#1156) — so the predicate is a single spatial call again and the
deviation is documented and covered by a test that will fail when the engine is
fixed. A second test asserts the plan stays a spatial join. Reverting also restores
GeoPandas' behaviour for negative and NaN bounds, which the fallback had broken.

Operators now refuse a bare expression, as assignment already did; previously
`left["v"] + raw_frame["v"]` resolved the right-hand reference against the left
frame and silently used the left frame's values.

Boolean indexing now checks the mask's source frame. Assignment rebinds the
frame, so a mask captured beforehand was quietly reused against the new one,
working by accident while the referenced column still existed.

sjoin rejects suffix-generated duplicates: left columns `v` and `v_left` against a
right `v` both wanted to be `v_left`, which failed deep in the planner. The error
now names the collision and suggests different suffixes.

dissolve's dropna covers IEEE NaN as well as SQL null, so a float key read from
pandas no longer forms its own group.

Assigning a geometry scalar or None over an existing geometry column keeps that
column's CRS, and None additionally stays a typed geometry column, both matching
GeoPandas. Assigning a number over a geometry column still clears it.

Corrects the documentation for `aggfunc="first"`: it is an unordered aggregate that
returns some group member and does not skip missing values, so the earlier claim
that it skips SQL nulls was wrong. Mixed 2D/3D dissolve groups are documented as
unsupported rather than silently failing an obscure way.
… handling

Five issues from review.

Floating-point columns are detected from the Arrow datatype rather than its
rendered string. `list<item: double>` and `struct<x: double>` contain "double"
while being nothing isnan() can be applied to, so dissolving by such a key failed
at planning time with dropna at its default.

Every scalar geometry assignment now takes the CRS-preserving path. Previously
only a plain geometry or None did: a `lit(geometry)` kept geometry typing but lost
the CRS, `lit(None)` dropped the typing altogether, NaN turned the column into
float64, and pandas.NA raised. Literals are unwrapped and rebuilt on the frame's
context (a bare lit() has none, so functions cannot be applied to it), and the
missing-value sentinels None, NaN and pandas.NA are all treated as missing
geometry, matching GeoPandas.

A dissolve group whose geometries are all null now yields an empty geometry
collection rather than null, as GeoPandas does; the two differ for isna, is_empty,
predicates and serialization. Coalescing loses the geometry type, so the result is
re-typed and the source CRS re-applied.

Operators accept zero-dimensional NumPy scalars, which they rejected while
assignment accepted them. Both now share one scalar classification so they cannot
drift apart again.

Corrects the comment explaining why the composed dwithin predicate was reverted:
besides disabling spatial planning, OR-ing ST_Intersects in also broke the bound's
semantics by matching coincident geometries at negative and NaN distances. Both
reasons are recorded so neither is reintroduced.
…oPandas

The engine fix for ST_Distance over properly crossing linestrings landed
(apache#1164, closing apache#1156), so the single ST_DWithin predicate is
now correct on its own: the pair that was documented as a deviation matches
exactly as GeoPandas does, with no workaround and no change to the plan shape.

Flips the limitation test into a parity test, and removes the deviation notes
from the README and the sjoin comment. The plan-shape test stays: the reason to
keep the predicate un-composed (spatial-join planning) outlives the bug that
tempted the composition.
@jiayuasu
jiayuasu force-pushed the feature/geopandas-sjoin-dissolve branch from e070b06 to 90c9088 Compare August 14, 2026 05:33
…ral CRS, narrow the division cast

Six layer fixes from review; two further findings are engine-side and documented.

sjoin's dwithin distance must now be a plain number. The predicate is built
against re-aliased copies of both frames, so a Series or expression as a distance
would resolve by name inside the join rather than against its own frame; both are
rejected with a focused error, while NumPy numeric scalars are accepted.

A geometry literal that carries its own CRS keeps it. Previously assigning an
EPSG:4326 GeoSeries literal over an EPSG:3857 column stamped 3857 over unchanged
coordinates — a silent relabeling. The destination CRS is now inherited only by
genuinely CRS-less geometry.

The true-division cast is restricted to integer-typed expressions, decided from
the projected schema (a plan build, not an execution). Casting everything to
double fixed integer truncation but corrupted decimals (1.23/0.1 became
12.299999...) and turned durations into float nanosecond counts; a decimal
division now stays exact.

Series opts out of NumPy ufunc dispatch, so `np.array([...]) + series` reaches
the reflected operator whole and is rejected, instead of NumPy broadcasting
element-by-element into an object array of lazy Series.

Scalars are normalized where they are used, not just classified: a 0-d array is
unwrapped to its Python value (it passed classification but failed literal
construction) and pandas.NA becomes SQL null for ordinary columns. NaN stays a
float value, as in pandas.

_is_floating unwraps dictionary encoding, so a dictionary<double> dissolve key
gets NaN handling; isnan() accepts dictionary columns, verified.

Corrects the empty-dissolve wording: the result is one row with an empty geometry
collection and null attribute values, not an all-null row.

Two verified engine-side issues are documented in the README rather than worked
around: ST_Touches misses a line whose interior passes exactly through a polygon
corner and ST_Within wrongly matches a line lying on a hole boundary; and an
outer spatial join whose preserved side is empty fails with an internal error
("bitmap for visited left side is not created") where GeoPandas returns an empty
frame.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant