feat(python/sedonadb-geopandas): add sjoin, dissolve, column assignment, and arithmetic - #1142
Draft
jiayuasu wants to merge 10 commits into
Draft
feat(python/sedonadb-geopandas): add sjoin, dissolve, column assignment, and arithmetic#1142jiayuasu wants to merge 10 commits into
jiayuasu wants to merge 10 commits into
Conversation
…nt, and arithmetic Extends the experimental GeoPandas-compatible API with the operations a real pipeline needs, matching GeoPandas semantics where they are well defined: - `sjoin(other, how, predicate, lsuffix, rsuffix, distance)` for inner/left/right joins on `intersects`, `within`, `contains`, `touches`, `crosses`, `overlaps`, `covers`, `covered_by`, and `dwithin`. Keeps one geometry column — the left frame's, or the right frame's for `how="right"`, matching which side GeoPandas keeps — and suffixes column names that occur on both sides. - `dissolve(by, aggfunc="first")`, unioning each group's geometry and carrying the remaining columns. - `__setitem__`, so a computed column can be assigned back onto a frame. - Arithmetic on `Series`, including the reflected forms. - `Series.expr`, exposing the underlying expression as an escape hatch. Two semantics worth calling out. `/` is true division as in pandas: the engine follows SQL, where dividing integers truncates, so the numerator is cast to double first rather than silently returning floored results; `//` is left unimplemented because SQL division truncates toward zero where Python floors. And because assignment rebinds the frame, a `Series` captured beforehand is stale and raises if combined with a later read instead of mixing two frames. Verified against GeoPandas: identical column shapes for all three `how` values (minus the `index_*` columns, which need a row index), identical row pairings, and identical dissolve areas and first-aggregated values.
`DataFrame.mutate` is not present in tagged 0.4.0, but the dependency floor allowed it, so `to_crs()` and column assignment would fail with an AttributeError in an otherwise valid installation. mutate first ships in 0.5.0. The floor is written as the prerelease `0.5.0a0` rather than `0.5.0` so the nightly wheels this is developed against (`0.5.0aN`, which PEP 440 orders below `0.5.0`) still satisfy it. Note the exposure predates column assignment: `to_crs()` has used mutate since the package was added.
…missing keys ST_Union_Agg only initializes for polygonal input, so dissolving a group of points or linestrings returned NULL geometry and a mixed group could silently discard geometry (apache#1093). Collect the group and union it afterwards instead, which is geometry-general. The union is a separate projection rather than part of the aggregate, since a scalar function wrapped around an aggregate is not a valid aggregate expression. Also adds `dropna=True`, matching GeoPandas, which drops rows whose group key is missing. Verified against GeoPandas: unioned geometry is now equal, and of the same type, for point, linestring, and polygon groups, and the previously covered areas and first-aggregated values are unchanged. Two differences are documented rather than fixed, both consequences of lazy aggregation: a NaN loaded from pandas is an ordinary value to the engine where GeoPandas' "first" skips it, and dissolving an empty frame without a key yields one all-null row because that is what a grouping-free SQL aggregate returns.
Three problems in assignment, all of which produced wrong data or wrong state rather than an error. Bare expressions are no longer accepted, and `Series.expr` is withdrawn. An expression records no origin, so `left["copied"] = right["v"].expr` resolved the column reference against the destination frame and silently wrote the destination's own values. There is no way to validate the origin of a bare expression after the fact, so the safe surface is a Series (whose frame is checked) or a literal (which holds a value, not a reference). Scalars are now classified properly. Testing for `__array__` was wrong in both directions: a list has no `__array__` and was broadcast whole into every row, while a NumPy scalar has one and was rejected. Sequences are now rejected explicitly, and array-likes are judged by dimensionality. The active geometry column is revalidated after assignment. Replacing it with a number left the name pointing at an integer column, so `.geometry` still returned a GeoSeries and `.area` failed later with a kernel error.
…me collisions Two problems in sjoin. `predicate="dwithin"` missed pairs GeoPandas matches. ST_Distance reports the endpoint gap rather than zero for geometries that properly cross without sharing a vertex, so two crossing linestrings looked 1.41 apart and failed a distance-0 bound. Since anything that crosses also intersects, and intersecting means a distance of zero, the predicate is now the union of ST_DWithin and ST_Intersects, which covers that case and leaves the others unchanged. The underlying distance behaviour is an engine issue, not something this layer can fix. Name collisions involving the retained geometry column raised a planning error. Collisions were computed before deciding which side's geometry is discarded, so a left geometry named `geom` joined against a frame with an ordinary `geom` column projected two columns called `geom`. Collisions are now computed over the columns actually emitted, and — following GeoPandas — the retained geometry keeps its name while only the opposite side's column is suffixed; ordinary collisions still suffix both sides. Adds regression tests for every finding in this pass, including the previously failing dissolve geometry types, the sequence and NumPy-scalar classification, the geometry-state revalidation, and both sjoin cases. Verified against GeoPandas for column shapes and row counts.
…rface Removes the reference to the withdrawn expression escape hatch, and records the dissolve aggregation differences (pandas NaN, and empty input without a key).
… provenance gaps Nine issues from review, the first of which reverses an earlier decision. The `dwithin` workaround is withdrawn. Composing `ST_DWithin OR ST_Intersects` returned the right rows but is not recognized by the spatial-join optimizer: `explain()` shows NestedLoopJoinExec where the single predicate produces SpatialJoinExec, making every dwithin join quadratic. That costs far more than the one case it fixed — crossing linestrings, which ST_Distance mis-measures (apache#1156) — so the predicate is a single spatial call again and the deviation is documented and covered by a test that will fail when the engine is fixed. A second test asserts the plan stays a spatial join. Reverting also restores GeoPandas' behaviour for negative and NaN bounds, which the fallback had broken. Operators now refuse a bare expression, as assignment already did; previously `left["v"] + raw_frame["v"]` resolved the right-hand reference against the left frame and silently used the left frame's values. Boolean indexing now checks the mask's source frame. Assignment rebinds the frame, so a mask captured beforehand was quietly reused against the new one, working by accident while the referenced column still existed. sjoin rejects suffix-generated duplicates: left columns `v` and `v_left` against a right `v` both wanted to be `v_left`, which failed deep in the planner. The error now names the collision and suggests different suffixes. dissolve's dropna covers IEEE NaN as well as SQL null, so a float key read from pandas no longer forms its own group. Assigning a geometry scalar or None over an existing geometry column keeps that column's CRS, and None additionally stays a typed geometry column, both matching GeoPandas. Assigning a number over a geometry column still clears it. Corrects the documentation for `aggfunc="first"`: it is an unordered aggregate that returns some group member and does not skip missing values, so the earlier claim that it skips SQL nulls was wrong. Mixed 2D/3D dissolve groups are documented as unsupported rather than silently failing an obscure way.
… handling Five issues from review. Floating-point columns are detected from the Arrow datatype rather than its rendered string. `list<item: double>` and `struct<x: double>` contain "double" while being nothing isnan() can be applied to, so dissolving by such a key failed at planning time with dropna at its default. Every scalar geometry assignment now takes the CRS-preserving path. Previously only a plain geometry or None did: a `lit(geometry)` kept geometry typing but lost the CRS, `lit(None)` dropped the typing altogether, NaN turned the column into float64, and pandas.NA raised. Literals are unwrapped and rebuilt on the frame's context (a bare lit() has none, so functions cannot be applied to it), and the missing-value sentinels None, NaN and pandas.NA are all treated as missing geometry, matching GeoPandas. A dissolve group whose geometries are all null now yields an empty geometry collection rather than null, as GeoPandas does; the two differ for isna, is_empty, predicates and serialization. Coalescing loses the geometry type, so the result is re-typed and the source CRS re-applied. Operators accept zero-dimensional NumPy scalars, which they rejected while assignment accepted them. Both now share one scalar classification so they cannot drift apart again. Corrects the comment explaining why the composed dwithin predicate was reverted: besides disabling spatial planning, OR-ing ST_Intersects in also broke the bound's semantics by matching coincident geometries at negative and NaN distances. Both reasons are recorded so neither is reintroduced.
…oPandas The engine fix for ST_Distance over properly crossing linestrings landed (apache#1164, closing apache#1156), so the single ST_DWithin predicate is now correct on its own: the pair that was documented as a deviation matches exactly as GeoPandas does, with no workaround and no change to the plan shape. Flips the limitation test into a parity test, and removes the deviation notes from the README and the sjoin comment. The plan-shape test stays: the reason to keep the predicate un-composed (spatial-join planning) outlives the bug that tempted the composition.
jiayuasu
force-pushed
the
feature/geopandas-sjoin-dissolve
branch
from
August 14, 2026 05:33
e070b06 to
90c9088
Compare
…ral CRS, narrow the division cast
Six layer fixes from review; two further findings are engine-side and documented.
sjoin's dwithin distance must now be a plain number. The predicate is built
against re-aliased copies of both frames, so a Series or expression as a distance
would resolve by name inside the join rather than against its own frame; both are
rejected with a focused error, while NumPy numeric scalars are accepted.
A geometry literal that carries its own CRS keeps it. Previously assigning an
EPSG:4326 GeoSeries literal over an EPSG:3857 column stamped 3857 over unchanged
coordinates — a silent relabeling. The destination CRS is now inherited only by
genuinely CRS-less geometry.
The true-division cast is restricted to integer-typed expressions, decided from
the projected schema (a plan build, not an execution). Casting everything to
double fixed integer truncation but corrupted decimals (1.23/0.1 became
12.299999...) and turned durations into float nanosecond counts; a decimal
division now stays exact.
Series opts out of NumPy ufunc dispatch, so `np.array([...]) + series` reaches
the reflected operator whole and is rejected, instead of NumPy broadcasting
element-by-element into an object array of lazy Series.
Scalars are normalized where they are used, not just classified: a 0-d array is
unwrapped to its Python value (it passed classification but failed literal
construction) and pandas.NA becomes SQL null for ordinary columns. NaN stays a
float value, as in pandas.
_is_floating unwraps dictionary encoding, so a dictionary<double> dissolve key
gets NaN handling; isnan() accepts dictionary columns, verified.
Corrects the empty-dissolve wording: the result is one row with an empty geometry
collection and null attribute values, not an all-null row.
Two verified engine-side issues are documented in the README rather than worked
around: ST_Touches misses a line whose interior passes exactly through a polygon
corner and ST_Within wrongly matches a line lying on a hole boundary; and an
outer spatial join whose preserved side is empty fails with an internal error
("bitmap for visited left side is not created") where GeoPandas returns an empty
frame.
This was referenced Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extends the experimental
sedonadb-geopandaspackage (#1052) with the operations a real pipeline needs: a spatial join, a dissolve, column assignment, and arithmetic on columns.What is added
sjoin(other, how, predicate, lsuffix, rsuffix, distance)—howofinner,left, orright, over theintersects,within,contains,touches,crosses,overlaps,covers,covered_by, anddwithinpredicates. The predicate reads left-relative-to-right as in GeoPandas. The result carries exactly one geometry column — the left frame's, or the right frame's forhow="right", matching which side GeoPandas keeps — and column names occurring on both sides get thelsuffix/rsuffixtreatment.dissolve(by, aggfunc="first")— unions each group's geometry viaST_Union_Aggand carries the remaining columns.by=Nonedissolves everything into one row.__setitem__—gdf["x"] = ...from aSeriesof the same frame, a SedonaDB expression, or a scalar to broadcast.Series—+,-,*,/, unary-, and the reflected forms.Series.expr— the underlying expression, as an escape hatch for anything the wrapper does not cover (series.expr.funcs.st_point(other.expr)), assignable back onto a frame.Two semantics worth review attention
/follows pandas, not SQL. The engine truncates when dividing integers, sov / 2over an integer column would return floored values where pandas returns0.5,1.5, and so on. Since the point of this package is that GeoPandas code keeps working, the numerator is cast to double first.//is deliberately not implemented rather than mapped onto SQL division, which truncates toward zero where Python floors — a subtly wrong operator seemed worse than a missing one.Assignment rebinds the frame. The underlying frame is immutable, so
gdf["x"] = ...replaces the frame this object points at. ASeriescaptured before that assignment therefore belongs to the previous frame, and combining it with a later read raises rather than silently mixing two frames. The error message says so explicitly.Deviations from GeoPandas, all documented in the README
sjoinproduces noindex_left/index_rightcolumn, anddissolveleaves the group keys as ordinary columns rather than moving them into the index. Both follow from there being no row index.Verification
Checked against GeoPandas rather than only for self-consistency:
sjoincolumn shapes match exactly for all threehowvalues (after dropping theindex_*columns), as do the active geometry column and the row count.sjoinrow pairings match.dissolveareas and first-aggregated column values match ([7.0, 4.0]and[1, 3]).46 tests pass locally and in a clean virtual environment installing
sedonadbfrom the nightly index — the environment shape that caught the missing dependency in #1134.ruff formatandruff checkare clean.Deferred
The
GeoSeriesaccessor batch (geom_type,x/y,bounds,envelope,boundary,to_wkt, and friends) is left for a follow-up. One compatibility wrinkle to settle there:ST_GeometryTypereturnsST_Polygonwhere GeoPandas'geom_typereturnsPolygon, so that mapping needs a decision rather than a quick pass-through.