feat(orm): cross-backend aggregation engine, fluent SetAggs, sqlite SQL builder refactor - #399
Open
medcl wants to merge 8 commits into
Open
feat(orm): cross-backend aggregation engine, fluent SetAggs, sqlite SQL builder refactor#399medcl wants to merge 8 commits into
medcl wants to merge 8 commits into
Conversation
…QL builder refactor - core/aggregate (new): backend-independent pipeline engine (derivative / sum_bucket / max_bucket / bucket_script / bucket_sort) plus shared result-shaping helpers (zero fill, ordering); backends apply pipelines uniformly via ApplyPipelines before returning from Aggregate - core/orm: SetAggs fluent aggregation setter (name/spec pairs, chainable) alongside SetAggregations; ormtest contract harness - core/api/crud (new): generated CRUD routes with optional MCP exposure - sqlite: extract BuildWhereClause / ExprFor / RangeExprFor / ToEpochSeconds from the query builder; epoch-second range handling, flattened doc support, index maintenance updates - elastic / easysearch: aggregation support wiring (orm_aggregate.go, aggs), cluster loader / hooks + client provider updates, search_decode helpers
…gations Covers pluggable backends (Elasticsearch/Easysearch + embedded SQLite), the QueryBuilder (bool clauses, sorting, pagination, source selection) and the aggregation engine (metrics, buckets, pipelines) introduced in 73b6159.
medcl
changed the title
Driven by a survey of coco's hand-written CRUD modules (connector, datasource, integration, llm, document, assistant), which need more than Permission/MCP/PrepareCreate/GuardDelete to migrate: - ExtraOptions func(action) []api.Option: additional route options (RequireLogin, CORS, sensitive-field masking/labels), appended after the permission and MCP options - IDParam: custom route param name (document uses :doc_id) - CtxDecorate: per-action orm context markers (DirectReadAccess, permission scopes, extra sharing flags) - UpdateMode partial|full|query with ProtectedFields + PrepareUpdate: coco's full-object merge semantics (load, decode over, orm.Update) and the ?replace= switch, on top of the existing partial delta - PostCreate/PostUpdate/PostDelete: best-effort post hooks (cache clears, CORS-origin sync, cascaded cleanups), logged not failed - PostGet: object refinement before the response envelope - PrepareSearch/PostSearch: injected filters (header scoping, path hierarchy), builder excludes, and per-hit response mapping - register: MCP tool only on GET _search (coco convention, avoids duplicate tool names); buildOptions extracted for testability crud_test: FullFlow data namespaced (shared sqlite store), call() strips the query string from :id extraction.
In partial-delta mode the generator only had the id on hand, so post hooks could not read merged fields (coco's integration module refreshes its CORS-origin registry from the updated object). Reload the object after UpdatePartialFields when a PostUpdate hook is configured; full mode already persists the decoded object.
Driven by the remaining coco migrations (connector, datasource, assistant): - SkipActions: actions whose routes stay hand-written when the generator cannot express the semantics (connector/datasource's replace-keeping-system-fields upsert, assistant's cache-first get) - UpdateModeFull now decodes the body into a map and merges over the loaded object, so ProtectedFields are restored from the stored record (the coco handlers wrote such protections to dead variables); PrepareUpdate receives the raw body as the delta in full mode
New workflow orm-integration-test.yml runs the //go:build integration
tagged suites on every PR touching the ORM/aggregate/elastic/sqlite
areas (or the branches main/optsqlite2):
- boots a real Easysearch via get.infini.cloud, waits for health
- go test -tags integration over core/aggregate, modules/sqlite,
modules/elastic (contract, conformance, and the sqlite-vs-elastic
parity suite)
- fixtures bootstrap from ES_ENDPOINT/ES_USERNAME/ES_PASSWORD and skip
gracefully when the cluster is unreachable
Supporting pieces so the suites actually run outside a laptop:
- core/orm/ormtest.SeedSystemCluster: shared env-driven system-cluster
seeding (each test binary registers its own ORM handler first)
- modules/elastic/integration_env_test.go: TestMain wiring for the
elastic suites (sqlite handler + cluster record)
- modules/sqlite/aggregate_parity_integration_test.go: the parity run
(needs both backends; sqlite pkg is the cycle-free meeting point)
Verified end to end against a live Easysearch on this machine: 11/11
conformance subtests, contract suite, and parity all green.
…ntegration suites Four real backend bugs the new CI suites caught when first run against a live cluster (previously the suites were never wired anywhere, so nothing executed them): - search on HTTP >= 400 logged the error but returned a nil-error response, nil-dereferencing every downstream consumer (v0 adapter) - date_histogram key_as_string kept ES's RFC3339-with-millis form (2026-08-13T00:00:00.000Z) instead of the canonical second-precision layout both backends share - numeric bucket keys stayed float64 while the suite (and sqlite) assert int64 epoch millis - percentiles keys kept ES's "50.0"/"100.0" form, so lookups by the bare percent missed (values read as 0) - metric aggregations over zero matched docs reported ES's value:0.0 placeholder as a set value; now cleared when hits.total == 0 - top_hits defaulted size=0, which ES rejects outright (numHits must be > 0); default to top-1 like the suite semantics - conformance/contract fixtures now create indexes with explicit mappings (dynamic mapping turned keyword fields into text, breaking aggregations) and drop leftovers between runs Also carries the orm_test duplicate json tag vet fix from the migration work.
SteveLauC
approved these changes
Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do
Rationale for this change
Standards checklist