Move temporary objects to the catalog#37811
Draft
SangJunBak wants to merge 6 commits into
Draft
Conversation
SangJunBak
force-pushed
the
jun/move-temp-to-catalog
branch
from
July 23, 2026 17:01
ce3ca26 to
f99c06a
Compare
Initial plan generated off the design doc. To be removed.
SangJunBak
force-pushed
the
jun/move-temp-to-catalog
branch
from
July 23, 2026 17:12
f99c06a to
bc91280
Compare
…(SQL-150) Adds StateUpdateKind::Session to the durable catalog, written on session connect and removed on graceful close. All session records are reclaimed when a catalog is opened with write intent, since opening fences out every previous owner (this also covers same-generation restarts). Converts mz_sessions from a coordinator-written builtin table into a BuiltinMaterializedView over the session records in mz_catalog_raw. The old shard is released through the usual builtin schema migration replacement step and finalized by the storage layer. Also adds ephemeral_owner_session: Option<Uuid> to ItemValue and SchemaValue (catalog version 91, backfilled as None), the durable foundation for temporary objects per the durable temporary objects design (doc/developer/design/20260706_sql_150_durable_temporary_objects.md).
Temporary items now write real durable Item rows, marked with ephemeral_owner_session = the creating session's UUID and parented to the temporary schema sentinel id (u0). The in-memory TemporaryItem side-channel is deleted: temp items flow through the normal durable update pipeline. On apply, an ephemeral item is routed into the owning connection's in-memory temporary schema, resolved through a new session-UUID-to-connection map derived from durable session records. Item name uniqueness is scoped by owning session, since every session's temporary schema shares the sentinel schema id. Session close drops the session's temporary items and its session record in one catalog transaction. Opening the catalog with write intent reclaims all ephemeral items alongside stale session records, covering crashes and restarts. Orphaned temp-table shards are finalized by the existing storage metadata reconciliation. BootstrapStateUpdateKind is deleted: it existed only to exclude the TemporaryItem variant, and memory StateUpdateKind now serves both uses. Verified manually: temp create/insert/query/rename/cascade-drop, same temp table name in two concurrent sessions, graceful-close cleanup, kill -9 + restart reclamation, DISCARD TEMP, temporary_objects.slt.
With temporary items durable in the catalog shard, mz_tables and mz_views can be derived from mz_internal.mz_catalog_raw instead of being written by the coordinator. Both show every item including temporary ones, matching the previous builtin tables. Temporary rows keep the temporary schema sentinel "0" in schema_id. Builtin tables and views are reported through two new generated constant views, mz_internal.mz_builtin_tables and mz_internal.mz_builtin_views, following the mz_builtin_materialized_views pattern. mz_builtin_views lists every builtin view except itself, since its definition cannot contain itself, so that one (new) view is absent from mz_views and relations derived from it. parse_catalog_create_sql now exposes 'definition' for views and 'source_id' for tables created from sources. The old mz_tables and mz_views shards are released through builtin schema migration replacement steps and finalized by the storage layer, like the other builtin table conversions. Verified manually: boot, user/temp tables and views appear with correct schema ids and definitions, pg_tables/pg_views joins, builtin arms populated (191 builtin views = all but mz_builtin_views). SLT: information_schema_tables, oid, temporary_objects.
Session connects and disconnects each write the durable catalog, and previously did so with one catalog transaction per session, serialized on the coordinator loop: one timestamp oracle round-trip plus one compare-and-append against the catalog shard per connection. A reconnect storm would queue real DDL behind thousands of such commits. Batch them instead. Session ops land in a pending buffer and a FlushSessionOps message, scheduled only when the buffer transitions from empty, commits everything pending in a single catalog transaction. Because the coordinator loop is single-threaded, ops that arrive while a flush is in flight accumulate for the next one, so batch size self-tunes to the commit latency. Startup completion is two-phase: handle_startup enqueues the CreateSession op together with everything needed to finish connection setup, and the flush completes each startup (active_conns insertion, metrics, StartupResponse) after the shared commit lands. Awaiting the commit inline would deadlock the loop on its own flush message. The response is still only sent after the session record is durable and applied, so temporary object creation can never race its own session record. Session close enqueues a fire-and-forget DropSession, except when the session owns temporary items, which keep dropping atomically with the session record in one inline transaction. Also fixes an active_sessions metric leak: the gauge was incremented before the session write and never decremented if the write failed. Verified manually: 50-connection burst all durable and reclaimed; debug logs show 60 session ops coalescing into 9 commits with batches up to 15; solo connects commit alone. SLT: temporary_objects, transactions.
SangJunBak
force-pushed
the
jun/move-temp-to-catalog
branch
2 times, most recently
from
July 23, 2026 18:43
f0bae71 to
d47a349
Compare
Batched session commits previously flushed as soon as the coordinator got to them, so under sustained connection churn the commit rate converged to one catalog compare-and-append per commit latency, on the order of 100-200 per second. Bound it with the session_op_flush_interval dyncfg (default 50ms): a flush scheduled within the interval of the previous one is delayed until the interval has passed, capping catalog-shard writes from connection churn at one commit per interval. The delay applies only under churn: the first op after an idle period still flushes immediately, so isolated connects keep their latency. Larger batches also stay well under persist's inline-write threshold, so these commits continue to skip blob storage. Verified manually: with the 50ms floor a 30-connection burst commits 61 session ops in 5 catalog transactions (batches up to 28, down from 9 transactions with batches up to 15); solo connects flush immediately; the interval is tunable via ALTER SYSTEM. SLT: temporary_objects.
SangJunBak
force-pushed
the
jun/move-temp-to-catalog
branch
from
July 23, 2026 20:12
d47a349 to
20a5da3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Batch session benchmark https://buildkite.com/materialize/nightly/builds/17518#019f904d-099f-4f89-889d-be35a5fb3861
Motivation
Why does this change exist? Link to a GitHub issue, design doc, Slack
thread, or explain the problem in a sentence or two. A reviewer who has
no context should understand why after reading this section.
If this implements or addresses an existing issue, it's enough to link to that:
Closes
Fixes
etc.
Description
What does this PR actually do? Focus on the approach and any non-obvious
decisions. The diff shows the code --- use this space to explain what the
diff can't tell a reviewer.
Verification
How do you know this change is correct? Describe new or existing automated
tests, or manual steps you took.