docs: document cluster autoscaling and graceful resizing by maheshwarip · Pull Request #37714 · MaterializeInc/materialize

maheshwarip · 2026-07-17T16:25:54Z

Docs for autoscaling!

Document two cluster-controller capabilities shipping in v26.34: - Hydration-burst autoscaling via the AUTO SCALING STRATEGY = (ON HYDRATION (HYDRATION SIZE = ..., LINGER DURATION = ...)) cluster option, which bursts a cluster to a larger size while it has un-hydrated objects to speed up hydration, then tears the burst replica down. Labeled public preview. - Graceful, asynchronous resizing: after v26.34 a bare ALTER CLUSTER SET (SIZE = ...) returns immediately and reconfigures in the background (provision new replicas, hydrate, cut over, retire old), gated behind {{< if-released "v26.34" >}} so the current downtime wording stays visible until release. Shared autoscaling prose lives in shared-content/cluster-hydration-burst.md and is included in both the CREATE CLUSTER and ALTER CLUSTER reference pages. Adds a Monitoring a resize section to ALTER CLUSTER covering the SHOW CLUSTERS activity column and the mz_cluster_reconfigurations, mz_cluster_auto_scaling_strategies, and mz_hydration_statuses catalog views. Documents the new SHOW CLUSTERS activity column and adds a brief autoscaling pointer to the clusters concept page. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Explain that provisioning the hydration-burst replica needs spare compute capacity (for example, available Kubernetes nodes on Materialize Self-Managed), that the burst is best-effort and never blocks the cluster (the steady-size replicas still come up and hydrate when resources are available), and that the burst replica is always cleaned up even if it was never provisioned. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

aljoscha

looks good, but I'd defer to @kay-kim for the style and vibe and whatnot, had some inline comments

Clarify that WAIT UNTIL READY and WAIT FOR no longer block the session, they only set the timeout and on-timeout action for the background resize. Drop the cutover framing, the old replicas are simply retired once the new ones are hydrated. Fold the activity column into the version-gated SHOW CLUSTERS output table. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Gate all cluster autoscaling documentation behind the v26.34 release, matching the graceful-resize content, since both ship in v26.34. Note that AUTO SCALING STRATEGY is only available on managed clusters, rename the "bursting" section to "autoscaling", document the ON HYDRATION sub-options in the syntax reference, add a catalog-view inspection example, and note the blue/green deployment speedup. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add a note clarifying that graceful, no-downtime resizing became the default in v26.34, and that earlier versions could incur downtime and required WAIT UNTIL READY for zero-downtime resizing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

A graceful resize provisions new replicas, waits for them to hydrate, then retires the old ones. There is no distinct cutover step. Align create-cluster.md with the corrected wording already used in alter-cluster.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Autoscaling and graceful cluster resizing have been released, so the if-released/if-unreleased shortcodes that gated this content on v26.34 are no longer needed. Unwrap the if-released blocks to keep their content as plain documentation, and remove the if-unreleased blocks that described the old pre-release behavior. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

kay-kim · 2026-07-21T20:19:07Z

+
+#### Speed up hydration by autoscaling to a larger size
+
+{{< include-md file="shared-content/cluster-hydration-burst.md" >}}


So, we're including this w. a CREATE CLUSTER example in an ALTER CLUSTER page. We might want to maybe tie it a little more gracefully to the resizing since we're putting here under resize between zero-downtime and monitoring sections.

So .. they way we currently present it ...

ALTER CLUSTER SET SIZE...

ALTER CLUSTER SET (AUTO SCALING ..) ...
If run in that sequence, what happens?

Done! Added a lead-in that frames autoscaling as a standing policy you can set via CREATE CLUSTER or add via ALTER CLUSTER, and links to the Configure autoscaling example. Left the exact SET (SIZE) + SET (AUTO SCALING) sequencing unstated for now, pending confirmation from @aljoscha

Sounds good. I just wanted to make sure we're bubbling up the option to set the auto scaling early enough before they first alter cluster to change size ... and then if they happened to glanced below, see the autoscaling part and realize they probably wanted auto scaling ... they probably will want an earlier signal

[Aside] Also, I think I'm in the middle of graceful resizing of the source_cluster (actually, going from 100cc -> 25cc with autoscaling to 50cc ) ... and just had some couple of questions about what I am seeing (not specifically related to this PR but might as well as):

So, can see we have the second replica per graceful resizing

Now, the objects screen shows that my sources are running

But, the sources show "Snapshotting" (is this expected?)

Because the query on mz_hydration _statuses show hydrated = true

This is probably a console bug; not a blocker but I'll file it in the console queue

Also, not even sure if bug because I think this is how we do graceful resizing also, but just double-checking: other than via the query (or that the sizes differ) ... there isn't another way to know which is the burst replica, yes?

Not to my knowledge

Rework the autoscaling docs to consistently describe the strategy as provisioning a separate burst replica rather than resizing the whole cluster, per review feedback. - Reframe AUTO SCALING STRATEGY descriptions (CREATE/ALTER syntax, concepts) around provisioning a burst replica. - Fold the v26.34 downtime note into prose, define "bare" resize, and keep pre-v26.34 coverage for self-managed users on CREATE/ALTER. - Clean up the mz_cluster_auto_scaling_strategies example: rename bursting_at to inflight_burst_size, add linger_seconds, clarify state as burst details. - Document the ON HYDRATION options in the CREATE CLUSTER syntax for parity. - Add a CREATE/ALTER lead-in tying autoscaling to the resize section. - Migrate the shared-content hydration-burst include to headless content. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Clarify that the burst replica retires once a single steady-size replica catches up (not all of them), reframe the hydration-burst copy around billing and the ON HYDRATION sub-strategy, and cross-link SHOW CLUSTERS from the autoscaling monitoring section. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Consolidate the hydration-burst headless content per Kay's review: fold the redundant ON HYDRATION paragraph into the intro, reframe the blue/green usefulness, add a lead-in before the example, drop the contradictory cleanup sentence, and broaden the hydrating-objects list to include Kafka upsert sources. Also reword the autoscaling lead-in on the clusters concept page. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

maheshwarip · 2026-07-23T14:09:47Z

@kay-kim I made a few tweaks to the language - lmk if this is good to go! If there are no major issues, I vote we approve & merge in

kay-kim · 2026-07-23T17:44:15Z

+
+  ```mzsql
+  ALTER CLUSTER c1
+  SET (SIZE = '100cc') WITH (WAIT UNTIL READY (TIMEOUT = '10m', ON TIMEOUT = 'ROLLBACK'));


Should we maybe remove the ON TIMEOUT = 'ROLLBACK' option? that way even harder for people to use COMMIT.

Fixes the lint-and-rustfmt check-whitespace failure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

maheshwarip requested a review from a team as a code owner July 17, 2026 16:25

maheshwarip requested a review from aljoscha July 17, 2026 16:28

maheshwarip marked this pull request as draft July 17, 2026 16:28

aljoscha reviewed Jul 20, 2026

View reviewed changes

Comment thread doc/user/content/sql/alter-cluster.md Outdated

Comment thread doc/user/content/sql/alter-cluster.md Outdated

Comment thread doc/user/content/sql/show-clusters.md Outdated

maheshwarip commented Jul 20, 2026

View reviewed changes

Comment thread doc/user/content/sql/alter-cluster.md Outdated

maheshwarip commented Jul 20, 2026

View reviewed changes

Comment thread doc/user/data/examples/alter_cluster.yml

maheshwarip marked this pull request as ready for review July 20, 2026 21:14

maheshwarip commented Jul 20, 2026

View reviewed changes

Comment thread doc/user/content/sql/alter-cluster.md Outdated

maheshwarip and others added 3 commits July 20, 2026 17:57

maheshwarip commented Jul 20, 2026

View reviewed changes

Comment thread doc/user/content/sql/alter-cluster.md

kay-kim reviewed Jul 21, 2026

View reviewed changes

bobbyiliev mentioned this pull request Jul 22, 2026

docs: Document autoscaling strategy inheritance in dbt blue/green deployments #37801

Draft

kay-kim reviewed Jul 22, 2026

View reviewed changes

Comment thread doc/user/content/concepts/clusters.md Outdated

Comment thread doc/user/content/headless/cluster-hydration-burst.md Outdated

Comment thread doc/user/content/headless/cluster-hydration-burst.md Outdated

Comment thread doc/user/content/headless/cluster-hydration-burst.md

kay-kim reviewed Jul 22, 2026

View reviewed changes

Comment thread doc/user/content/headless/cluster-hydration-burst.md Outdated

maheshwarip and others added 2 commits July 23, 2026 09:58

Update cluster-hydration-burst.md

49b394c

maheshwarip enabled auto-merge (squash) July 23, 2026 15:44

kay-kim reviewed Jul 23, 2026

View reviewed changes

Updated IA on the alter cluster page

f74990b

kay-kim approved these changes Jul 23, 2026

View reviewed changes

maheshwarip and others added 2 commits July 23, 2026 15:48

Fixed links

9bd924a

docs: remove trailing whitespace in alter-cluster.md

5721836

Fixes the lint-and-rustfmt check-whitespace failure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

maheshwarip merged commit 0829028 into main Jul 23, 2026
11 checks passed

maheshwarip deleted the autoscaling branch July 23, 2026 19:58


		#### Speed up hydration by autoscaling to a larger size

		{{< include-md file="shared-content/cluster-hydration-burst.md" >}}

Uh oh!

Conversation

maheshwarip commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aljoscha left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kay-kim Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

maheshwarip Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kay-kim Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

kay-kim Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

maheshwarip Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

kay-kim Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

maheshwarip Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maheshwarip commented Jul 23, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kay-kim Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

maheshwarip commented Jul 17, 2026 •

edited

Loading

maheshwarip Jul 21, 2026 •

edited

Loading