additionalVolumeClaimTemplates: is "fixed at creation" a permanent constraint, or could additions be allowed?

[matanper]

Company or project name

No response

Use case

Two situations, both on clusters that already exist and hold data:

1. Growing capacity. Adding a disk is the natural way to extend a cluster that's filling up. Today that's only available to clusters whose authors anticipated it at creation time; everyone else has to recreate.

2. Per-tenant storage isolation. We need a dedicated volume encrypted with a specific customer's managed KMS key, holding exactly one tenant's table and nothing else. This requirement arrives well after the cluster is built — it comes from a contract, not from capacity planning — and the key doesn't exist until the customer creates it. Recreating a multi-terabyte production cluster to attach it isn't practical.

Describe the solution you'd like

Allow additions to additionalVolumeClaimTemplates after cluster creation.

Note the restriction is currently enforced in two places, so the webhook alone isn't the whole change:

// internal/webhook/v1alpha1/common.go:125 — rejects the spec change
if len(oldTemplates) != len(newTemplates) {
    return errors.New("additionalVolumeClaimTemplates cannot be added or removed after cluster creation")
}

// internal/controller/resourcemanager.go:269 — discards it on reconcile
statefulSet.Spec.VolumeClaimTemplates = input.Existing.STS.Spec.VolumeClaimTemplates

One mechanism that avoids the immutability problem entirely: volumeClaimTemplates are only used for provisioning, and a StatefulSet uses a PVC that already exists under the name it expects. So the operator can create the PVC directly per replica and reference it from the pod template — which is mutable — leaving volumeClaimTemplates untouched.

Describe alternatives you've considered

• Declare spare disks at creation time. Doesn't fit either use case: you'd pay for idle volumes indefinitely, and you cannot pre-declare a StorageClass for a customer KMS key that doesn't exist yet.
• Declare the disk via settings.extraConfig without a claim. This appears to work and silently doesn't. With no PVC, the disk's path is just a directory on the main data volume — ClickHouse registers the disk, system.disks shows it, a storage policy over it works, and parts report landing on it. For an encryption-isolation requirement that's the worst failure mode available: every signal says success while the data sits on the wrong volume under the wrong key. The tell is total_space matching the main volume.
• Recreate the ClickHouseCluster CR against the existing PVCs. Plausible, but it means deliberate downtime on a production cluster and depends on adoption behaving exactly as hoped.
• Recreate the StatefulSet with orphaned PVCs. Also viable and possibly what you'd prefer — more invasive, but keeps volumeClaimTemplates authoritative.
• A downstream patch. What we do today. We'd rather not carry it, which is why we're asking here.

Additional context

No response

[melancholictheory]

Not a maintainer, so this is a contributor's note rather than an answer on whether the constraint should stay.

I went digging into whether the recreate route is as awkward as it sounds, and it looks cheaper than expected.

The StatefulSets the operator creates end up with persistentVolumeClaimRetentionPolicy: {whenDeleted: Retain, whenScaled: Retain}, so deleting one leaves its PVCs behind. Those PVCs are named <disk>-<statefulset>-0, which is exactly what a recreated StatefulSet asks for, so existing disks get picked back up and only the new one is provisioned.

The operator already deletes and recreates a StatefulSet deliberately. resourcemanager.go compares the clickhouse.com/statefulset-version annotation and removes the StatefulSet on a breaking change, so the delete-then-requeue path exists and is exercised today.

Adding a disk changes the StatefulSet spec hash, which puts the replica into StageHasDiff. reconcileReplicaResources picks exactly one replica per reconcile in that stage, so recreation is already serialised across replicas instead of hitting them all at once.

That would reduce the change to relaxing the length check in the webhook and detecting added volumeClaimTemplates just before the line that copies the existing ones. The cost is one pod restart per replica, taken one replica at a time.

Happy to put a PR together along those lines if that direction is acceptable, or to leave it if you would rather go with the PVC-per-replica mechanism. That one keeps the pods up but is a much bigger semantic change, and which trade-off is right here is yours to call.

Separately, the extraConfig-without-a-claim behaviour you describe in the alternatives looks like its own problem, independent of what happens with this request. Silently landing parts on the main volume while system.disks reports success is a nasty failure mode for anyone using it for isolation rather than capacity.