Set a default fsGroup for Kubernetes persistent volumes

[Mitch Denny (mitchdenny)]

Description

This is a usability fix targeted for Aspire 13.5.

The problem

A Kubernetes volume access mode such as ReadWriteOnce controls how the volume can be attached and mounted; it does not grant the container's Linux process permission to write to the mounted filesystem.

That distinction produces a confusing experience with first-class Aspire persistent volumes. The PVC can be Bound, the pod can be Running, and the volume can be mounted successfully, while the application still fails on its first write because the filesystem is owned by root:root and the container runs as a non-root user. From the application's perspective this surfaces as an I/O permission error even though the deployment otherwise appears healthy.

The workaround before this change

Today, users must understand the underlying Kubernetes ownership model and customize every affected workload themselves:

builder.AddProject<Projects.WebFrontend>("webfrontend")
    .WithPersistentVolume(data, "/data")
    .PublishAsKubernetesService(resource =>
    {
        var podSpec = resource.Workload?.PodTemplate.Spec
            ?? throw new InvalidOperationException("The Kubernetes workload was not generated.");

        podSpec.SecurityContext ??= new();
        podSpec.SecurityContext.FsGroup = 2000;
        podSpec.SecurityContext.FsGroupChangePolicy = "OnRootMismatch";
    });

The alternatives are similarly low-level, such as adding a privileged init container to run chown, changing the image to use a known UID/GID, or configuring storage-driver-specific mount options. Requiring one of these workarounds makes a newly provisioned persistent volume look broken by default and leaks Kubernetes filesystem details into otherwise straightforward AppHost code.

How this change fixes it

Workloads bound through either WithPersistentVolume(...) overload now receive this pod security context automatically:

securityContext:
  fsGroup: 2000
  fsGroupChangePolicy: OnRootMismatch

fsGroup adds a supplemental group to the processes in the pod and, for supported volume types, instructs Kubernetes or the CSI driver to make the mounted volume accessible to that group. It does not change the image-defined UID or primary GID, so Aspire does not need to know which identity the image uses.

Aspire uses a stable group ID rather than selecting a new value on each deployment because numeric ownership is persisted on the volume. OnRootMismatch avoids unnecessary recursive ownership changes when the volume already has the expected group.

Existing AppHost code therefore requires no extra configuration:

var data = k8s.AddPersistentVolume("data")
    .WithCapacity("10Gi");

builder.AddProject<Projects.WebFrontend>("webfrontend")
    .WithPersistentVolume(data, "/data");

This behavior is limited to first-class WithPersistentVolume(...) bindings. Ordinary workloads and legacy PVC generation through the Kubernetes environment's default storage type are unchanged. Read-only mounts remain read-only.

Using a different fsGroup

The default is applied before existing PublishAsKubernetesService callbacks run, so a workload or cluster that requires a specific group can replace it:

builder.AddProject<Projects.WebFrontend>("webfrontend")
    .WithPersistentVolume(data, "/data")
    .PublishAsKubernetesService(resource =>
    {
        var podSpec = resource.Workload?.PodTemplate.Spec
            ?? throw new InvalidOperationException("The Kubernetes workload was not generated.");

        podSpec.SecurityContext ??= new();
        podSpec.SecurityContext.FsGroup = 3000;
    });

The generated OnRootMismatch policy is retained unless the callback also replaces it.

Opting out

A workload can remove the generated pod security context through the same customization mechanism:

builder.AddProject<Projects.WebFrontend>("webfrontend")
    .WithPersistentVolume(data, "/data")
    .PublishAsKubernetesService(resource =>
    {
        var podSpec = resource.Workload?.PodTemplate.Spec
            ?? throw new InvalidOperationException("The Kubernetes workload was not generated.");

        podSpec.SecurityContext = null;
    });

This is useful when ownership is managed by the image, an admission controller, or storage-specific configuration. Workloads that need other pod security-context settings can instead clear or replace only FsGroup and FsGroupChangePolicy.

Security and compatibility considerations

The group ID 2000 is an Aspire-managed default, not a Kubernetes-reserved value. Some storage drivers do not support fsGroup, CSI drivers may apply the group at mount time themselves, and cluster admission policies can restrict allowed group ranges. The existing customization callback remains the escape hatch for those environments.

This change intentionally grants processes in the pod supplemental group access to supported mounted volumes and can update persisted POSIX group ownership. It does not change the image-defined UID or primary GID, and it does not make a read-only mount writable.

Validation

• Publisher tests verify the generated defaults for first-class persistent-volume bindings, including read-only mounts.
• Publisher tests verify that Kubernetes customization can override the group to 3000 or remove the generated security context entirely.
• The live AKS deployment test DeployAksPersistentVolumeSurvivesRedeploy passed against commit 7b565f163e. It verifies that the application runs as non-root UID 1654, receives supplemental group 2000, sees /srv/data owned by group 2000, and can write. It then redeploys with fsGroup: 3000, confirms the same PVC is reused, verifies UID 1654 now has supplemental group 3000 and /srv/data is owned by group 3000, reads the persisted data, and creates a new file.

Checklist

• Is this feature complete?
  • Yes. Ready to ship.
  • No. Follow-up changes expected.
• Are you including unit tests for the changes and scenario tests if relevant?
  • Yes
  • No
• Did you add public API?
  • Yes
    • If yes, did you have an API Review for it?
      • Yes
      • No
    • Did you add <remarks /> and <code /> elements on your triple slash comments?
      • Yes
      • No
  • No
• Does the change make any security assumptions or guarantees?
  • Yes
    • If yes, have you done a threat model and had a security review?
      • Yes
      • No
  • No

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 19374

Or

• Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 19374"

[github-actions]

🚀 Deployment tests starting on PR #19374...

This will deploy to real Azure infrastructure. Results will be posted here when complete.

View workflow run

[Copilot]

Pull request overview

Adds default Kubernetes filesystem-group permissions for workloads using first-class persistent volumes.

Changes:

• Emits fsGroup: 2000 with OnRootMismatch.
• Documents customization through PublishAsKubernetesService.
• Adds publisher snapshots and AKS deployment coverage for defaults and overrides.

Show a summary per file

File	Description
src/Aspire.Hosting.Kubernetes/KubernetesResource.cs	Applies the default pod security context.
src/Aspire.Hosting.Kubernetes/KubernetesPersistentVolumeExtensions.cs	Documents the new behavior.
tests/Aspire.Hosting.Kubernetes.Tests/KubernetesPublisherTests.cs	Tests defaults, overrides, and removal.
tests/Aspire.Deployment.EndToEnd.Tests/AksPersistentVolumeDeploymentTests.cs	Verifies deployed AKS volume access.
tests/Aspire.Hosting.Kubernetes.Tests/Snapshots/KubernetesPublisherTests.PublishAsync_WithFirstClassPersistentVolume_OnProject_BindsViaMountPathOverload#00.verified.yaml	Captures the project manifest default.
tests/Aspire.Hosting.Kubernetes.Tests/Snapshots/KubernetesPublisherTests.PublishAsync_WithFirstClassPersistentVolume_KubernetesCustomizationOverridesDefaultFsGroup.verified.yaml	Captures an overridden group.
tests/Aspire.Hosting.Kubernetes.Tests/Snapshots/KubernetesPublisherTests.PublishAsync_WithFirstClassPersistentVolume_KubernetesCustomizationCanRemoveDefaultSecurityContext.verified.yaml	Captures removal of the context.
tests/Aspire.Hosting.Kubernetes.Tests/Snapshots/KubernetesPublisherTests.PublishAsync_WithFirstClassPersistentVolume_FallsThroughForUnboundVolumes#00.verified.yaml	Captures mixed-volume behavior.
tests/Aspire.Hosting.Kubernetes.Tests/Snapshots/KubernetesPublisherTests.PublishAsync_WithFirstClassPersistentVolume_BindsByName_PromotesToStatefulSet#00.verified.yaml	Captures name-based binding behavior.

Review details

Suppressed comments (1)

src/Aspire.Hosting.Kubernetes/KubernetesPersistentVolumeExtensions.cs:289

• This is a public API reference, so it should use see cref rather than code formatting; otherwise IntelliSense and generated API documentation cannot link users to the customization method.

    /// <c>PublishAsKubernetesService</c> to customize the pod security context when

• Files reviewed: 9/9 changed files
• Comments generated: 2
• Review effort level: Balanced

[github-actions]

🚀 Deployment tests starting on PR #19374...

This will deploy to real Azure infrastructure. Results will be posted here when complete.

View workflow run

[github-actions]

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

[Copilot]

Review details

Suppressed comments (1)

src/Aspire.Hosting.Kubernetes/KubernetesResource.cs:192

• Using one global GID changes more than volume ownership: Kubernetes defines fsGroup as a supplemental group for every process in every container, so any image files or other mounts already group-readable/writable by numeric GID 2000 become accessible too. Because GIDs are not namespaced and 2000 is commonly available for image-defined users/groups, this security-sensitive access expansion can silently cross an image's intended permission boundary. Avoid assigning an arbitrary universal group by default; require an explicit/opt-in group (ideally through a first-class binding option) or otherwise obtain a group guaranteed by the target cluster/workload.

            securityContext.FsGroup ??= DefaultPersistentVolumeFsGroup;

• Files reviewed: 9/9 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[github-actions]

🚀 Deployment tests starting on PR #19374...

This will deploy to real Azure infrastructure. Results will be posted here when complete.

View workflow run

[Adam Ratzman (adamint)]

I think the fixed fsGroup: 2000 default needs another pass before this is ready. It is rejected by OpenShift’s default restricted SCC because the value is outside the namespace-allocated range, and it causes a one-time recursive permission rewrite for existing PVCs on upgrade. I also left a smaller test comment to assert the workload is actually non-root.

The implementation itself worked in the targeted publisher tests and the live AKS deployment path. That AKS run was at the parent commit, but the only delta to the current head is the XML-doc link fix.

[github-actions]

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

[github-actions]

🚀 Deployment tests starting on PR #19374...

This will deploy to real Azure infrastructure. Results will be posted here when complete.

View workflow run

[github-actions]

Tests selector (audit mode)

The full test matrix and all jobs still run in audit mode. The tests and jobs below are what selective CI would run under enforcement.

4 / 100 test projects · 3 jobs, from 9 changed files.

Selected test projects (4 / 100)

Aspire.Deployment.EndToEnd.Tests, Aspire.Hosting.Azure.Kubernetes.Tests, Aspire.Hosting.Docker.Tests, Aspire.Hosting.Kubernetes.Tests

Selected jobs (3)

deployment-e2e, extension-e2e, typescript-api-compat

---

How these were chosen — grouped by what changed

🔧 src/Aspire.Hosting.Kubernetes/KubernetesPersistentVolumeExtensions.cs (changed source)
→ 1 directly: Aspire.Hosting.Kubernetes.Tests
→ 2 via the project graph: Aspire.Hosting.Azure.Kubernetes.Tests (2 hops), Aspire.Hosting.Docker.Tests

🔧 src/Aspire.Hosting.Kubernetes/KubernetesResource.cs (changed source)
→ 1 directly: Aspire.Hosting.Kubernetes.Tests

🧪 tests/Aspire.Deployment.EndToEnd.Tests/AksPersistentVolumeDeploymentTests.cs (changed test)
→ 1 directly: Aspire.Deployment.EndToEnd.Tests

🧪 tests/Aspire.Hosting.Kubernetes.Tests/KubernetesPublisherTests.cs (changed test)
→ 1 directly: Aspire.Hosting.Kubernetes.Tests

🧪 tests/Aspire.Hosting.Kubernetes.Tests/Snapshots/KubernetesPublisherTests.PublishAsync_WithFirstClassPersistentVolume_BindsByName_PromotesToStatefulSet#00.verified.yaml (changed test)
→ 1 directly: Aspire.Hosting.Kubernetes.Tests

🧪 tests/Aspire.Hosting.Kubernetes.Tests/Snapshots/KubernetesPublisherTests.PublishAsync_WithFirstClassPersistentVolume_FallsThroughForUnboundVolumes#00.verified.yaml (changed test)
→ 1 directly: Aspire.Hosting.Kubernetes.Tests

🧪 tests/Aspire.Hosting.Kubernetes.Tests/Snapshots/KubernetesPublisherTests.PublishAsync_WithFirstClassPersistentVolume_KubernetesCustomizationCanRemoveDefaultSecurityContext.verified.yaml (changed test)
→ 1 directly: Aspire.Hosting.Kubernetes.Tests

🧪 tests/Aspire.Hosting.Kubernetes.Tests/Snapshots/KubernetesPublisherTests.PublishAsync_WithFirstClassPersistentVolume_KubernetesCustomizationOverridesDefaultFsGroup.verified.yaml (changed test)
→ 1 directly: Aspire.Hosting.Kubernetes.Tests

🧪 tests/Aspire.Hosting.Kubernetes.Tests/Snapshots/KubernetesPublisherTests.PublishAsync_WithFirstClassPersistentVolume_OnProject_BindsViaMountPathOverload#00.verified.yaml (changed test)
→ 1 directly: Aspire.Hosting.Kubernetes.Tests

Job reasons

Job	Triggered by
deployment-e2e	• tests/Aspire.Deployment.EndToEnd.Tests/AksPersistentVolumeDeploymentTests.cs • affected project Aspire.Hosting.Azure.Kubernetes
extension-e2e	• src/Aspire.Hosting.Kubernetes/KubernetesPersistentVolumeExtensions.cs, src/Aspire.Hosting.Kubernetes/KubernetesResource.cs • affected project Aspire.Hosting.Kubernetes
typescript-api-compat	affected project Aspire.Hosting.Kubernetes

---

Selection computed for commit 7b565f1.

[Copilot]

Review details

• Files reviewed: 9/9 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Mitch Denny (mitchdenny)]

PR Testing Report

PR Information

• PR Number: Set a default fsGroup for Kubernetes persistent volumes #19374
• Title: Set a default fsGroup for Kubernetes persistent volumes
• Head Commit: 7b565f163e7b02d50d4a2a79a1de485f9881a497
• Tested At: 2026-08-14

Artifact Version Verification

• Expected Commit: 7b565f163e7b02d50d4a2a79a1de485f9881a497
• Manifest-scenario version: 13.6.0-pr.19374.g867218f5
• Live deployment source: 7b565f163e7b02d50d4a2a79a1de485f9881a497
• Status: ✅ Verified

The PR CLI and package hive were installed with the published "Dogfood this PR" command. The repo container install selected the matching Linux ARM64 artifact. Manifest scenarios used a separate isolated local install because the minimal container runner does not include the .NET SDK required to compile and publish an AppHost. The subsequent PR commit only strengthened the deployment test assertions; the targeted deployment workflow rebuilt the solution, packages, and CLI at that latest head before running the live test.

Changes Analyzed

Files Changed

• src/Aspire.Hosting.Kubernetes/KubernetesResource.cs
• src/Aspire.Hosting.Kubernetes/KubernetesPersistentVolumeExtensions.cs
• tests/Aspire.Hosting.Kubernetes.Tests/KubernetesPublisherTests.cs
• tests/Aspire.Hosting.Kubernetes.Tests/Snapshots/KubernetesPublisherTests.PublishAsync_WithFirstClassPersistentVolume_BindsByName_PromotesToStatefulSet#00.verified.yaml
• tests/Aspire.Hosting.Kubernetes.Tests/Snapshots/KubernetesPublisherTests.PublishAsync_WithFirstClassPersistentVolume_FallsThroughForUnboundVolumes#00.verified.yaml
• tests/Aspire.Hosting.Kubernetes.Tests/Snapshots/KubernetesPublisherTests.PublishAsync_WithFirstClassPersistentVolume_KubernetesCustomizationCanRemoveDefaultSecurityContext.verified.yaml
• tests/Aspire.Hosting.Kubernetes.Tests/Snapshots/KubernetesPublisherTests.PublishAsync_WithFirstClassPersistentVolume_KubernetesCustomizationOverridesDefaultFsGroup.verified.yaml
• tests/Aspire.Hosting.Kubernetes.Tests/Snapshots/KubernetesPublisherTests.PublishAsync_WithFirstClassPersistentVolume_OnProject_BindsViaMountPathOverload#00.verified.yaml
• tests/Aspire.Deployment.EndToEnd.Tests/AksPersistentVolumeDeploymentTests.cs

Change Categories

• CLI changes
• Hosting integration changes
• Dashboard changes
• Template changes
• Client/component changes
• VS Code extension changes
• CI infrastructure changes
• Unit, snapshot, and deployment test changes

Test Scenarios Executed

Scenario 1: Default persistent-volume manifest

Objective: Verify that a fresh AppHost using the PR package emits the automatic pod filesystem group for a first-class persistent volume.

Coverage Type: Happy path

Status: ✅ Passed

Steps:

1. Created a fresh aspire-empty C# AppHost from the PR template hive.
2. Added Aspire.Hosting.Kubernetes version 13.6.0-pr.19374.g867218f5 from the PR package hive.
3. Added a Kubernetes environment, a first-class data persistent volume, and a container mounting it at /data.
4. Published the Helm chart with the PR CLI.
5. Inspected and asserted the generated workload and PVC manifests.

Evidence:

• scenario-default/DefaultVolume/apphost.cs
• scenario-default/publish.log
• scenario-default/assertions.log
• scenario-default/output/templates/service/statefulset.yaml
• scenario-default/output/templates/data/data.yaml

Observations:

• The workload was promoted to a StatefulSet.
• The volume source references the generated data PVC.
• The pod spec contains fsGroup: 2000.
• The pod spec contains fsGroupChangePolicy: "OnRootMismatch".

---

Scenario 2: fsGroup customization

Objective: Verify that the existing Kubernetes customization hook overrides Aspire's default group.

Coverage Type: Boundary/customization

Status: ✅ Passed

Steps:

1. Created a separate fresh AppHost from the PR hive.
2. Added the first-class persistent volume.
3. Used PublishAsKubernetesService to set FsGroup to 3000.
4. Published and inspected the generated StatefulSet.

Evidence:

• scenario-override/OverrideVolume/apphost.cs
• scenario-override/publish.log
• scenario-override/assertions.log
• scenario-override/output/templates/service/statefulset.yaml

Observations:

• The generated group is 3000, not 2000.
• OnRootMismatch remains present when only the group is overridden.

---

Scenario 3: fsGroup opt-out

Objective: Verify that users can remove the generated pod security context.

Coverage Type: Unhappy path/customization

Status: ✅ Passed

Steps:

1. Created a separate fresh AppHost from the PR hive.
2. Added the first-class persistent volume.
3. Used PublishAsKubernetesService to set the generated pod SecurityContext to null.
4. Published and inspected the generated StatefulSet.

Evidence:

• scenario-optout/OptOutVolume/apphost.cs
• scenario-optout/publish.log
• scenario-optout/assertions.log
• scenario-optout/output/templates/service/statefulset.yaml

Expected Unhappy-Path Outcome: The persistent volume remains bound, but Aspire emits no pod securityContext.

Observations:

• The StatefulSet and PVC binding remain present.
• No securityContext is emitted.

---

Scenario 4: Unbound legacy-volume boundary

Objective: Verify that the new default does not affect a workload that has a volume but no first-class persistent-volume binding.

Coverage Type: Negative/boundary

Status: ✅ Passed

Steps:

1. Created a separate fresh AppHost from the PR hive.
2. Added a Kubernetes environment and a container with an ordinary named volume.
3. Published and inspected the generated workload.

Evidence:

• scenario-boundary/LegacyVolume/apphost.cs
• scenario-boundary/publish.log
• scenario-boundary/assertions.log
• scenario-boundary/output/templates/service/deployment.yaml

Expected Boundary Outcome: The workload remains a Deployment, the unbound volume remains emptyDir, and no pod securityContext is generated.

Observations:

• The workload remains a Deployment.
• The volume is emitted as emptyDir.
• No securityContext is emitted.

---

Scenario 5: Live AKS persistent-volume redeployment

Objective: Verify real Azure Disk write access and ownership transition using the latest PR artifacts.

Coverage Type: Deployment end-to-end

Status: ✅ Passed

Workflow: https://github.com/microsoft/aspire/actions/runs/31772264851

Evidence:

• https://github.com/microsoft/aspire/actions/runs/31772264851/job/94681733158
• strengthened-aks-results/Deployment.EndToEnd-Deployment.trx
• strengthened-aks-results/recordings/DeployAksPersistentVolumeSurvivesRedeploy.cast

Observations:

• The first pod ran as non-root UID 1654, with groups 1654 2000; /srv/data had group 2000.
• The application wrote a marker file using the default group.
• After redeployment, the replacement pod still ran as UID 1654, with groups 1654 3000; /srv/data had group 3000.
• The StatefulSet reused the same PVC while replacing the pod.
• Existing persisted data remained readable.
• The replacement pod created a new file, proving directory write access after the group transition.
• The test passed in 9 minutes 8 seconds.

Summary

Scenario	Status	Notes
Default persistent-volume manifest	✅ Passed	StatefulSet, PVC, group 2000, and OnRootMismatch verified
fsGroup customization	✅ Passed	Group 3000 replaces the default
fsGroup opt-out	✅ Passed	Pod security context removed
Unbound legacy-volume boundary	✅ Passed	Deployment and emptyDir unchanged
Live AKS persistent-volume redeployment	✅ Passed	Non-root UID, supplemental groups, mount ownership, PVC reuse, persisted read, and new write verified

Overall Result

✅ PR VERIFIED

[Adam Ratzman (adamint)]

/backport to release/13.5

[github-actions]

Started backporting to release/13.5 (link to workflow run)

[Mitch Denny (mitchdenny)]

/backport to release/13.5

[github-actions]

Started backporting to release/13.5 (link to workflow run)

[aspire-repo-bot]

Pull request created: #1502

Generated by PR Documentation Check · auto · 41.8 AIC · ⌖ 15.6 AIC · ⊞ 19.6K

[aspire-repo-bot]

📝 Documentation has been drafted in microsoft/aspire.dev#1502 targeting release/13.5.

Added a new "Default pod security context" section to deployment/kubernetes/persistent-volumes.mdx documenting the automatic fsGroup: 2000 / fsGroupChangePolicy: OnRootMismatch pod security context applied to WithPersistentVolume(...) workloads, including how to override the group or remove the security context via PublishAsKubernetesService, and caveats about storage-driver/CSI/admission-policy support.

Note

This draft PR needs human review before merging.