Skip to content

Repository files navigation

s3proxy

Version: 0.4.1 Type: application AppVersion: 3.3.0

A Helm chart for deploying S3Proxy - Access other storage backends via the S3 API

Requirements

Repository Name Version
oci://ghcr.io/comet-ml comet-common 0.3.0

Prerequisites

  • Kubernetes 1.19+
  • Helm 3.2.0+
  • PV provisioner support in the underlying infrastructure (if using filesystem backend with persistence)

S3Proxy version compatibility

The chart tracks S3Proxy (andrewgaul/s3proxy) through appVersion (currently 3.3.0); override it with image.tag. Minimum supported version is 2.7.0.

S3Proxy 3.0.0 deprecated the Apache jclouds storage backends (s3, aws-s3, azureblob, filesystem, transient) in favor of SDK / NIO2 providers. Upstream has announced that 3.3.0 is the last release to bundle jclouds and that future releases "will lack its Atmos and B2 storage backends" (no jclouds-free release has shipped yet; 3.3.0 remains the latest). The jclouds providers still work on 3.x, but are deprecated:

  • filesystem / transient: already default to the non-deprecated *-nio2 variants (nio2: true).
  • azureblob: defaults to provider: azureblob-sdk (the Azure SDK provider, which signs correctly against custom endpoints such as Azurite). The legacy jclouds azureblob provider is deprecated and mis-signs against custom endpoints; set provider: azureblob only if you specifically need it. On real Azure, azureblob-sdk may require config.backends.azureblob.regions for bucket creation.
  • s3, googleCloudStorage, openstackSwift: SDK providers exist upstream (aws-s3-sdk, google-cloud-storage-sdk, openstack-swift-sdk). rackspaceCloudfiles is OpenStack-Swift-compatible and may be served by openstack-swift-sdk. Migrating the chart defaults to the SDK providers is tracked separately.
  • b2 (and Atmos, if ever added) are jclouds-only with no SDK successor. These are the backends upstream has said future releases will drop.

Our compatibility plan

  • Through S3Proxy 3.3.0 (chart 0.x): all current backends, including the jclouds-only b2, are supported. This is where the chart sits today.
  • Guardrail: the chart hard-fails at render time if b2 (or atmos) is enabled while the effective S3Proxy version (image.tag, else appVersion) is greater than 3.3.0, so an upgrade past the jclouds sunset cannot silently ship a broken backend. Pin image.tag to 3.3.0 or earlier, or disable the backend.
  • When S3Proxy ships its first jclouds-free release: the chart moves to 1.x.x (major bump), migrates the remaining backends to their SDK providers, and drops b2/atmos. Tracked in the SDK-migration follow-up.

Installation

Install the chart

# Install with default values (filesystem backend)
helm install my-s3proxy oci://ghcr.io/comet-ml/s3proxy

# Install with custom values
helm install my-s3proxy oci://ghcr.io/comet-ml/s3proxy -f override-values.yaml

Configuration

The following section lists the configurable parameters of the s3proxy chart and their default values.

Values

Scroll sideways to see all columns.

Key Description Type Default
affinity Affinity for pod assignment object {}
autoscaling.enabled Enable HPA bool false
autoscaling.maxReplicas Maximum number of replicas int 100
autoscaling.minReplicas Minimum number of replicas int 1
autoscaling.targetCPUUtilizationPercentage Target CPU utilization percentage int 80
config.auth.identity S3 Access Key ID for client authentication string ""
config.auth.secret S3 Secret Access Key for client authentication string ""
config.auth.type Authorization type (none, aws-v2, aws-v4, aws-v2-or-v4) string "aws-v4"
config.backends.azureblob.account Storage account name string ""
config.backends.azureblob.bucketLocators Buckets routed to this backend (S3Proxy bucket-locator; glob patterns supported). Only relevant when multiple backends are enabled; a bucket matching no backend's list falls through to the first-enabled backend. list []
config.backends.azureblob.enabled Enable Azure Blob Storage backend bool false
config.backends.azureblob.endpoint Azure endpoint string https://{{ .Values.config.backends.azureblob.account }}.blob.core.windows.net
config.backends.azureblob.key Storage account key configuration object {"existingSecret":"","secretKey":"accountKey","value":""}
config.backends.azureblob.key.existingSecret Name of existing secret containing the storage account key string ""
config.backends.azureblob.key.secretKey Key in the existing secret containing the storage account key string "accountKey"
config.backends.azureblob.key.value Storage account key value string ""
config.backends.azureblob.provider Provider type. Defaults to azureblob-sdk (Azure SDK): it signs correctly against custom endpoints (Azurite, Azure Gov/China, private endpoints) and is the non-deprecated provider on S3Proxy 3.x. The legacy jclouds azureblob provider is deprecated upstream and mis-signs against custom endpoints; on real Azure azureblob-sdk may require regions to be set for bucket creation. string "azureblob-sdk"
config.backends.azureblob.regions jclouds region(s) for the backend, emitted as jclouds.regions=. The azureblob-sdk provider requires this on real Azure to create buckets (without it aws s3 mb fails with InvalidLocationConstraint / "no jclouds.regions configured"). Comma-separated for multiple. Not needed against Azurite. Leave empty to omit. string ""
config.backends.azureblob.sasToken SAS token configuration object {"existingSecret":"","secretKey":"sasToken","value":""}
config.backends.azureblob.sasToken.existingSecret Name of existing secret containing the SAS token string ""
config.backends.azureblob.sasToken.secretKey Key in the existing secret containing the SAS token string "sasToken"
config.backends.azureblob.sasToken.value SAS token value string ""
config.backends.b2.account B2 account ID string ""
config.backends.b2.applicationKey B2 application key configuration object {"existingSecret":"","secretKey":"applicationKey","value":""}
config.backends.b2.applicationKey.existingSecret Name of existing secret containing the application key string ""
config.backends.b2.applicationKey.secretKey Key in the existing secret containing the application key string "applicationKey"
config.backends.b2.applicationKey.value Application key value string ""
config.backends.b2.bucketLocators Buckets routed to this backend (S3Proxy bucket-locator; glob patterns supported). Only relevant when multiple backends are enabled; a bucket matching no backend's list falls through to the first-enabled backend. list []
config.backends.b2.enabled Enable Backblaze B2 backend bool false
config.backends.filesystem.basedir Base directory for filesystem backend string "/data/s3proxy"
config.backends.filesystem.bucketLocators Buckets routed to this backend (S3Proxy bucket-locator; glob patterns supported). Only relevant when multiple backends are enabled; a bucket matching no backend's list falls through to the first-enabled backend. list []
config.backends.filesystem.credential jclouds credential. S3Proxy requires jclouds.credential in every backend properties file; the filesystem backend ignores the value. An empty value falls back to "local". string "local"
config.backends.filesystem.enabled Enable filesystem backend bool true
config.backends.filesystem.identity jclouds identity. S3Proxy requires jclouds.identity in every backend properties file; the filesystem backend ignores the value, so the "local" placeholder is fine. An empty value falls back to "local". string "local"
config.backends.filesystem.nio2 Use NIO2 implementation (filesystem-nio2) instead of standard filesystem bool true
config.backends.googleCloudStorage.bucketLocators Buckets routed to this backend (S3Proxy bucket-locator; glob patterns supported). Only relevant when multiple backends are enabled; a bucket matching no backend's list falls through to the first-enabled backend. list []
config.backends.googleCloudStorage.clientEmail Service account email or user email string ""
config.backends.googleCloudStorage.enabled Enable Google Cloud Storage backend bool false
config.backends.googleCloudStorage.privateKey Private key configuration object {"existingSecret":"","secretKey":"private.key","value":""}
config.backends.googleCloudStorage.privateKey.existingSecret Name of existing secret containing the private key string ""
config.backends.googleCloudStorage.privateKey.secretKey Key in the existing secret containing the private key (in PEM format) string "private.key"
config.backends.googleCloudStorage.privateKey.value Private key value (in PEM format) string ""
config.backends.googleCloudStorage.projectID GCP project ID string ""
config.backends.openstackSwift.authURL Authentication URL string ""
config.backends.openstackSwift.bucketLocators Buckets routed to this backend (S3Proxy bucket-locator; glob patterns supported). Only relevant when multiple backends are enabled; a bucket matching no backend's list falls through to the first-enabled backend. list []
config.backends.openstackSwift.enabled Enable OpenStack Swift backend bool false
config.backends.openstackSwift.password Password configuration object {"existingSecret":"","secretKey":"password","value":""}
config.backends.openstackSwift.password.existingSecret Name of existing secret containing the password string ""
config.backends.openstackSwift.password.secretKey Key in the existing secret containing the password string "password"
config.backends.openstackSwift.password.value Password value string ""
config.backends.openstackSwift.region Region string ""
config.backends.openstackSwift.tenantName Tenant name string ""
config.backends.openstackSwift.userName Username string ""
config.backends.rackspaceCloudfiles.apiKey API key configuration object {"existingSecret":"","secretKey":"apiKey","value":""}
config.backends.rackspaceCloudfiles.apiKey.existingSecret Name of existing secret containing the API key string ""
config.backends.rackspaceCloudfiles.apiKey.secretKey Key in the existing secret containing the API key string "apiKey"
config.backends.rackspaceCloudfiles.apiKey.value API key value string ""
config.backends.rackspaceCloudfiles.bucketLocators Buckets routed to this backend (S3Proxy bucket-locator; glob patterns supported). Only relevant when multiple backends are enabled; a bucket matching no backend's list falls through to the first-enabled backend. list []
config.backends.rackspaceCloudfiles.enabled Enable Rackspace Cloud Files backend bool false
config.backends.rackspaceCloudfiles.region Region (uk or us) string "us"
config.backends.rackspaceCloudfiles.userName Username string ""
config.backends.s3.accessKeyID S3 Access Key ID for backend string ""
config.backends.s3.aws Use AWS-specific S3 provider (aws-s3) instead of generic S3 provider bool true
config.backends.s3.bucketLocators Buckets routed to this backend (S3Proxy bucket-locator; glob patterns supported). Only relevant when multiple backends are enabled; a bucket matching no backend's list falls through to the first-enabled backend. list []
config.backends.s3.enabled Enable S3 backend bool false
config.backends.s3.endpoint S3 endpoint string ""
config.backends.s3.region AWS region string ""
config.backends.s3.secretAccessKey S3 Secret Access Key configuration object {"existingSecret":"","secretKey":"secretAccessKey","value":""}
config.backends.s3.secretAccessKey.existingSecret Name of existing secret containing the secret access key string ""
config.backends.s3.secretAccessKey.secretKey Key in the existing secret containing the secret access key string "secretAccessKey"
config.backends.s3.secretAccessKey.value Secret access key value string ""
config.backends.transient.bucketLocators Buckets routed to this backend (S3Proxy bucket-locator; glob patterns supported). Only relevant when multiple backends are enabled; a bucket matching no backend's list falls through to the first-enabled backend. list []
config.backends.transient.credential jclouds credential. S3Proxy requires jclouds.credential in every backend properties file; the transient backend ignores the value. An empty value falls back to "local". string "local"
config.backends.transient.enabled Enable transient (in-memory) backend bool false
config.backends.transient.identity jclouds identity. S3Proxy requires jclouds.identity in every backend properties file; the transient backend ignores the value, so the "local" placeholder is fine. An empty value falls back to "local". string "local"
config.backends.transient.nio2 Use NIO2 implementation (transient-nio2) instead of standard transient bool true
config.buckets.alias Map virtual bucket names to actual backend buckets object {}
config.cors.allowCredential Allow credentials bool false
config.cors.allowHeaders Allowed headers list ["Accept","Content-Type"]
config.cors.allowMethods Allowed methods list ["GET","PUT","POST","HEAD","DELETE"]
config.cors.allowOrigins Allowed origins (e.g., ["https://example.com", "https://.+\\.example\\.com"]) list []
config.cors.enabled Enable CORS support bool false
config.logLevel Log level for S3Proxy (DEBUG, INFO, WARN, ERROR) string "INFO"
config.middlewares.eventualConsistency Enable eventual consistency modeling bool false
config.middlewares.largeObjectMocking Enable large object mocking bool false
config.middlewares.readOnly Make backend read-only bool false
config.middlewares.shardedBackend Enable sharded backend containers bool false
config.tls.enabled Enable native in-pod HTTPS (S3Proxy secure-endpoint). When enabled, S3Proxy serves HTTPS only on service.targetPort (the plaintext endpoint is not bound), so TLS is terminated in the pod rather than at the ingress. Requires a PKCS12 (or JKS) keystore and its password. tcpSocket health probes are unaffected (they do not perform a TLS handshake). bool false
config.tls.keystore.existingSecret Name of an existing Secret holding the keystore file (binary PKCS12/JKS). Takes precedence over value. Use this for a customer-provided keystore or a cert-manager-issued one (Certificate.spec.keystores.pkcs12). string ""
config.tls.keystore.secretKey Key within keystore.existingSecret (or, when value is used, within the chart's own Secret) that holds the keystore file string "keystore.p12"
config.tls.keystore.value Inline base64-encoded keystore, stored in the chart's own Secret and mounted as a file. Used only when keystore.existingSecret is empty. Convenient for testing; prefer existingSecret in production. string ""
config.tls.keystorePassword.existingSecret Name of an existing Secret holding the keystore password. Takes precedence over value. string ""
config.tls.keystorePassword.secretKey Key within keystorePassword.existingSecret that holds the keystore password string "keystore-password"
config.tls.keystorePassword.value Inline keystore password, stored in the chart's own Secret and merged into the backend properties by the secret-merge initContainer (kept out of the ConfigMap). Used only when keystorePassword.existingSecret is empty. string ""
config.virtualHost Virtual Host configuration string ""
configMergeImage.pullPolicy Config merge container image pull policy string "IfNotPresent"
configMergeImage.repository Config merge container image repository string "busybox"
configMergeImage.tag Config merge container image tag string "1.36"
extraEnvVars Additional environment variables list []
extraVolumeMounts Additional volume mounts list []
extraVolumes Additional volumes list []
fullnameOverride String to fully override s3proxy.fullname template string ""
image.pullPolicy Image pull policy string "IfNotPresent"
image.repository S3Proxy image repository string "andrewgaul/s3proxy"
image.tag Overrides the image tag whose default is the chart appVersion string ""
imagePullSecrets Image pull secrets list []
ingress.annotations Ingress annotations object {}
ingress.className Ingress class name string ""
ingress.enabled Enable ingress bool false
ingress.hosts Ingress hosts configuration list []
ingress.tls TLS configuration list []
nameOverride String to partially override s3proxy.fullname template string ""
nodeSelector Node selector for pod assignment object {}
persistence.accessMode PVC Access Mode string "ReadWriteOnce"
persistence.annotations PVC annotations object {}
persistence.enabled Enable persistence using PVC bool true
persistence.existingClaim Use existing PVC string ""
persistence.size PVC Storage Request string "10Gi"
persistence.storageClass Storage Class string "" (uses default StorageClass)
podAnnotations Annotations to add to the pod object {}
podLabels Extra labels to add to the pod (merged with the chart's selector labels) object {}
podSecurityContext Pod security context object {}
replicaCount Number of S3Proxy replicas int 1
resources Resource limits and requests object {}
securityContext Container security context object {}
service.annotations Service annotations object {}
service.port Service port int 9000
service.targetPort Target port (controls both the container port and S3Proxy bind port) int 9000
service.type Kubernetes service type string "ClusterIP"
serviceAccount.annotations Annotations to add to the service account object {}
serviceAccount.create Specifies whether a service account should be created bool false
serviceAccount.name The name of the service account to use. If not set and create is true, a name is generated using the fullname template string ""
terminationGracePeriodSeconds Pod termination grace period. Covers the 5s preStop sleep plus the in-flight request drain (up to 30s) that S3Proxy releases after 3.3.0 perform on SIGTERM. int 40
tolerations Tolerations for pod assignment list []

Usage Examples

Example 1: Filesystem Backend with Authentication

# values-filesystem.yaml
config:
  auth:
    type: "aws-v4"
    identity: "myaccesskey"
    secret: "mysecretkey"
  backends:
    filesystem:
      enabled: true
      nio2: true
    filesystem:
      basedir: "/data/s3proxy"

persistence:
  enabled: true
  size: 20Gi

Install:

helm install s3proxy-fs ./s3proxy -f values-filesystem.yaml

Example 2: AWS S3 Backend

# values-aws-s3.yaml
config:
  auth:
    type: "aws-v4"
    identity: "proxy-access-key"  # For clients connecting to s3proxy
    secret: "proxy-secret-key"
  backend:
    provider: "aws-s3"
    awsS3:
      region: "us-west-2"
      accessKeyID: "aws-access-key-id"  # For s3proxy to connect to AWS
      secretAccessKey: "aws-secret-access-key"

persistence:
  enabled: false  # Not needed for S3 backend

Install:

helm install s3proxy-s3 ./s3proxy -f values-aws-s3.yaml

Example 3: Azure Blob Storage Backend

# values-azure.yaml
config:
  auth:
    type: "aws-v4"
    identity: "myaccesskey"
    secret: "mysecretkey"
  backend:
    provider: "azureblob"
    azureblob:
      account: "mystorageaccount"
      key: "storageaccountkey"

persistence:
  enabled: false  # Not needed for Azure backend

Install:

helm install s3proxy-azure ./s3proxy -f values-azure.yaml

Example 4: Google Cloud Storage Backend

# values-gcs.yaml
config:
  auth:
    type: "aws-v4"
    identity: "myaccesskey"
    secret: "mysecretkey"
  backend:
    provider: "google-cloud-storage"
    googleCloudStorage:
      projectID: "my-project"
      clientEmail: "service-account@my-project.iam.gserviceaccount.com"
      privateKey: |
        -----BEGIN RSA PRIVATE KEY-----
        ...
        -----END RSA PRIVATE KEY-----

persistence:
  enabled: false  # Not needed for GCS backend

Example 5: Anonymous Access (No Authentication)

# values-anonymous.yaml
config:
  auth:
    type: "none"
  backends:
    transient:
      enabled: true
      nio2: true  # In-memory storage

persistence:
  enabled: false

Testing the Installation

Once deployed, you can test S3Proxy using the AWS CLI:

# Get the service endpoint
kubectl get svc

# Port-forward for local testing
kubectl port-forward svc/my-s3proxy 8080:8080

# Configure AWS CLI (if authentication is enabled)
export AWS_ACCESS_KEY_ID=myaccesskey
export AWS_SECRET_ACCESS_KEY=mysecretkey

# Test S3 operations
aws --endpoint-url http://localhost:8080 s3 ls
aws --endpoint-url http://localhost:8080 s3 mb s3://test-bucket
aws --endpoint-url http://localhost:8080 s3 cp test.txt s3://test-bucket/
aws --endpoint-url http://localhost:8080 s3 ls s3://test-bucket/

TLS / HTTPS (native, in-pod)

By default S3Proxy binds plain HTTP and TLS is expected to be terminated at the ingress. Set config.tls.enabled=true to have S3Proxy itself terminate HTTPS in the pod (S3Proxy's secure-endpoint). When enabled:

  • S3Proxy serves HTTPS only on service.targetPort (the plaintext endpoint is not bound). The container/Service port is named https.
  • The health probes are tcpSocket, which only check the TCP accept (no TLS handshake), so they keep working unchanged against the TLS port.
  • A PKCS12 keystore is expected (Jetty's default keystore type; JKS also works if supplied). S3Proxy exposes only the keystore path and password, so the chart does not set a keystore type.
  • The keystore password is never written to the ConfigMap; it is merged into the backend properties from the Secret by the config-merge initContainer.

Option A: existing Secret (recommended; also the cert-manager path)

Reference a Secret that already holds the keystore file and the password. This is also how a cert-manager Certificate with spec.keystores.pkcs12 delivers a keystore (point keystore.existingSecret at that Secret, and keystorePassword.existingSecret at the password Secret it references):

config:
  tls:
    enabled: true
    keystore:
      existingSecret: my-tls        # holds the PKCS12 archive
      secretKey: keystore.p12
    keystorePassword:
      existingSecret: my-tls        # holds the password (may be the same Secret)
      secretKey: keystore-password

Option B: inline keystore + password

Provide the base64-encoded keystore and the password inline; both are stored in the chart's own Secret. Convenient for testing; prefer Option A in production.

config:
  tls:
    enabled: true
    keystore:
      value: "<base64-encoded PKCS12 keystore>"
    keystorePassword:
      value: "changeit"

Create a PKCS12 keystore, for example:

# From an existing cert + key:
openssl pkcs12 -export -inkey tls.key -in tls.crt \
  -out keystore.p12 -passout pass:changeit
# Inline value:
base64 -w0 keystore.p12
# Or as an existing Secret (Option A):
kubectl create secret generic my-tls \
  --from-file=keystore.p12=keystore.p12 \
  --from-literal=keystore-password=changeit

Testing over HTTPS (the CI keystore above is self-signed, so skip verification):

aws --endpoint-url https://localhost:9000 --no-verify-ssl s3 ls

CORS Configuration

To enable CORS support:

config:
  cors:
    enabled: true
    allowOrigins:
      - "https://example.com"
      - "https://.+\\.example\\.com"
    allowMethods:
      - "GET"
      - "PUT"
      - "POST"
      - "HEAD"
      - "DELETE"
    allowHeaders:
      - "Accept"
      - "Content-Type"
    allowCredential: true

Middleware Configuration

S3Proxy supports various middlewares:

config:
  middlewares:
    readOnly: false           # Make backend read-only
    eventualConsistency: true # Enable eventual consistency modeling
    shardedBackend: true      # Enable sharded backend containers
    largeObjectMocking: false # Enable large object mocking

Bucket Configuration

Bucket Aliasing

Map virtual bucket names to actual backend buckets:

config:
  buckets:
    alias:
      virtual-bucket: "real-backend-bucket"
      another-bucket: "actual-bucket-name"

Bucket Locator (routing buckets to backends)

When more than one backend is enabled, assign buckets to a specific backend with that backend's own bucketLocators list. S3Proxy reads bucket-locators per backend, so each list is emitted only into that backend's properties file. Glob patterns are supported. A bucket that matches no backend's list falls through to the first-enabled backend (the default).

config:
  backends:
    s3:
      enabled: true
      # ... credentials ...
      bucketLocators:
        - "prod-*"
        - "customer-data"
    filesystem:
      enabled: true
      bucketLocators:
        - "scratch-*"   # everything else also lands here (first-enabled default)

Monitoring

Check S3Proxy logs:

kubectl logs deployment/my-s3proxy

Upgrading

helm upgrade my-s3proxy ./s3proxy -f my-values.yaml

Uninstalling

helm uninstall my-s3proxy

This will remove all resources created by the chart. If using persistence, the PVC will be retained by default.

Troubleshooting

Common Issues

  1. Authentication failures: Ensure config.auth.identity and config.auth.secret are set correctly for client authentication.

  2. Backend connection issues: Verify backend credentials are correctly configured in the appropriate section (e.g., config.backend.awsS3.*).

  3. Persistence issues: Check that your cluster has a default StorageClass or specify one explicitly.

  4. Port conflicts: If port 8080 is already in use, change service.port and service.targetPort.

References

License

This Helm chart is provided as-is. S3Proxy itself is licensed under the Apache License 2.0.


Autogenerated from chart metadata using helm-docs v1.14.2

About

No description, website, or topics provided.

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages