feat(docs): update the loculus docs with AWS specific configuration information - #7313
feat(docs): update the loculus docs with AWS specific configuration information#7313anna-parker wants to merge 4 commits into
Conversation
|
This PR may be related to: #4926 |
|
Claude finished @anna-parker's task in 3m 21s —— View job ReviewDocs-only change plus one schema fix. The core claim checks out: Three inline comments, one of which I'd treat as blocking:
Two things that could be worth adding while you're covering AWS specifics (not blocking, both outside the changed lines):
The IAM section reads accurately against what the backend actually calls —
No code changes made — this was a review only. |
| "Statement": [ | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": ["s3:GetObject", "s3:PutObject", "s3:PutObjectTagging", "s3:DeleteObject"], | ||
| "Resource": "arn:aws:s3:::my-loculus-bucket/*" | ||
| } | ||
| ] |
There was a problem hiding this comment.
This policy is missing s3:ListBucket, which the backend needs in practice.
S3Service.getFileSize (backend/.../files/S3Service.kt:234) does a HeadObject and treats a 404 as "file not uploaded yet". On AWS, if the identity has s3:GetObject but not s3:ListBucket on the bucket, a HeadObject for a nonexistent key returns 403 AccessDenied instead of 404 (AWS docs). getFileSize then rethrows, s3ErrorMapping maps the unknown code to RuntimeException("Unexpected S3 error: ..."), and FileMappingPreconditionValidator.validateFilesUploaded returns a 500 instead of the intended 422 "No file uploaded for file IDs: ...".
So an admin following this doc gets a 500 on every submission that references a file ID that was never uploaded — a normal user error.
| "Statement": [ | |
| { | |
| "Effect": "Allow", | |
| "Action": ["s3:GetObject", "s3:PutObject", "s3:PutObjectTagging", "s3:DeleteObject"], | |
| "Resource": "arn:aws:s3:::my-loculus-bucket/*" | |
| } | |
| ] | |
| { | |
| "Effect": "Allow", | |
| "Action": ["s3:GetObject", "s3:PutObject", "s3:PutObjectTagging", "s3:DeleteObject"], | |
| "Resource": "arn:aws:s3:::my-loculus-bucket/*" | |
| }, | |
| { | |
| "Effect": "Allow", | |
| "Action": "s3:ListBucket", | |
| "Resource": "arn:aws:s3:::my-loculus-bucket" | |
| } | |
| ] | |
| } |
| "default": "https://s3-<host value>", | ||
| "description": "The base URL of the S3-compatible service, including the protocol (e.g. https://). This is passed directly to the AWS SDK's endpoint override, which requires a scheme; a value without one will fail with 'The scheme of the endpoint override must not be null.' The backend addresses objects path-style (endpoint/bucket/key), so for AWS S3 use the regional endpoint rather than a bucket-specific virtual-hosted one, e.g. https://s3.eu-central-1.amazonaws.com for a bucket in eu-central-1. Ignored when runDevelopmentS3 is true: the dev S3 (MinIO) URL is generated automatically, scheme included." |
There was a problem hiding this comment.
Two things here:
-
Enforce the scheme in the schema. Since the whole point of this change is that a missing protocol prefix breaks the backend at runtime, a
patternwould turn that into ahelmvalidation error instead of a crash after deploy.s3.bucket.endpointhas no default invalues.yaml, so the pattern only applies when a user actually sets it (dev/MinIO deployments that rely onrunDevelopmentS3are unaffected):"type": "string", "pattern": "^https?://",
-
The
defaultis misleading.loculus.s3Url(kubernetes/loculus/templates/_urls.tpl:23) only produceshttps://s3<sep><host>whenrunDevelopmentS3is true andenvironment == "server"; otherwise it'shttp://<localHost>:8084. The description in the same field says the value is ignored whenrunDevelopmentS3is true — so the documented "default" is exactly the case where the field does nothing. Droppingdefault(or changing it to something like""/ none) would be less confusing than documenting a value that never applies to this key.
Minor: this description renders into a table cell via docs/src/components/SchemaDocs.astro; at ~600 chars it will be a fairly unwieldy cell. Consider trimming to the essentials (scheme required, path-style/regional endpoint) and leaving the SDK error message detail to the prose doc.
Also, if the quoted SDK error was paraphrased rather than copied from a log, worth pasting it verbatim — AWS SDK v2 raises The URI scheme of endpointOverride must not be null., which is what an admin would actually grep for.
|
|
||
| ## Configuring an S3 bucket (e.g. AWS) | ||
|
|
||
| See [Configuring extra file submission](/docs/src/content/docs/for-administrators/configuring-extra-files.md) for the full walkthrough (bucket config, credentials via sealed secrets, CORS, bucket policy). |
There was a problem hiding this comment.
Root-absolute links in GitHub-rendered markdown resolve against github.com, not the repo root, so this points at https://github.com/docs/src/... (404). A relative path works from kubernetes/README.md:
| See [Configuring extra file submission](/docs/src/content/docs/for-administrators/configuring-extra-files.md) for the full walkthrough (bucket config, credentials via sealed secrets, CORS, bucket policy). | |
| See [Configuring extra file submission](../docs/src/content/docs/for-administrators/configuring-extra-files.md) for the full walkthrough (bucket config, credentials via sealed secrets, CORS, bucket policy). |
(There's an existing instance of the same pattern at line 55, so this isn't introduced by the PR — but worth not adding another.) Linking to the published docs page instead would also work and is arguably friendlier for readers.
| @@ -130,7 +170,6 @@ You can set a permissive CORS policy on your bucket with `s3cmd setcors cors.xml | |||
| <AllowedMethod>HEAD</AllowedMethod> | |||
| <AllowedMethod>POST</AllowedMethod> | |||
There was a problem hiding this comment.
it also seems to work without POST - but potentially we should still add it
resolves #4926
Additionally fixes the values.schema.json: Protocol prefix is actually required and backend will not be able to talk to S3 without it.
Screenshot
PR Checklist
The implemented feature is covered by appropriate, automated tests.Any manual testing that has been done is documented (i.e. what exactly was tested?)🚀 Preview: Add
previewlabel to enable