fix: Update release scripts to copy files into provision image rather than bind mount - #2719
fix: Update release scripts to copy files into provision image rather than bind mount#2719dannycjones wants to merge 4 commits into
Conversation
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
I think this is a useful improvement to the release scripts, this should remain open for review. |
There was a problem hiding this comment.
Pull request overview
Updates the local dev/release verification Docker setup to avoid bind-mounting temporary/local files into containers (which can break on VM-based runtimes like Colima), by moving Spark/HMS/provision artifacts into build contexts and baking the provision script into an image.
Changes:
- Switch
docker-composebuild contexts todev/containers/{hms,spark}and replace the provisioner bind-mount with a dedicated provision image build. - Add a custom Spark image that installs required Iceberg/Hadoop/AWS JARs and Spark defaults for REST-catalog-based testing.
- Add a provisioner image containing
provision.pyto generate test data via Spark Connect.
Reviewed changes
Copilot reviewed 2 out of 7 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| dev/docker-compose.yaml | Updates build contexts and changes provision service from bind-mount to image build. |
| dev/containers/spark/spark-defaults.conf | Adds Spark configuration for Iceberg REST catalog and event logging. |
| dev/containers/spark/Dockerfile | Introduces a Spark image that downloads required runtime JARs and starts Spark Connect. |
| dev/containers/provision/provision.py | Adds provisioning script to create test namespaces/tables/data via Spark Connect. |
| dev/containers/provision/Dockerfile | Adds a provision image that runs the provisioning script and exposes completion via a marker file. |
| dev/containers/hms/Dockerfile | Adds HMS image setup including extra AWS/Hadoop JARs and core-site.xml injection. |
| dev/containers/hms/core-site.xml | Adds Hadoop S3A config for HMS to talk to MinIO/warehouse. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Which issue does this PR close?
N/A
What changes are included in this PR?
When running on a system that uses a VM to run containers, such as Colima on macOS, not all local directories may be available to the VM the container runtime. For Colima, it only mounts the home directory within the VM and thus files in the home directory (common case) can be successfully bind-mounted into the containers, this will break where we use temporary directories in the release scripts.
This change avoids the dependency on bind mounting the temporary directory by copying the Spark script into a built image for the provisioner image.
This does not address all issues on macOS as HMS does not have an ARM-based image (leading to apparent dreadful runtime due to emulation), however this eliminates one problem. If we move to a HMS v4+ in the future, this change gets us closer to reliable release candidate verification on macOS.
Are these changes tested?
Yes, changes have been tested locally by bringing up the containers and running the tests.