[proposal] Eliminate the source-code copy into emptyDir on webserver start (investigation)

[henzigo]

Context

Every webserver pod start copies the whole application source from the image into an emptyDir
(cp -r -n /var/www/html/. /tmp/source-codes) so the nginx sidecar can serve the same files.
Cost: tens of seconds of startup, duplicated disk usage on the node, and the init container
is pure overhead — the code is already in the image.

Proposal (investigation)

Evaluate alternatives to the copy:

• A: multi-stage build that puts the static/public assets into the nginx image — nginx
  serves its own copy, php-fpm serves from its image, no shared volume at all
• B: nginx sidecar runs the application image with an nginx binary (single image, no copy)
• C: keep the shared volume but populate it via an image subPath-style approach /
  readOnly bind of the image filesystem (not natively supported — likely rejected)
• D: status quo + medium: Memory and/or sizeLimit as micro-optimizations

Constraint: must keep the "symlink switch" deploy semantics the app relies on
($realpath_root in the nginx config) — verify with the application team.

Deliverable: decision record; implementation follow-up depends on project-base image changes.

Tasks

• Measure current copy cost on a real project (startup time, bytes)
• Prototype option A or B
• Decision record

Related: #26 (warmup init container)

[henzigo]

Initial assessment

1. Why the copy exists and what actually depends on it

The copy-source-codes-to-volume init container exists for exactly one structural reason: two different images must see one filesystem tree at /var/www/html. The nginx sidecar is stock nginx:1.29-alpine (no app code), yet the vhost serves static files from disk. The emptyDir is the lowest-common-denominator way to give it the tree. php-fpm mounting the same emptyDir is a side effect of that design (inherited from the Capistrano-era "shared docroot" mental model), not a hard requirement — cron, consumers and the migration job all run the very same app image without any copy, which is strong evidence php-fpm can run directly from its image.

What in files/nginx/project-nginx.conf actually touches the nginx container's filesystem:

Directive / location	Filesystem dependency
root /var/www/html/web (both server blocks)	Path must exist for $realpath_root and try_files; the :80 health block uses stub_status only — no disk access
location ~ ^/(?:[^/]+/)?(admin|ckeditor|elfinder…|build|bundles) → try_files $uri @app	stats + serves files (admin assets, Webpack build/, Symfony bundles/)
location ~ ^/(public)/ → try_files $uri $custom_error_target	stats + serves files
location ~ …(file|customer-file|personal-overview-export|social-network/login|convertim) → try_files $uri @app	stats disk, but these are backend routes — in practice $uri should never match a file; could become a plain return 470
$realpath_root in /resolve-friendly-url, order/payment-status-notify, @app, @imageResizer	nginx resolves the real path of the docroot on its own filesystem at request time — the directory must exist in the nginx container (files themselves need not)
Everything else (/health, /content/** → S3 proxy_pass, /_next/ → storefront, @storefront, @404, the image-redirect regexes)	no filesystem access — pure proxy/fastcgi/return

So the true static surface nginx needs is web/ minus web/content (proxied to S3): essentially web/build, web/bundles, web/public and a handful of root files. vendor/, src/, var/ — the bulk of the tree — are copied today for no nginx benefit at all.

One latent bug worth noting in passing: cp -r -n with -n (no-clobber) means that if the init container is OOM-killed or evicted mid-copy and restarts, partially-written files from the first attempt are skipped, not repaired — the pod can come up with truncated assets.

2. Options analysis

a. Multi-stage build: static assets baked into a project nginx image

Nginx serves its own copy of web/; php-fpm runs from the app image; no shared volume.

• project-base / CI: new build stage (FROM nginx:1.29-alpine + COPY --from=app /var/www/html/web /var/www/html/web), push a second per-project image, tag in lockstep with the app image.
• Chart: almost nothing — webserver.nginx.image.repository is already a value CI can set per project; drop the init container + source-codes volume, mount nothing into nginx.
• try_files semantics: unchanged — files exist locally, $realpath_root resolves fine.
• Rolling-update asset skew: no regression versus today. Today each pod already carries its own asset snapshot; during a rollout an old pod can receive a request for a new-hash asset (or vice versa) and 404s into @app/@storefront. That behavior is identical with a baked nginx image, as long as CI deploys both images from the same commit (checksums/rollout already keyed to the release). Versioned/hashed filenames (Webpack Encore manifest) make this safe in practice.
• Trade-off: nginx image stops being "stock" — security patching of nginx now rides on project rebuilds, and every project's CI grows a second image build/push.

b. nginx sidecar runs the app image

The app image (shopsys project-base php-fpm image) almost certainly does not contain an nginx binary today — this must be confirmed on the pilot (which nginx in the app image). Assuming it doesn't:

• Cost to add: apk add nginx in project-base ≈ 5–10 MB image growth; negligible. Both containers then run the same image, so the node stores the layers once — no disk duplication at all, and no copy.
• Chart: nginx container image defaults to webserver.image, with command: ["nginx", "-g", "daemon off;"]; config mounts stay as they are. $realpath_root, try_files — everything works, because the full tree is genuinely there.
• Memory: nginx worker footprint unchanged; the big image only costs pull time once per node (already paid for php-fpm).
• Trade-offs: nginx version/patching becomes coupled to app releases (inverse of option a's trade-off); running nginx from a php-oriented image needs care with user/ports (currently listens on 8080 + 80, fine unprivileged with the existing custom nginx.conf); the stock-image simplicity ("just bump the tag in values") is lost.

c. nginx as a pure proxy (no disk at all)

Route build/bundles/public through php-fpm or a dedicated static-file container. Not recommended: serving hashed static assets through the fastcgi path burns php-fpm workers (pm.max_children = 20) on byte-shuffling, loses sendfile, and adds latency exactly on the hottest, cheapest requests. A separate static-server container is just option (a) with extra steps. Agree with the issue's instinct to reject the option-C family.

d. Status quo optimizations (chart-only)

This is the underrated option, because php-fpm does not need the emptyDir:

• php-fpm mounts its own image at /var/www/html (just remove the mount — configMap/secret subPath mounts stay); writes (var/cache, var/log) go to the container's writable layer, exactly as cron/consumers already do. opcache is configured for immutable code (validate_timestamps = 0, revalidate_path = 0) — nothing about the emptyDir is load-bearing for PHP.
• The init container copies only web/ (or even only web/{build,bundles,public} + root assets) for nginx. That drops the copy from the full tree (vendor + src + var) to tens of MB — likely a 10–20× reduction.
• With the scope reduced, emptyDir: {medium: Memory, sizeLimit: 256Mi} becomes viable (RAM-backed stat/serve, counts against pod memory — unacceptable for the full tree, fine for web/). sizeLimit is good hygiene either way.
• Interaction with [app] Run cache warmup as an init container instead of postStart (port of deployment#73) #26 (warmup init container): [app] Run cache warmup as an init container instead of postStart (port of deployment#73) #26's design writes the warmed cache into the shared source-codes volume so php-fpm inherits it. If php-fpm runs from its image, [app] Run cache warmup as an init container instead of postStart (port of deployment#73) #26 needs a small dedicated var/cache emptyDir shared between the warmup init container and php-fpm instead. Solvable, but the two issues must be designed together.
• Verification needed before shipping: (1) does web/bundles contain symlinks into vendor/ (Symfony assets:install --symlink)? If yes, copying only web/ leaves dangling links — the pilot check is find /var/www/html/web -type l | head; the fix is assets:install in copy mode at image build, or cp -rL. (2) Does anything write into web/ at runtime that nginx must serve? From the vhost it appears not (all content/** goes to S3, resized images are served live via @imageResizer), but this is the question for the application team.

3. The "symlink switch" / $realpath_root constraint

The comment in the config ("use $realpath_root instead of $document_root because of symlink switching when deploying") is a Capistrano-era artifact. Its purpose: when current/ is a symlink flipped between releases/N directories on a shared host, $realpath_root gives PHP the resolved release path so opcache (keyed by realpath) doesn't serve stale code across releases.

In Kubernetes none of that exists: each pod's code is immutable for its lifetime, deploys are pod replacement, there is no symlink to flip — and the opcache config already assumes immutability (validate_timestamps = 0). Therefore:

• $realpath_root → $document_root is safe in principle, with one pilot check: confirm /var/www/html/web is not itself a symlink inside the image (readlink -f /var/www/html/web). If it isn't, the two variables are byte-identical today.
• The switch is enabling, not just cosmetic: $document_root is pure string substitution, so nginx no longer needs the docroot to exist on its own filesystem for the fastcgi-only locations (/resolve-friendly-url, payment-status-notify, @app, @imageResizer). Combined with option (d)'s scope reduction, nginx's disk needs shrink to exactly the try_files locations.
• Still verify with the application team, per the issue — but the expected answer is "constraint no longer applies".

4. Startup-time and disk impact of the current copy

Not determinable from this repo (no app image reference is committed — CI injects it). Typical Shopsys production trees run 0.5–1 GB (vendor ≈ 250–450 MB, warmed var/cache 50–150 MB, web/ assets 20–100 MB), which on overlayfs/network-backed node disk yields the "tens of seconds" the issue describes — paid on every pod start, including HPA scale-ups, where it directly delays capacity. Measurement plan for the pilot:

# tree size, and how much of it nginx actually needs
kubectl exec deploy/webserver-php-fpm -c php-fpm -- sh -c \
  'du -sh /var/www/html /var/www/html/vendor /var/www/html/web /var/www/html/var 2>/dev/null'

# real init-container duration (copy cost)
kubectl get pod -l app=webserver-php-fpm -o jsonpath='{range .items[*].status.initContainerStatuses[*]}{.name}{": "}{.state.terminated.startedAt}{" -> "}{.state.terminated.finishedAt}{"\n"}{end}'

# preconditions for options (a)/(d)
kubectl exec deploy/webserver-php-fpm -c php-fpm -- sh -c \
  'find /var/www/html/web -type l | head; readlink -f /var/www/html/web; which nginx || echo "no nginx in app image"'

# runtime writes into web/? (run after some traffic)
kubectl exec deploy/webserver-php-fpm -c php-fpm -- sh -c \
  'find /var/www/html/web -newer /var/www/html/composer.json -type f | grep -v ^/var/www/html/web/content | head'

5. Effort estimates and recommendation

Option	Effort	project-base changes	Key risks
(a) baked nginx image	M (chart S + CI/image M, cross-repo, per-project rollout)	yes — second image build + push in CI	CI/tag lockstep between two images; nginx patching moves into project rebuilds
(b) nginx in app image	M (chart S + image S, cross-repo)	yes — apk add nginx + entrypoint	nginx lifecycle coupled to app releases; unprivileged-run details; verify image really lacks nginx first
(c) pure proxy	M, rejected	no	php-fpm worker exhaustion on static traffic, latency regression
(d) chart-only: php-fpm from image + copy only web/ (+ $document_root, sizeLimit, optional medium: Memory)	S–M	no	symlinked bundles/; runtime writes into web/; must co-design with #26 (cache volume)

Recommended path:

1. Run the pilot measurements above (an afternoon; answers tree size, symlinks, nginx-in-image, runtime writes, actual copy duration).
2. Ship option (d) as the chart-only quick win — it removes ~90 % of the copy cost, needs no project-base coordination, and the $realpath_root → $document_root change rides along after app-team sign-off. Design it together with [app] Run cache warmup as an init container instead of postStart (port of deployment#73) #26 so the warmup cache gets its own small emptyDir.
3. Treat option (a) as the target end-state (kills the init container entirely, keeps images single-purpose) once project-base CI capacity exists; keep (b) as the fallback if maintaining a second per-project image proves annoying in CI.

Option C is confirmed rejected: Kubernetes has no native read-only image-volume mount on the clusters in scope (the image: volume type is alpha/feature-gated), and proxy-everything regresses performance.

Also worth noting (found during the investigation): cp -r -n in the init container skips-but-doesn't-repair truncated files if the init container restarts mid-copy — worth fixing regardless of which option wins.