[proposal] Migration off archived ingress-nginx (Gateway API / successor) (investigation)

[henzigo]

Context

The ingress-nginx repository is archived/read-only (March 2026) — no bug fixes or security
updates are coming. This package depends on it in several places:

• the e-shop and MCP/RabbitMQ ingresses (ingressClassName: nginx, nginx.ingress.kubernetes.io/*
  annotations: basic auth, whitelist-source-range, satisfy, server-snippet, proxy-body-size)
• the https-first redirect chain via configuration-snippet (kept deliberately because the
  built-in redirect annotations are broken — app-root redirect occurs before https when ssl-redirect is true kubernetes/ingress-nginx#6340; a native
  fallback style exists for locked-down controllers)
• Cloudflare real_ip_header server-snippet

Proposal (investigation, decision record as output)

Evaluate the migration path off archived ingress-nginx:

• Gateway API with a maintained implementation (Envoy Gateway, Cilium, Istio, NGINX
  Gateway Fabric, HAProxy...) — map every feature we use (basic auth! source-IP filtering,
  redirects with correct https-first ordering, body size, real client IP behind Cloudflare)
  to Gateway API primitives/policies; basic auth is the hardest (not in core Gateway API)
• InGate (the announced ingress-nginx successor) — maturity check
• staying on a maintained ingress-nginx fork/distribution, if one emerges

Constraints: the redirect chain must stay https-first (see #6340), HTTP basic auth with IP
bypass is a hard requirement for non-production environments, and the migration must be
per-project/per-environment switchable (values-driven), not a big bang.

Tasks

• Feature mapping matrix (current annotations → Gateway API / successor equivalents)
• PoC of the hardest parts (basic auth + https-first redirects) on one candidate
• Decision record + phased migration plan

Follow-up from #1 / PR #43.

[henzigo]

Initial assessment

Phase 1 of the investigation: landscape verification (as of August 2026), feature mapping, strategy comparison, urgency, and effort estimates. Sources at the bottom.

1. Landscape (verified August 2026)

ingress-nginx is retired. Announced November 2025, best-effort maintenance ended with retirement in March 2026; the repo is archived/read-only. The official statement (Kubernetes Steering + Security Response Committees, 2026-01-29) is explicit: "There will be no more releases for bug fixes, security patches, or any updates of any kind after the project is retired." Final release lines: v1.13.8 (k8s 1.29–1.33), v1.14.4 (k8s 1.33–1.34), v1.15.0 (up to k8s 1.35). Published images and Helm charts remain downloadable indefinitely.

InGate is dead. The announced successor never attracted sufficient contributors and was retired together with ingress-nginx (November 2025 announcement). It is not an option — remove it from consideration.

Maintained Ingress-API controllers:

Controller	Status	Notes for us
F5 NGINX Ingress Controller (nginx/kubernetes-ingress)	Actively maintained, Apache 2.0, F5 explicitly positions it as the long-term ingress-nginx alternative; 2026 roadmap prioritizes ingress-nginx feature parity (external auth landing in the next major release)	Different product, different annotations (nginx.org/*, not nginx.ingress.kubernetes.io/*). Real nginx underneath, so raw-nginx semantics (snippets, satisfy any, real_ip_header) are fully reachable
Traefik v3.6+	Actively maintained; ships a dedicated ingress-nginx compatibility provider that reads nginx.ingress.kubernetes.io/* annotations natively	Covers surprisingly much (see matrix), but satisfy is explicitly unsupported
HAProxy Ingress (jcmoraisjr) / haproxytech kubernetes-ingress	Both alive; supports auth-type: basic + allowlist-source-range (old whitelist- alias kept)	HAProxy could express the IP-bypass OR natively (http-request auth unless { src ... }) but only via config snippets; smaller ecosystem, secondary candidate

Gateway API implementations: Gateway API itself is mature — v1.4 (Oct 2025) and v1.5 (Feb 2026) promoted most widely-used features to the Standard channel. Conformant, production-grade implementations: Envoy Gateway (the "default" standalone choice), Cilium (requires Cilium as CNI), Istio (mature, heavy for our use case), Traefik v3, NGINX Gateway Fabric (2.x, nginx data plane, has SnippetsFilter since 1.5 and a dedicated AuthenticationFilter for basic auth). cert-manager supports Gateway API without a feature gate since 1.15 (still opt-in via config), added experimental ListenerSet support in v1.20 (Feb 2026), and published an explicit ingress-nginx-EOL guidance post.

2. Feature mapping matrix

Column (a) = best Ingress-API alternative (F5 NIC, with Traefik-compat noted); column (b) = Gateway API, concrete implementation NGINX Gateway Fabric (NGF), with Envoy Gateway (EG) noted.

Feature we use today	(a) F5 NGINX Ingress Controller (Ingress API)	(b) Gateway API (NGF; EG noted)
ingressClassName: nginx	Different class (values-driven already: ingress.className)	Gateway/GatewayClass + HTTPRoute.parentRefs — structural change
cert-manager.io/cluster-issuer on Ingress	Unchanged — cert-manager watches Ingress regardless of controller	Annotation moves to the Gateway; HTTP01 gatewayHTTPRoute solver; must enable Gateway support in cert-manager
proxy-body-size	nginx.org/client-max-body-size	NGF: ClientSettingsPolicy (body.maxSize). EG: no first-class body-size limit (buffer filter workarounds only)
https-first redirect chain (configuration-snippet with if/return 308, kept because built-in annotations are broken, kubernetes/ingress-nginx#6340)	nginx.org/server-snippets — raw nginx, identical snippet, identical semantics	Natively clean, better than today: HTTPRoute on the HTTP listener with RequestRedirect (scheme=https, 308) fires first; a second HTTPRoute on the HTTPS listener for the www/non-www variant does the hostname redirect. https-first ordering falls out of the listener structure — no snippets, Standard channel
Traefik-compat note	configuration-snippet is partially emulated (supports if, return code) — our exact snippets might work but regex named captures are unverified → PoC item	Traefik GW API: redirect filters supported
Basic auth (auth-type/auth-secret/auth-realm)	nginx.org/basic-auth-secret + nginx.org/basic-auth-realm (native). Traefik-compat: supported natively	NGF: AuthenticationFilter (ExtensionRef; secret type nginx.org/htpasswd). EG: SecurityPolicy.basicAuth. Not in Gateway API core anywhere
whitelist-source-range + satisfy: any (IP bypass of basic auth) — the rare combo	Full parity via snippets: satisfy any; allow …; deny all; in nginx.org/location-snippets next to native basic auth (snippets must be enabled on the controller). Traefik-compat: satisfy explicitly unsupported — the bypass silently becomes AND/broken	NGF: AuthenticationFilter + SnippetsFilter combo should express it (raw nginx) but is unproven — this is the PoC item (there are open issues around snippet/auth interaction, nginx/nginx-gateway-fabric#4794). EG: AND (basicAuth + authorization.clientCIDRs) works; OR is not supported — confirmed in envoyproxy/gateway#7699; would require a custom ext_authz service
Plain IP whitelist (MCP, RabbitMQ ingresses)	Snippets (allow/deny); native accessControl policy exists only for their VirtualServer CRD	NGF: SnippetsFilter. EG: SecurityPolicy.authorization.clientCIDRs (first-class)
Cloudflare real IP (server-snippet: real_ip_header CF-Connecting-IP)	nginx.org/server-snippets — identical; or global ConfigMap real-ip settings. Traefik: no CF-Connecting-IP support (only XFF via forwardedHeaders.trustedIPs, or a plugin)	NGF: SnippetsFilter (server context). EG: first-class — ClientTrafficPolicy.clientIPDetection.customHeader: CF-Connecting-IP
Path-based domains, ImplementationSpecific for dotted paths	Same Ingress API; the dotted-path restriction was ingress-nginx's strict-validate-path-type, not the API — verify NIC's path validation in PoC	HTTPRoute PathPrefix has no dot restriction — the workaround becomes unnecessary
Same-host ingress merging (ingress-mcp.yaml relies on ingress-nginx merging all ingresses of one hostname into one server block)	Gotcha: F5 NIC does not merge same-host Ingresses automatically — requires nginx.org/mergeable-ingress-type: master/minion annotations. Concrete migration work for the MCP ingress	HTTPRoutes to the same Gateway listener merge naturally; server-scoped settings live on the Gateway/policies — the current comment block in ingress-mcp.yaml becomes moot

3. Migration strategies

(a) Minimal — swap controller, keep Ingress API. Best coverage: F5 NGINX Ingress Controller. Every feature we use maps, including the two hard ones (satisfy any combo and Cloudflare real IP), because snippets are raw nginx. Template impact: mechanical annotation rewrite (nginx.ingress.kubernetes.io/* → nginx.org/* + snippets), plus master/minion for the MCP ingress; the chart structure, TLS/cert-manager flow, and values schema stay intact. A ingress.flavor: ingress-nginx|nginx-f5 values switch with per-flavor annotation helpers makes it per-environment switchable. Traefik's compat provider would be even cheaper (near-zero template change) but is disqualified as the primary path by the unsupported satisfy (our IP bypass is a hard requirement) and missing CF-Connecting-IP.

(b) Full Gateway API. New parallel templates (httproute-domains.yaml, redirects, MCP, RabbitMQ) plus implementation-specific policy CRDs, behind ingress.mode: ingress|gateway — per-environment switchable, both template sets coexist. Open design questions: which chart owns the Gateway (likely shopsys-infra, with app charts attaching HTTPRoutes via parentRefs), cert-manager annotation relocation to the Gateway, and one implementation-specific escape hatch for basic-auth-with-bypass (NGF: AuthenticationFilter+SnippetsFilter; EG: ext_authz sidecar). Redirect handling gets better (native, ordered, no snippets); the auth combo gets worse (nowhere first-class).

4. Timeline pressure

Not urgent, but not ignorable. Archived ≠ broken: pinned final releases (v1.13.8/v1.14.4/v1.15.0) keep working, images stay available. The real clocks:

• CVE exposure: unmaintained, internet-facing nginx+lua. The IngressNightmare precedent (CVE-2025-1974, 9.8 critical, March 2025) shows this codebase produces critical CVEs; the next one gets no patch. Mitigations meanwhile: pin the final release, keep the admission webhook non-exposed, Cloudflare in front already absorbs a class of attacks.
• Kubernetes compatibility horizon: v1.15.0 is tested up to k8s 1.35. Nothing is expected to break on 1.36/1.37 (the networking APIs are stable), but it is untested and unsupported — practically, migrate before adopting k8s > 1.35 on any cluster.
• Commercial stopgap exists if ever needed (HeroDevs NES sells extended ingress-nginx support).

Realistic window: decide within ~6 months, complete migration within ~12.

5. Effort estimates and recommendation

Strategy	Effort	Prerequisites	Key risks
Traefik ingress-nginx-compat (fallback only)	S–M	Traefik ≥ 3.6 on clusters	satisfy any broken, CF real IP missing — fails hard requirements as-is
(a) F5 NGINX Ingress Controller	M	Controller rollout per cluster, snippets enabled, annotation helper layer in _helpers.tpl	Snippet validation differences; master/minion merging for MCP; F5 stewardship (open source, but single-vendor)
(b) Gateway API + NGF	L	Gateway API CRDs on clusters, cert-manager Gateway support enabled, parallel template set + values switch	satisfy any via AuthenticationFilter+SnippetsFilter unproven (PoC gate); NGF 2.x is the youngest of the mature implementations
(b) Gateway API + Envoy Gateway	L–XL	Same, plus a custom ext_authz service for the auth-OR-IP combo	Extra component to build/operate solely for the bypass; otherwise the strongest first-class feature set (incl. CF-Connecting-IP)

Recommendation:

1. Now (S): stay on pinned final ingress-nginx (correct k8s-version line), keep priority/low. No emergency.
2. Phase 2 (the existing PoC task): PoC the hard combo — basic auth + IP bypass + https-first redirect chain — on F5 NIC and NGF, in that order. These two share nginx semantics, so our battle-tested snippets carry over; the PoC decides whether NGF's filter/snippet combo actually expresses satisfy any.
3. Strategic direction: Gateway API is the ecosystem's endpoint (ingress-nginx's ~50% share is migrating there; v1.5 stabilized the features we need except auth). But its auth story for our combo is immature today. Likely optimal sequence: (a) F5 NIC as the low-risk intermediate (M) to exit the unmaintained codebase quickly, then (b) Gateway API (L) behind the values switch once the PoC passes and NGF/EG auth features mature — F5's 2026 roadmap (external auth in next major NIC release) and NGF's auth filters are both moving fast, so re-evaluate the (b) candidate in Q1 2027.
4. Either way, first refactor the ingress annotation blocks into flavor-aware helpers in _helpers.tpl so templates stop hard-coding nginx.ingress.kubernetes.io/* — that abstraction is prerequisite work for both strategies and is useful immediately.

Sources: ingress-nginx retirement announcement · Steering/SRC statement, Jan 2026 · final releases (v1.13.8 etc.) · Google OSS blog, Feb 2026 · Traefik ingress-nginx provider annotation reference · F5 NIC as long-term alternative · NGINX Kubernetes OSS roadmap H1 2026 · F5 NIC annotations · NGF basic authentication · NGF snippets · EG basic auth · EG IP restriction · EG discussion: IP allowlist OR basic auth not supported · EG client IP detection (custom header) · cert-manager on ingress-nginx EOL & Gateway API · cert-manager Gateway usage · Gateway API v1.5 release · Gateway API v1.4 release · Gateway API implementations · HAProxy Ingress configuration keys · HeroDevs ingress-nginx EOL support