Skip to content

tests: add minimum version floor to PrepareVersion() to skip EOL streams#3386

Open
redhat-chai-bot wants to merge 1 commit into
openshift:masterfrom
redhat-chai-bot:fix-version-resolution-min-floor
Open

tests: add minimum version floor to PrepareVersion() to skip EOL streams#3386
redhat-chai-bot wants to merge 1 commit into
openshift:masterfrom
redhat-chai-bot:fix-version-resolution-min-floor

Conversation

@redhat-chai-bot

@redhat-chai-bot redhat-chai-bot commented Jul 17, 2026

Copy link
Copy Markdown

Summary

Add a minimum supported version floor (4.16) to PrepareVersion() so it skips EOL minor streams when resolving z-1 and y-1 versions for CI tests.

Problem

The staging OCM API still lists versions from old minor streams (e.g. 4.14.58) as having available z-stream upgrades, but the staging infrastructure no longer supports creating clusters on those versions. When PrepareVersion() iterates all versions from newest to oldest and selects 4.14.58, cluster creation fails with "no available channels found for version 4.14.58", causing a cascade:

  • Cluster creation fails silently
  • No cluster-id file is written to SHARED_DIR
  • The readiness step fails with "no Cluster ID defined to wait for"
  • All dependent tests skip

This affects at least 9 permafailing jobs (2 directly from version mismatch + 7 from the cascade).

Fix

Added 25 lines to tests/utils/handler/resources_handler_prepare.go:

  1. minSupportedMinor = 16 constant — defines 4.16 as the minimum supported minor stream
  2. isBelowMinSupportedVersion() helper — parses version strings and filters out versions from EOL streams
  3. Skip guards in both staging iteration loops (y-stream and z-stream paths)

The fix is minimal and conservative: it only adds filtering guards inside existing loops, doesn't change other logic, and gracefully handles unparseable versions by not filtering them.

Verification

  • go vet ./tests/utils/handler/ — PASS
  • go build ./tests/utils/handler/ — PASS

@dustman9000 requested in Slack thread

Summary by CodeRabbit

  • Bug Fixes
    • Upgrade channel selection now excludes OCP versions below minor version 16 for the applicable streams.
    • This prevents unsupported older versions from being considered as upgrade targets.

When dynamically selecting a version for upgrade testing in
non-production environments, PrepareVersion() iterates all available
versions from newest to oldest looking for one with z-stream or
y-stream upgrades. This can pick versions from EOL minor streams
(e.g. 4.14.58) whose upgrade channels are no longer available in
staging, causing cluster creation to fail.

Add a minimum supported minor version filter (4.16) that skips
versions from minor streams below this threshold. This prevents
CI jobs like e2e-rosa-sts-upgrade-z-stream-f3 from selecting
stale versions that will inevitably fail.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-ci

openshift-ci Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: redhat-chai-bot
Once this PR has been reviewed and has the lgtm label, please assign jerichokeyne for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Jul 17, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 7a525c36-bcf2-4ba8-a5cf-e0a41ff1b7dd

📥 Commits

Reviewing files that changed from the base of the PR and between 3bb1e1a and 6d417ea.

📒 Files selected for processing (1)
  • tests/utils/handler/resources_handler_prepare.go

📝 Walkthrough

Walkthrough

PrepareVersion now defines a minimum supported OCP minor version of 16 and uses a helper to identify older candidates. Flexy upgrade selection skips below-threshold versions in streams y and z with debug logging. Unparseable version strings are retained.

Suggested reviewers: jerichokeyne, olucasfreitas

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly describes the main change: adding a minimum version floor to PrepareVersion to skip EOL streams.
Description check ✅ Passed The description includes the problem, fix, and verification, though several template sections like related issues and step-by-step testing are not filled out.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed No Ginkgo test titles were added or changed; the PR only updates helper logic in resources_handler_prepare.go, so the stable-name check is not applicable.
Test Structure And Quality ✅ Passed No Ginkgo test code was added or changed; the PR only edits a helper in resources_handler_prepare.go, so the test-structure check is not applicable.
Microshift Test Compatibility ✅ Passed Only a helper in tests/utils/handler changed; no Ginkgo tests or OpenShift API usage was added, so MicroShift compatibility check is not applicable.
Single Node Openshift (Sno) Test Compatibility ✅ Passed PR only changes a non-test helper; no new Ginkgo tests, contexts, or node/topology assumptions were added.
Topology-Aware Scheduling Compatibility ✅ Passed Change only filters OCP versions in a test utility; it adds no deployment, controller, or scheduling constraints.
Ote Binary Stdout Contract ✅ Passed resources_handler_prepare.go only adds logrus calls in a helper; logger writes to GinkgoWriter, and no stdout writes were added in main/init/TestMain/suite setup.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed No new Ginkgo e2e tests were added; the only change is a version-filtering helper in a non-test utility, with no IPv4 or external connectivity assumptions.
No-Weak-Crypto ✅ Passed Touched code only adds version-floor filtering in PrepareVersion; no weak crypto, custom crypto, or secret comparisons appear in the changed file.
Container-Privileges ✅ Passed Only a Go helper changed; no container/K8s manifests or privilege settings like privileged, hostPID, or allowPrivilegeEscalation were modified.
No-Sensitive-Data-In-Logs ✅ Passed No new logs expose passwords, tokens, PII, hostnames, or customer data; the added log lines only mention version IDs and skipped minors.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci

openshift-ci Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

@redhat-chai-bot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-presubmits-pr-rosa-sts-advanced 6d417ea link false /test e2e-presubmits-pr-rosa-sts-advanced
ci/prow/govulncheck 6d417ea link false /test govulncheck
ci/prow/commits 6d417ea link true /test commits
ci/prow/security 6d417ea link false /test security

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Comment thread tests/utils/handler/resources_handler_prepare.go
@amandahla

Copy link
Copy Markdown
Contributor

@dustman9000 you mentioned "The staging OCM API still lists versions from old minor streams (e.g. 4.14.58) as having available z-stream upgrades, but the staging infrastructure no longer supports creating clusters on those versions. '
Shouldnt this be fixed in the OCM API? I'm afraid that hardcoding here in the test will create the same situation in the future.

@jerichokeyne

Copy link
Copy Markdown
Contributor

The error is actually with our code for finding a valid channel for installing the EOL versions, not with installing the EOL versions themselves. I've been investigating this issue for a little while, but haven't fixed yet since I've had to work on higher priority items for a while. The issue is really with the GetCurrentlyChannel function in resources_handler_prepare.go and the fact that we currently always pass --channel when running rosa create cluster, but if we just don't provide that when installing a EOL version it works just fine

@amandahla

Copy link
Copy Markdown
Contributor

thanks @jerichokeyne for pointing that out!

Lets see if Coderabbit can help us advance ;-)

The issue is really with the GetCurrentlyChannel function in resources_handler_prepare.go and the fact that we currently always pass --channel when running rosa create cluster, but if we just don't provide that when installing a EOL version it works just fine
@coderabbitai give us some help here, investigate the issue with GetCurrentlyChannel

@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

target="$(fd -t f '^resources_handler_prepare\.go$' . | head -n 1)"
if [[ -z "$target" ]]; then
  echo "resources_handler_prepare.go not found" >&2
  exit 1
fi

echo "== Outline: $target =="
ast-grep outline "$target" --items all --view expanded || true

echo
echo "== GetCurrentlyChannel definitions and call sites =="
rg -n -C 8 '\bGetCurrentlyChannel\b' --glob '*.go' .

echo
echo "== rosa create cluster and --channel construction =="
rg -n -C 10 -P 'rosa\s+create\s+cluster|--channel\b|\bchannel\b' --glob '*.go' tests

echo
echo "== Relevant tests =="
fd -t f -e go . tests | xargs rg -n -C 5 '\bGetCurrentlyChannel\b|PrepareVersion\b|--channel\b' || true

Length of output: 6242


🏁 Script executed:

#!/bin/bash
set -u

file="tests/utils/handler/resources_handler_prepare.go"

echo "== PrepareVersion and GetCurrentChannel =="
sed -n '30,240p' "$file"

echo
echo "== Exact GetCurrentChannel definitions and call sites =="
rg -n -C 12 '\bGetCurrentChannel\s*\(' --glob '*.go' . || true

echo
echo "== ROSA create-cluster command construction and channel argument propagation =="
rg -n -C 12 -P '(?:CreateCluster|create\s+cluster|--channel\b|Channel\s*:|Channel\s*=)' --glob '*.go' tests pkg || true

echo
echo "== Version/channel-related configuration fields and test coverage =="
rg -n -C 8 -P '(?:CurrentChannel|Channel|PrepareVersion|versionRequirement)' --glob '*_test.go' --glob '*.go' tests/utils || true

Length of output: 235911


@amandahla I traced the path, and @jerichokeyne’s diagnosis is supported by the code.

What happens now

  1. GenerateClusterCreateFlags() resolves the requested version.
  2. When no channel_group is configured and the environment is non-production, tests/utils/handler/cluster_handler.go at Lines 253–263 always calls GetCurrentChannel() and adds --channel <result>.
  3. GetCurrentChannel() in tests/utils/handler/resources_handler_prepare.go at Line 175:
    • calls OCM for channels associated with the exact version;
    • selects any channel whose name ends in the version’s major.minor;
    • prefers stable-*, otherwise uses the first matching channel.

It does not validate that this channel remains usable for provisioning that exact EOL version in staging. Thus, a stale API response such as stable-4.14 gets passed to:

rosa create cluster --version 4.14.58 --channel stable-4.14

If staging no longer supports that channel, creation fails even though omitting --channel lets the backend select its default and create the EOL cluster.

Recommendation

The durable fix is to make --channel optional when there is no usable channel, rather than maintaining a minimum-version floor in PrepareVersion():

  • Keep real OCM lookup/transport errors as errors.
  • Treat “no applicable/provisionable channel for this version” as ("", nil) (or a typed/sentinel result).
  • In GenerateClusterCreateFlags(), append --channel only for a confirmed usable channel; otherwise log that the channel is intentionally omitted.
  • Add focused coverage for:
    • a supported version → --channel is included;
    • an EOL version with no usable channel → no --channel, no error;
    • OCM lookup failure → error is preserved.

The current minSupportedMinor = 16 change only avoids selecting old versions for flexy y-*/z-* requirements. It does not address explicit EOL versions, and it will require another manual update when the support floor changes.

@amandahla

Copy link
Copy Markdown
Contributor

Tests:
2. Version mismatch — 4.14.58 channel not available (2 jobs)
e2e-rosa-hcp-advanced-regional-f3 and e2e-rosa-sts-upgrade-z-stream-f3 fail because staging no longer offers channel data for 4.14.58. Fix: update the test profile to reference a currently-available version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants