USHIFT-6973 Add CIS Level2 Smoke Testing#7041
Conversation
Signed-off-by: Jonathan H. Cope <jcope@redhat.com>
Introduce a periodic CI scenario that installs MicroShift on a CIS Level 2 hardened system and validates it functions correctly. The test provisions a VM from the source image-installer ISO, registers it with subscription-manager, installs OpenSCAP and Ansible, applies CIS Level 2 hardening via ansible-role-rhel9-cis, then runs four validation checks: OpenSCAP scan produces a report, CIS failure count stays within threshold, all pods are running, and a smoke test route is accessible. New files: - test/scenarios/periodics/el98-src@cis-lvl2.sh - test/suites/cis/validate-cis-lvl2.robot - test/assets/cis/cis-harden.yml - test/assets/cis/cis-requirements.yml - test/image-blueprints/layer2-presubmit/group1/rhel98-source.image-installer The .image-installer marker enables build_images.sh to produce rhel-9.8-microshift-source.iso, which is required for the liveimg kickstart pattern (mutable RPM-based system needed for post-boot CIS.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: copejon The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
WalkthroughAdds a RHEL 9.8 MicroShift test blueprint, Ansible-based CIS Level 2 hardening, VM orchestration, and Robot Framework validation for OpenSCAP compliance, pod health, and route access. ChangesCIS Level 2 validation
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Scenario
participant VM
participant Ansible
participant OpenSCAP
participant MicroShift
Scenario->>VM: Provision and configure RHEL 9.8 host
Scenario->>Ansible: Install requirements and apply CIS Level 2 playbook
Ansible->>VM: Apply hardening changes
Scenario->>VM: Reboot and wait for SSH
Scenario->>MicroShift: Restore firewall rules and start services
Scenario->>OpenSCAP: Run CIS profile scan
Scenario->>MicroShift: Check pods and route access
🚥 Pre-merge checks | ✅ 4 | ❌ 11❌ Failed checks (1 warning, 10 inconclusive)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/suites/cis/validate-cis-lvl2.robot`:
- Around line 83-89: Update the grep command in Get CIS Failure Count to run
with sudo=True so the root-owned OSCAP_RESULTS_FILE is readable, and replace “||
echo 0” with “|| true” so zero matches preserve grep’s single 0 output without
causing conversion errors.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 8a8b0b74-4e7f-4bf5-8ca5-67d5fd5b23cf
📒 Files selected for processing (5)
test/assets/cis/cis-harden.ymltest/assets/cis/cis-requirements.ymltest/image-blueprints/layer2-presubmit/group1/rhel98-source.image-installertest/scenarios/periodics/el98-src@cis-lvl2.shtest/suites/cis/validate-cis-lvl2.robot
| Get CIS Failure Count | ||
| [Documentation] Parse the OpenSCAP results XML and count failures | ||
| ${stdout} ${stderr} ${rc}= Execute Command | ||
| ... grep -c '<result>fail</result>' ${OSCAP_RESULTS_FILE} || echo 0 | ||
| ... sudo=False return_rc=True return_stdout=True return_stderr=True | ||
| ${fail_count}= Convert To Integer ${stdout.strip()} | ||
| RETURN ${fail_count} |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Get CIS Failure Count can produce false negatives and errors on zero failures.
Two issues in the grep command:
-
sudo=Falseon a root-owned file:Run CIS Level 2 Scancreates the results file withsudo=True, so it's owned by root. On a CIS-hardened system (restrictive umask), the SSH user likely can't read it. When grep fails,|| echo 0outputs"0"— the test passes with zero failures even if the file was unreadable. TheVerify Remote File Exists With Sudokeyword in the prior test case confirms these files require sudo. -
|| echo 0duplicates output on zero matches:grep -calready prints"0"and exits 1 when there are no matches.|| echo 0then appends another"0", producing stdout"0\n0".Convert To Integerfails on this, erroring the test in the best-case scenario (zero failures).
🐛 Proposed fix
Get CIS Failure Count
[Documentation] Parse the OpenSCAP results XML and count failures
${stdout} ${stderr} ${rc}= Execute Command
- ... grep -c '<result>fail</result>' ${OSCAP_RESULTS_FILE} || echo 0
- ... sudo=False return_rc=True return_stdout=True return_stderr=True
+ ... grep -c '<result>fail</result>' ${OSCAP_RESULTS_FILE} || true
+ ... sudo=True return_rc=True return_stdout=True return_stderr=True
${fail_count}= Convert To Integer ${stdout.strip()}
RETURN ${fail_count}sudo=True ensures the root-owned file is readable. || true suppresses grep's non-zero exit on zero matches without adding duplicate output — grep -c already prints the count.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| Get CIS Failure Count | |
| [Documentation] Parse the OpenSCAP results XML and count failures | |
| ${stdout} ${stderr} ${rc}= Execute Command | |
| ... grep -c '<result>fail</result>' ${OSCAP_RESULTS_FILE} || echo 0 | |
| ... sudo=False return_rc=True return_stdout=True return_stderr=True | |
| ${fail_count}= Convert To Integer ${stdout.strip()} | |
| RETURN ${fail_count} | |
| Get CIS Failure Count | |
| [Documentation] Parse the OpenSCAP results XML and count failures | |
| ${stdout} ${stderr} ${rc}= Execute Command | |
| ... grep -c '<result>fail</result>' ${OSCAP_RESULTS_FILE} || true | |
| ... sudo=True return_rc=True return_stdout=True return_stderr=True | |
| ${fail_count}= Convert To Integer ${stdout.strip()} | |
| RETURN ${fail_count} |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@test/suites/cis/validate-cis-lvl2.robot` around lines 83 - 89, Update the
grep command in Get CIS Failure Count to run with sudo=True so the root-owned
OSCAP_RESULTS_FILE is readable, and replace “|| echo 0” with “|| true” so zero
matches preserve grep’s single 0 output without causing conversion errors.
|
@copejon: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
There was a problem hiding this comment.
We should avoid creation of another ISO just for this test.
If we need RPM host, please follow the existing pattern of RPM scenarios
| run_command_on_vm host1 "sudo reboot" || true | ||
| sleep 10 | ||
| local -r ip=$(get_vm_property host1 ip) | ||
| wait_for_ssh "${ip}" |
There was a problem hiding this comment.
This is not reliable because we do not guarantee the host / ssh would go down in 10s.
Consider doing reboot in RF code where we have reliable sequences checking boot id.
Alternatively, check for some other scenarios for reboots.
| @@ -0,0 +1,7 @@ | |||
| roles: | |||
| - name: ansible-role-rhel9-cis | |||
| src: https://github.com/RedHatOfficial/ansible-role-rhel9-cis | |||
There was a problem hiding this comment.
Please, create another set of tests for RHEL 10.
https://github.com/RedHatOfficial/ansible-role-rhel10-cis
2 participants
Introduce a periodic CI scenario that installs MicroShift on a
CIS Level 2 hardened system and validates it functions correctly.
The test provisions a VM from the source image-installer ISO,
registers it with subscription-manager, installs OpenSCAP and
Ansible, applies CIS Level 2 hardening via ansible-role-rhel9-cis,
then runs four validation checks: OpenSCAP scan produces a report,
CIS failure count stays within threshold, all pods are running,
and a smoke test route is accessible.
New files:
The .image-installer marker enables build_images.sh to produce
rhel-9.8-microshift-source.iso, which is required for the liveimg
kickstart pattern (mutable RPM-based system needed for post-boot
CIS.
Summary by CodeRabbit