Support CPD 5.3.1 - Helm-based install for WML, WSL, Spark, Cognos Analytics and SPSS - #555
Support CPD 5.3.1 - Helm-based install for WML, WSL, Spark, Cognos Analytics and SPSS#555Sakshi-Singhroha1 wants to merge 85 commits into
Conversation
…Jobs for WML, Spark, SPSS
…stead of lt(semver,semver)
…fore namespace-scoped helm install
… for Helm 5.3.1+ installs
…I groups for Helm install
- Gate OperandRegistry/OperandConfig to < 5.3.1 to prevent OLM resource drift when services are installed via Helm - Gate spark-hb-control-plane patch job to < 5.3.1; OLM-managed deployment does not exist in Helm install model - Add CRD establishment wait inside WSL install_component() between cluster-scoped apply and namespace-scoped helm upgrade to prevent race condition - Extend Helm Job --wait timeout from 30m to 60m for WML, SPSS and Spark to match WSL and avoid premature timeout on loaded clusters
* Added CPD 5.3.1 gitops support * [minor] Fix CatalogSources template - add CPD 5.3.1 block (MASCORE-15507) * For CPD 5.3.1 services are installed via Helm * MASCORE-15507: Fix WSL Helm Job - install deps in sequence; add Helm Jobs for WML, Spark, SPSS * MASCORE-15507: Pass cpd_product_version to WSL, Spark, SPSS ArgoCD Applications * MASCORE-15507: Fix heredoc in WSL Helm Job - use printf to avoid YAML/shell conflict * Fix semver comparison - use toString to handle unquoted cpd_product_version value * MASCORE-15507: skip OLM CCS/OpenSearch post-sync patching for CPD 5.3.1+ * MASCORE-15507: bump job version to v2 to force new job name after script change * minor update * changes undo * MASCORE-15507: gate CCS/datarefinery/ws-runtimes OLM subscriptions behind < 5.3.1 * MASCORE-15507: fix semver comparison syntax - use .Compare pattern instead of lt(semver,semver) * MASCORE-15507: install cluster-scoped chart first to register CRDs before namespace-scoped helm install * MASCORE-15507: grant cpd-sa cluster-level CRD/ClusterRole permissions for Helm 5.3.1+ installs * MASCORE-15507: expand cpd-sa ClusterRole with escalate, roles, CPD API groups for Helm install * MASCORE-15507: pin cli image to digest instead of :latest in Helm install jobs * adding temp changes for facilities * undone postbackflow.sh script * MASCORE-15507: fix CPD 5.3.1 Helm service install correctness - Gate OperandRegistry/OperandConfig to < 5.3.1 to prevent OLM resource drift when services are installed via Helm - Gate spark-hb-control-plane patch job to < 5.3.1; OLM-managed deployment does not exist in Helm install model - Add CRD establishment wait inside WSL install_component() between cluster-scoped apply and namespace-scoped helm upgrade to prevent race condition - Extend Helm Job --wait timeout from 30m to 60m for WML, SPSS and Spark to match WSL and avoid premature timeout on loaded clusters --------- Co-authored-by: Sakshi Singhroha <Sakshi.Singhroha@ibm.com> Co-authored-by: Nikita Kumble <nikitakumble@gmail.com>
… Jobs Static-named ArgoCD sync hook Jobs (install-wsl-helm, install-wml-helm, install-spss-helm, install-spark-helm) were failing with: 'spec.template: Invalid value ... field is immutable' This happens when an old Job with the same name exists from a prior sync and ArgoCD tries to patch it. Since Job spec.template is immutable, the patch is rejected. Fix: add HookFailed to hook-delete-policy on all four Jobs so ArgoCD deletes the Job on both success AND failure, ensuring no stale Job exists when the next sync runs.
The wait loop was polling for: opensearchclusters.opensearch.cloudpackopen.ibm.com (does not exist) The actual CRD name registered by the IBM OpenSearch Helm chart is: clusters.opensearch.cloudpackopen.ibm.com This caused the WSL Helm Job to always timeout after 5 minutes at the OpenSearch step, blocking CCS, DataRefinery, WS-Runtimes and WSL from ever installing.
…f oc wait oc wait exits immediately with error if the resource does not exist. The CCS CR (ccs-cr) is created by the CCS operator after Helm installs it, so it is not immediately present. Replace oc wait with a polling loop using --ignore-not-found, same pattern used for all other CRD waits in this script. Polls every 30s up to 60 minutes.
…fter CCS Helm install
…us for CCS CR readiness
…version semver checks
…stance-root README, add Job to WML README
…rn and $_cli_image_digest to WSL and Spark Helm install jobs
| @@ -0,0 +1,19 @@ | |||
| {{- if lt (semver "5.3.1" | (semver (.Values.cpd_product_version | default "0.0.0" | toString)).Compare) 0 }} | |||
There was a problem hiding this comment.
cognos analytics is required for cpd version >= 5.3.1 right? Can you check the condition if that is the case? Verify in all cognos related resource
There was a problem hiding this comment.
Cognos Analytics was not deployed via this GitOps repo before this story. It was newly added as part of MASCORE-15507 for CPD 5.3.1+ only via Helm. Cognos Analytics is only available for CPD >= 5.3.1. The ArgoCD Application in ibm-mas-instance-root is already gated at >= 5.3.1, but the template itself also needs a version gate . Fixed 01-ibm-cognos-analytics-cr.yaml
| - -c | ||
| - | | ||
| set -e | ||
| helm repo add ibm-charts https://raw.githubusercontent.com/IBM/charts/master/repo/ibm-helm |
There was a problem hiding this comment.
helm charts are installed using k8s job. Is there any particular reason to use this way instead of using helm chart as source directly? How does this behave whenever the job re-runs due to cli image update?
There was a problem hiding this comment.
We use a K8s Job for three reasons:
-
Multiple charts must install in sequence— cluster-scoped chart first (CRDs), wait for CRDs to be ready, then namespace-scoped chart (operator + CR). ArgoCD native Helm can only handle one chart at a time with no waiting in between.
-
CRDs must not be Helm-owned — we use
helm template | oc applyfor the cluster-scoped chart so Helm doesn't track ownership of CRDs. If Helm owned them, uninstalling one instance would delete the CRDs and break all other instances on the same cluster sharing them. -
Need to wait for the service to be fully ready —
--wait --timeout 60mmakes the Job block until the operator finishes installing (e.g. WmlBase = Completed), not just until the objects are created. ArgoCD native Helm doesn't support this level of waiting.
On re-run when CLI image updates: The Job name includes a hash of the values + CLI image digest. When the CLI image changes, the hash changes, so ArgoCD creates a new Job with the new image. The old Job is automatically deleted via HookSucceeded,HookFailed delete policy — meaning ArgoCD deletes it as soon as it completes (success or failure), so there is never a conflict between old and new Jobs. The re-run itself is safe because helm upgrade --install is idempotent — if the chart and values haven't changed, Helm detects no difference and returns immediately.
Cognos Analytics is only available on CPD >= 5.3.1 and is installed via Helm only. There is no OLM path for Cognos - it was never available on CPD < 5.3.1. The Subscription file was incorrect.
Cognos Analytics is only available on CPD >= 5.3.1 via Helm. There is no OLM/CR path for older versions. Removed the incorrect version gate and the CognosAnalytics CR block that was gating on CPD < 5.3.1.
This reverts commit 759bdd9.
This reverts commit c702932.
Cognos Analytics was never deployed via this GitOps repo before this story. It was newly added as part of MASCORE-15507 for CPD 5.3.1+ only via Helm. No OLM path is needed. - Remove 00-ibm-cognos-analytics_Subscription.yaml entirely - Remove version gate and CognosAnalytics CR block from 01-ibm-cognos-analytics-cr.yaml — Helm install job always applies
The Ibmcpd CR spec.imagePullSecret was set to 'ibm_entitlement_key' (underscores) instead of the actual Kubernetes Secret name 'ibm-entitlement-key' (dashes). This caused the CP4D operator's Ansible playbook to fail to pull Common Services images, resulting in: 'Fail to wait for commonservice CR to reach Succeeded status' Fixes MASCORE-15507
… OutOfSync loop
When only an ArgoCD Hook Job exists in a chart (no non-hook resources),
ArgoCD enters an OutOfSync/Progressing loop after the job deletes itself
(HookSucceeded,HookFailed delete policy). The app shows OutOfSync because
ArgoCD sees zero managed resources.
Add a stable ConfigMap (sync-wave 091, before the hook job at 092) to
WML, Spark, SPSS, Cognos Analytics and WSL 5.3.1 Helm branches so
ArgoCD always has a persistent resource to track.
Fixes: wml.noble8.inst02, spark.noble8.inst02, spss.noble8.inst02
cognos.noble8.inst02, wsl.noble8.inst02 OutOfSync/Progressing
… WSL CI verify_chart_readme_tables.py requires every resource kind in a chart to be documented in the README. The persistent ConfigMap added to the CPD 5.3.1 Helm branches was not yet listed. Fixes CI lint failure: 'resource kind not documented in README table: ConfigMap'
rbinns
left a comment
rbinns
left a comment
There was a problem hiding this comment.
We are deploying CPD as an instance application in cluster scoped mode. If we have 2 instances on the same cluster, these are going to conflict with on another. We will have 2 instances trying to manage cluster scoped resources and will have race conditions on this.
There was a problem hiding this comment.
This pull secret change affects all of the different versions not just 531. Why was this change made?
There was a problem hiding this comment.
This is a bug fix that applies to all CPD versions, not just 5.3.1. The Kubernetes Secret is always created as ibm-entitlement-key (with dashes) — confirmed in both the instance and operators namespaces
There was a problem hiding this comment.
This app/template should be scoped to the 5.3.1 release only - currently will run with any version installed
There was a problem hiding this comment.
The ArgoCD Application is already gated at >= 5.3.1 in 120-ibm-cognos-analytics-app.yaml but the template itself was missing the version gate. Since Cognos Analytics has no OLM path and is 5.3.1 Helm only, wrapped the entire 01-ibm-cognos-analytics-cr.yaml with {{- if not (lt (semver "5.3.1" | ...) 0) }} so it renders nothing for CPD < 5.3.1
There was a problem hiding this comment.
What is this file used for? This was only for OLM previously. Creating an OLM file that doesn't do anything isn't going to be helpful.
Here is exception: Could use this config map for lookup of values for versions later on rather than hardcoding the versions into the helm chart deploy.
There was a problem hiding this comment.
We are continuing to support the older versions of the CPD (as long as IBM does). removal of parameters is going to cause issues.
There was a problem hiding this comment.
Fixed. Restored spss_install_plan, canvasbase_channel, and canvasbase_install_plan to the app template, and restored both OLM Subscriptions (ibm-cpd-spss + ibm-cpd-canvasbase) with installPlanApproval to the chart — all gated behind cpd_product_version < 5.3.1 so older CPD versions are fully supported.
Restored the 3 missing params and the canvasbase OLM Subscription — only active for CPD < 5.3.1, no impact on newer versions.
There was a problem hiding this comment.
This comment applies for cognos, spark, spss, wml, wsl.
These items are not correctly controlled for version. if the version specified is >= 5.3.1 then we will always install 5.3.1 as the helm chart versions being deployed hard code the versions.
to make this more tolerant going forward, should probably include the app versions in gitops
There was a problem hiding this comment.
Just to make sure I captured this - we should not be using cluster-scoped deploys for this. The resources created will then become managed by multiple deploys of this.
There was a problem hiding this comment.
I don't believe that using the helm command in a kubernetes job is the pattern we want to use for deployment of helm charts. We should be creating argo apps below the instance set that deploy the helm charts - for example https://argo-cd.readthedocs.io/en/latest/user-guide/helm/
These apps should be conditions on 5.3.1 release (for now). Versions should be handled as well within this so that we deploy the correct charts at the correct version (i.e. cognos etc).
The others CP4D deployment should be excluded when version < 5.3.1 (i.e. the whole directory so these conditions need to move to root applications .
…esource to cluster-scoped
Three fixes across all 5 CPD >= 5.3.1 native ArgoCD Helm App templates:
1. Version lookup from olm-utils-cm
- Replace all hardcoded targetRevision strings with values read at
render time from the olm-utils-cm ConfigMap (deployed by wave 110).
- Uses Helm lookup() with per-component fallback defaults so helm
template dry-runs (where the cluster is not reachable) still work.
- Components resolved: wml, ibm_redis_cp, analyticsengine, canvasbase,
spss, opencontent_opensearch, ccs, datarefinery, ws_runtimes, ws,
cognos_analytics
2. SkipDryRunOnMissingResource moved to cluster-scoped apps
- Cluster-scoped charts install CRDs; ArgoCD needs to skip the dry-run
for those apps on first deploy when the CRDs don't exist yet.
- Namespace-scoped apps no longer carry this option — they run after the
CRDs are established by the preceding cluster-scoped wave.
3. OLM Subscription files unchanged
- Existing files gated < 5.3.1 are correct and intentional; no changes.
…ervices
Previously, Helm chart targetRevision values for WML, WSL, Spark, SPSS
and Cognos Analytics were resolved only from olm-utils-cm (cluster
lookup) or hardcoded defaults. This meant operators could not pin or
override versions through the gitops config repo — any CPD version
>= 5.3.1 would always get the 5.3.1 chart versions baked into the
ConfigMap at install time.
Resolution priority is now (highest first):
1. Explicit gitops config value (e.g. ibm_wml.wml_chart_version)
2. olm-utils-cm ConfigMap lookup (auto, from wave 110-ibm-cp4d)
3. Hardcoded default (dry-run safety net only)
New optional values added to each service values.yaml:
WML: wml_chart_version, redis_chart_version
Spark: spark_chart_version
SPSS: spss_chart_version, canvasbase_chart_version
WSL: ws_chart_version, ccs_chart_version, datarefinery_chart_version,
ws_runtimes_chart_version, opensearch_chart_version
Cognos: cognos_chart_version (under ibm_cp4d)
All values default to empty string so existing deployments are
unaffected — the olm-utils-cm lookup continues to provide the version
automatically when no explicit value is set.
The cpd-sa-helm-cluster-role-{instance_id} ClusterRole and its
ClusterRoleBinding were created to give the cpd-sa ServiceAccount
cluster-level permissions (CRD management, escalate on ClusterRoles,
SCC use) so it could run 'helm install' for the CPD 5.3.1+ service
charts inside a K8s Job pod.
Those Helm Job pods are now gone — all CPD >= 5.3.1 service installs
are handled by native ArgoCD Applications which run as ArgoCD itself
(already has cluster admin). cpd-sa is no longer used for Helm installs.
Leaving this ClusterRole in place:
- Creates an over-privileged escalate-capable ClusterRole with no consumer
- Violates least-privilege (escalate + CRD delete + SCC use granted unnecessarily)
- Was flagged by rbinns as a PR blocker
ClusterRole and ClusterRoleBinding removed. The cpd-sa ServiceAccount
and its existing namespace-scoped Role (admin RoleBinding to the
instance namespace) are unaffected.
The {{- else }} branches containing batch/v1 Job objects (the old
helm-install shell-script approach) and their stub ConfigMap anchors
were still present in the >= 5.3.1 paths of these instance chart
templates. These are now unreachable dead code — the root-app
ibm-mas-instance-root never creates an ArgoCD Application pointing
at these instance charts for CPD >= 5.3.1.
Removed from:
- 120-ibm-wml/templates/02-ibm-wml-cr.yaml
- 120-ibm-spark/templates/01-ibm-spark-cr.yaml
- 120-ibm-spss/templates/01-ibm-spss-cr.yaml
- 120-ibm-wsl/templates/01-ibm-wsl-cr.yaml
- 120-ibm-cognos-analytics/templates/01-ibm-cognos-analytics-cr.yaml
Each file now contains only the OLM path gated behind
{{- if lt ... 5.3.1 }} — rendering nothing for CPD >= 5.3.1.
For CPD >= 5.3.1 the native ArgoCD Helm App (wave 122) installs the WML Helm chart but there is no mechanism to confirm the WmlBase CR reaches Completed status and to surface WML credentials to AWS Secrets Manager. This commit adds: - instance-applications/120-ibm-wml/templates/03-ibm-wml-post-verify.yaml A Job (gated >= 5.3.1) that polls WmlBase wml-cr until wmlStatus=Completed, then writes url/username/password to SM path <account>/<cluster>/<instance>/wml-cp4d. RBAC (SA, Role, RoleBinding) is scoped to cpd_operators_namespace and cpd_instance_namespace following the WSL post-verify pattern. - root-applications/ibm-mas-instance-root/templates/120-ibm-wml-app.yaml Wave-123 ArgoCD Application (wml-post-verify.<cluster>.<instance>) using the AVP plugin to inject SM credentials into the post-verify Job. Renders only for CPD >= 5.3.1 (inside the existing $is531 block). - instance-applications/120-ibm-wml/values.yaml Added account_id, region_id, cluster_id, instance_id, sm_aws_access_key_id, sm_aws_secret_access_key, cli_image_repo keys consumed by the post-verify job.
The cognos-analytics Helm chart renders a CAService CR. The operator immediately mutates spec.license (adding license: Enterprise) and adds spec.enableInstanaMetricCollection: false. ArgoCD sees these as drift and keeps the app OutOfSync. Add ignoreDifferences for both fields plus ServiceAccount/imagePullSecrets (standard pattern across all service apps). Add RespectIgnoreDifferences to syncOptions so ArgoCD applies the exclusions during sync.
… SPSS (wave 122) SPSS operator at wave 122 tries to create a WS dependency CR of kind ws.cpd.ibm.com/v1beta1.WS. If ws-cluster-scoped runs at wave 123 (after SPSS), the CRD doesn't exist yet and SPSS fails with: 'Failed to find exact match for ws.cpd.ibm.com/v1beta1.WS' Move ws-cluster-scoped from wave 123 to wave 119 so it runs alongside the other cluster-scoped CRD installs, well before SPSS at wave 122.
…124) SPSS operator creates a WS CR and waits for Watson Studio as a dependency. The WSL operator (ws chart) deploys at wave 124, so SPSS namespace-scoped app must be at wave 125 or later to avoid a race condition where SPSS tries to create a WS CR before the WS operator is running. Also update wave comment from 123 to 119 on wsl-cluster section (cosmetic).
The instance-applications/120-ibm-wsl post-verify job (cpd-wsl-post-verify) was only deployed via the CPD < 5.3.1 AVP app path. For CPD >= 5.3.1 the native Helm apps are used for the WS operator/CR, but the post-verify job still needs to run to write the wsl-cp4d and wsl-project-id secrets to AWS SM. Without these secrets, inst02-watsonstudio-system AVP render fails. Wave 126 ensures it runs after wsl (124) and spss (125).
5 participants
Issue
https://jsw.ibm.com/browse/MASCORE-15507
https://jsw.ibm.com/browse/MASCORE-15752
https://jsw.ibm.com/browse/MASCORE-16163
Description
Updates GitOps automation to support CPD 5.3.1 in addition to existing CPD 5.2 and 5.3 support. For CPD 5.3.1+, OLM-based installation is replaced with Helm-based installation using IBM Helm charts.
Summary of changes
CPD 5.3.1 OLM bootstrap (
101-ibm-sync-jobs-cp4d, 110-ibm-cp4d)WML (
120-ibm-wml)WSL (
120-ibm-wsl)Spark (
120-ibm-spark)Cognos Analytics (
120-ibm-cognos-analytics)SPSS Modeler (
120-ibm-spss) — re-added with CPD 5.3.1 supportRe-added chart with OLM Subscription + SPSSModeler CR for CPD < 5.3.1
Added Helm-based install job for CPD 5.3.1+ installing canvasbase (v12.1.0, dependency) then spss (v12.1.0)
Added ArgoCD Application template 120-ibm-spss-app.yaml
Post-sync job (
121-ibm-post-sync-job-cp4d-services)Skip OLM-specific CCS/OpenSearch post-sync patching for CPD 5.3.1+
Bumped $_job_version to v2
RBAC
Test Results
Tested on noble8 cluster with CPD 5.3.1 (inst02):
wml.noble8.inst02 → WmlBase/wml-cr Completed 100%
wsl.noble8.inst02 → Synced Healthy
spark.noble8.inst02 → Synced Healthy
cognos.noble8.inst02 → CAService/ca-addon-cr Completed 100%
spss.noble8.inst02 → canvasbase + spss Helm charts installed, canvasbase Completed 100%