[minor] Update DB2 Audit logs to S3 buckets using CRON Job

instance-applications/120-ibm-db2u-database/README.md

@@ -4,30 +4,34 @@ Create a Db2u database for a MAS app.
 
 <!--docs-include-start-->
 
+## Overview
 
-Contains a presync hook (`00-presync-await-crd_Job.yaml`) that ensures we wait for the db2uclusters CRD to be installed before attempting to sync.
+This chart deploys and configures a Db2u database instance for use by a MAS application. It manages the full lifecycle of the database including TLS certificates, storage, backup, audit log extraction, and HADR services.
+
+Contains a presync hook (`00-presync-await-crd_Job.yaml`) that ensures we wait for the `db2uclusters` CRD to be installed before attempting to sync.
 
 Contains a job that runs last (`05-postsync-setup-db2_Job.yaml`). This registers the `${ACCOUNT_ID}/${CLUSTER_ID}/${MAS_INSTANCE_ID}/db2/${DB2_INSTANCE_NAME}/config` secret in the **Secrets Vault** used to share some information that is generated at runtime with other ArgoCD Applications. This job also performs some special configuration steps that are required if the Db2u database is intended for use by the Manage MAS Application.
 
 ## Resources Created
 
-| Resource Type | Resource Name | Namespace | Condition | Installed By |
-|--------------|---------------|-----------|-----------|--------------|
-| `StorageClass` | Db2 storage class definitions | DB2 application namespace / cluster | When storage classes are managed by this chart | `application_admin_role` |
-| `ServiceAccount` | Pre/post-sync DB2 job service accounts | DB2 application namespace | Always | `application_admin_role` |
-| `Role` | Pre/post-sync DB2 job roles | DB2 application namespace and related namespaces | Always | `application_admin_role` |
-| `RoleBinding` | Pre/post-sync DB2 job role bindings | DB2 application namespace and related namespaces | Always | `application_admin_role` |
-| `Issuer` | DB2 TLS issuers | DB2 application namespace | Always | `application_admin_role` |
-| `Certificate` | DB2 TLS certificates | DB2 application namespace | Always | `application_admin_role` |
-| `Db2uInstance` | Db2u instance CR | DB2 application namespace | Always | `application_admin_role` |
-| `CronJob` | Db2 backup cron job | DB2 application namespace | When backups are enabled | `application_admin_role` |
-| `ConfigMap` | Db2 script/config maps | DB2 application namespace | Always | `application_admin_role` |
-| `Route` | Db2 TLS route | DB2 application namespace | When route exposure is enabled | `application_admin_role` |
-| `Service` | Db2 services, including HADR services | DB2 application namespace | Always | `application_admin_role` |
-| `Service` | Private NLB service | DB2 application namespace | When `private_nlb.enabled` is true | `application_admin_role` |
-| `Secret` | Post-sync DB2 generated secret | DB2 application namespace | Always | `application_admin_role` |
-| `NetworkPolicy` | HADR network policy | DB2 application namespace | When HADR is enabled | `application_admin_role` |
-| `Job` | Pre/post-sync DB2 setup jobs | DB2 application namespace | Always | `application_admin_role` |
+| Resource Type | Resource Name | Namespace | Condition                                                   | Installed By |
+|--------------|---------------|-----------|-------------------------------------------------------------|--------------|
+| `StorageClass` | Db2 storage class definitions | DB2 application namespace / cluster | When storage classes are managed by this chart              | `application_admin_role` |
+| `ServiceAccount` | Pre/post-sync DB2 job service accounts | DB2 application namespace | Always                                                      | `application_admin_role` |
+| `Role` | Pre/post-sync DB2 job roles | DB2 application namespace and related namespaces | Always                                                      | `application_admin_role` |
+| `RoleBinding` | Pre/post-sync DB2 job role bindings | DB2 application namespace and related namespaces | Always                                                      | `application_admin_role` |
+| `Issuer` | DB2 TLS issuers | DB2 application namespace | Always                                                      | `application_admin_role` |
+| `Certificate` | DB2 TLS certificates | DB2 application namespace | Always                                                      | `application_admin_role` |
+| `Db2uInstance` | Db2u instance CR | DB2 application namespace | Always                                                      | `application_admin_role` |
+| `CronJob` | Db2 backup cron job | DB2 application namespace | When backups are enabled (`db2_backup_bucket_name` set)     | `application_admin_role` |
+| `CronJob` | Db2 audit extract cron job | DB2 application namespace | When backup bucket is enabled (`db2_audit_bucket_name` set) | `application_admin_role` |
+| `ConfigMap` | Db2 script/config maps | DB2 application namespace | Always                                                      | `application_admin_role` |
+| `Route` | Db2 TLS route | DB2 application namespace | When route exposure is enabled                              | `application_admin_role` |
+| `Service` | Db2 services, including HADR services | DB2 application namespace | Always                                                      | `application_admin_role` |
+| `Service` | Private NLB service | DB2 application namespace | When `private_nlb.enabled` is true                          | `application_admin_role` |
+| `Secret` | Post-sync DB2 generated secret | DB2 application namespace | Always                                                      | `application_admin_role` |
+| `NetworkPolicy` | HADR network policy | DB2 application namespace | When HADR is enabled                                        | `application_admin_role` |
+| `Job` | Pre/post-sync DB2 setup jobs | DB2 application namespace | Always                                                      | `application_admin_role` |
 
 ## Configuration
 
@@ -207,4 +211,52 @@ The NLB is created independently for each instance (e.g. facilities, manage) usi
 
 If `private_nlb.enabled: true` and either `subnet_ids` or `allowed_cidrs` is
 empty, Helm will fail immediately with a clear error message before rendering
-any resources. This prevents a broken or unrestricted NLB from being deployed..
+any resources. This prevents a broken or unrestricted NLB from being deployed.
+
+## Prerequisites
+
+- The `db2uclusters` CRD must be available on the cluster (ensured by the presync hook).
+- An S3-compatible backup bucket must be provisioned when backup or audit log upload is enabled.
+- Secrets for S3 credentials, cluster domain, and Secrets Manager access must be pre-populated in the Secrets Vault before sync.
+
+## Examples
+
+### Minimal deployment
+
+```yaml
+db2_namespace: db2u-manage
+db2_instance_name: db2u-manage
+db2_dbname: BLUDB
+db2_version: "11.5.9.0"
+db2_tls_version: "1.2"
+db2_table_org: ROW
+mas_application_id: manage
+cluster_domain: "<path:secrets/path:cluster_domain>"
+```
+
+### With backup and audit log upload enabled
+
+```yaml
+db2_namespace: db2u-manage
+db2_instance_name: db2u-manage
+db2_dbname: BLUDB
+db2_backup_bucket_name: "<path:secrets/path:bucket_name>"
+db2_backup_bucket_endpoint: "<path:secrets/path:bucket_endpoint>"
+db2_backup_bucket_access_key: "<path:secrets/path:access_key>"
+db2_backup_bucket_secret_key: "<path:secrets/path:secret_key>"
+auto_backup: true
+mas_application_id: manage
+cluster_domain: "<path:secrets/path:cluster_domain>"
+```
+
+## Troubleshooting
+
+- **Presync job stuck** — verify the `db2uclusters` CRD is installed by the DB2U operator before the ArgoCD sync wave reaches this chart.
+- **Postsync job failing** — check the job logs in the DB2 namespace; common causes are missing S3 credentials or an unreachable backup bucket.
+- **Audit CronJob not running** — confirm `db2_backup_bucket_name` is set and the instance name does not contain `sdb` (audit cron is disabled for SDB instances).
+- **AWS CLI missing** — `db2AuditExtract.sh` will install the AWS CLI automatically on first run via `curl`/`unzip` into `/mnt/backup/`.
+
+## Related Documentation
+
+- [Instance Base Values Reference](../../docs/reference/instance-base-values.md)
+- [IBM Db2u Operator Documentation](https://www.ibm.com/docs/en/db2/11.5)

instance-applications/120-ibm-db2u-database/files/CopyDBScripts.sh

@@ -35,6 +35,7 @@ cp -rp reorgTablesIndexesInplace.sh ${INSTHOME}/bin/
 cp -rp extract_authorization.sh  ${INSTHOME}/bin
 cp -rp HADRMON.sh ${INSTHOME}/bin
 cp -rp auditExtractUpload.sh ${INSTHOME}/bin/
+cp -rp db2AuditExtract.sh ${INSTHOME}/bin/
 
 
 echo -e "\nCopying the file to bin/ITCS104 directory under Instance Home . . ."

instance-applications/120-ibm-db2u-database/files/db2AuditExtract.sh

@@ -0,0 +1,220 @@
+#!/bin/bash
+
+# ----------------------------------------------------------------------------
+#% Script Name  : db2AuditExtract.sh
+#% Description  : Archive and extract DB2 audit logs as DEL/ASC files, upload
+#%                each *.del file to S3, then remove source files.
+#%
+#%  **  THIS MUST BE RUN AS THE DB2 INSTANCE OWNER (db2inst1)  **
+#%
+#%  USAGE:  db2AuditExtract.sh <application_name> [dbname]
+#%
+#%  Steps:
+#%   1.  mkdir /tmp/auditarchive
+#%   2.  rm /tmp/auditarchive/*.del
+#%   3.  db2audit flush
+#%   4.  db2audit archive database BLUDB to /tmp/auditarchive
+#%   5.  db2audit archive to /tmp/auditarchive  (instance log)
+#%   6.  db2audit extract delasc to /tmp/auditarchive  (database log)
+#%   7.  db2audit extract delasc to /tmp/auditarchive  (instance log)
+#%   8.  Copy db2audit.db.BLUDB.log.0.20*   from /mnt/blumeta0/audit → /tmp/auditarchive
+#%   9.  Copy db2audit.instance.log.0.20*   from /mnt/blumeta0/audit → /tmp/auditarchive
+#%  10.  Upload ALL files from /tmp/auditarchive to S3
+#%  11.  rm -rf /tmp/auditarchive
+#%  12.  Delete the *.log.0.20* source files from /mnt/blumeta0/audit
+#%  13.  (Conditional) Delete pre-existing *.del files from /mnt/blumeta0/audit
+#%       (prints list before deleting)
+# ----------------------------------------------------------------------------
+
+set -eo pipefail
+
+# ── Logging helper ─────────────────────────────────────────────────────────
+log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*"; }
+
+# ── Validate input ─────────────────────────────────────────────────────────
+APP_NAME="${1:-}"
+if [ -z "${APP_NAME}" ]; then
+  echo "ERROR :: Usage: $0 <application_name>"
+  exit 1
+fi
+
+# ── Constants ──────────────────────────────────────────────────────────────
+ARCHIVE_DIR="/tmp/auditarchive"
+AUDIT_BASE="/mnt/blumeta0/audit"
+DBNAME="${2:-BLUDB}"  # Passed as 2nd arg from CronJob; falls back to BLUDB
+DATE=$(date +"%Y-%m-%d")
+DT=$(date +"%Y-%m-%d_%H%M%S")
+DELETE_AUDIT_BASE_DEL="false"   # Set to "true" to delete *.del files from ${AUDIT_BASE} (step 13)
+
+# ── Source DB2 environment (DB2 profile uses unbound vars — disable nounset) ─
+set +u
+. "${HOME}/sqllib/db2profile"
+set -u
+
+# ── Load COS/S3 credentials (CONTAINER, SERVER, PARM1, PARM2) ─────────────
+. /mnt/backup/bin/.PROPS
+
+# ── Install AWS CLI if not already present ────────────────────────────────
+AWS_CLI="/mnt/backup/aws/dist/aws"
+log "INFO  :: Checking AWS CLI at ${AWS_CLI}"
+if ! "${AWS_CLI}" --version >/dev/null 2>&1; then
+  log "INFO  ::   Not found — installing AWS CLI to /mnt/backup/"
+  cd /mnt/backup
+  curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
+  unzip awscliv2.zip -d /mnt/backup/
+  log "INFO  ::   AWS CLI installed at ${AWS_CLI}"
+else
+  log "INFO  ::   AWS CLI already present: $(${AWS_CLI} --version 2>&1)"
+fi
+export AWS_ACCESS_KEY_ID="${PARM1}"
+export AWS_SECRET_ACCESS_KEY="${PARM2}"
+export AWS_DEFAULT_REGION=$(echo "${SERVER}" | sed 's|.*s3\.\([^.]*\)\.amazonaws.*|\1|')
+
+S3_TARGET="s3://${CONTAINER}/audit_logs/${APP_NAME}/${DATE}/"
+
+# ── Ensure db2audit is always restarted on exit ────────────────────────────
+trap 'log "INFO  :: Restarting db2audit after job"; db2audit start >/dev/null 2>&1 || true' EXIT
+
+# ── Banner ─────────────────────────────────────────────────────────────────
+log "INFO  :: ============================================================"
+log "INFO  :: DB2 Audit Extract — ${DT}"
+log "INFO  :: Application : ${APP_NAME} | Database : ${DBNAME}"
+log "INFO  :: Work dir    : ${ARCHIVE_DIR}"
+log "INFO  :: S3 target   : ${S3_TARGET}"
+log "INFO  :: ============================================================"
+
+# ============================================================================
+# 1–2.  Prepare working directory
+# ============================================================================
+log "INFO  :: [1] mkdir ${ARCHIVE_DIR}"
+mkdir -p "${ARCHIVE_DIR}"
+
+log "INFO  :: [2] Removing any stale .del files from ${ARCHIVE_DIR}"
+rm -f "${ARCHIVE_DIR}"/*.del 2>/dev/null || true
+
+# ============================================================================
+# 3.  Flush in-memory audit buffer to disk
+# ============================================================================
+log "INFO  :: [3] db2audit flush"
+db2audit flush
+
+# ============================================================================
+# 4.  Archive the database audit log to /tmp/auditarchive
+# ============================================================================
+log "INFO  :: [4] db2audit archive database ${DBNAME} to ${ARCHIVE_DIR}"
+db2audit archive database "${DBNAME}" to "${ARCHIVE_DIR}"
+
+# ============================================================================
+# 5.  Archive the instance audit log to /tmp/auditarchive
+# ============================================================================
+log "INFO  :: [5] db2audit archive to ${ARCHIVE_DIR}  (instance log)"
+db2audit archive to "${ARCHIVE_DIR}"
+
+# ============================================================================
+# 6.  Extract archived database log → *.del
+# ============================================================================
+log "INFO  :: [6] db2audit extract delasc (database log)"
+DB_LOGS=$(ls "${ARCHIVE_DIR}"/db2audit.db."${DBNAME}".log.0.* 2>/dev/null || true)
+if [ -z "${DB_LOGS}" ]; then
+  log "WARN  ::     No database archive log found in ${ARCHIVE_DIR} — skipping extract"
+else
+  db2audit extract delasc to "${ARCHIVE_DIR}" from files ${DB_LOGS}
+fi
+
+# ============================================================================
+# 7.  Extract archived instance log → *.del
+# ============================================================================
+log "INFO  :: [7] db2audit extract delasc (instance log)"
+INST_LOGS=$(ls "${ARCHIVE_DIR}"/db2audit.instance.log.0.* 2>/dev/null || true)
+if [ -z "${INST_LOGS}" ]; then
+  log "WARN  ::     No instance archive log found in ${ARCHIVE_DIR} — skipping extract"
+else
+  db2audit extract delasc to "${ARCHIVE_DIR}" from files ${INST_LOGS}
+fi
+
+# ============================================================================
+# 8–9.  Copy historical log files from /mnt/blumeta0/audit to /tmp/auditarchive
+# ============================================================================
+log "INFO  :: [8] Copying db2audit.db.${DBNAME}.log.0.20* from ${AUDIT_BASE}"
+cp "${AUDIT_BASE}"/db2audit.db."${DBNAME}".log.0.20* "${ARCHIVE_DIR}/" 2>/dev/null \
+  && log "INFO  ::     Database logs copied" \
+  || log "WARN  ::     No matching db2audit.db.${DBNAME}.log.0.20* files found — skipping"
+
+log "INFO  :: [9] Copying db2audit.instance.log.0.20* from ${AUDIT_BASE}"
+cp "${AUDIT_BASE}"/db2audit.instance.log.0.20* "${ARCHIVE_DIR}/" 2>/dev/null \
+  && log "INFO  ::     Instance logs copied" \
+  || log "WARN  ::     No matching db2audit.instance.log.0.20* files found — skipping"
+
+# ============================================================================
+# 10.  Upload ALL files from /tmp/auditarchive to S3
+# ============================================================================
+log "INFO  :: [10] Uploading all files from ${ARCHIVE_DIR} to ${S3_TARGET}"
+
+ALL_FILES=$(ls "${ARCHIVE_DIR}"/* 2>/dev/null || true)
+if [ -z "${ALL_FILES}" ]; then
+  log "WARN  ::      No files found in ${ARCHIVE_DIR} — nothing to upload"
+else
+  ERRORS=0
+  for F in ${ALL_FILES}; do
+    FILE_NAME=$(basename "${F}")
+    log "INFO  ::      [s3] ${FILE_NAME} → ${S3_TARGET}${FILE_NAME}"
+    "${AWS_CLI}" s3 cp "${F}" "${S3_TARGET}${FILE_NAME}" \
+      && log "INFO  ::           Upload confirmed" \
+      || { log "ERROR ::           Upload FAILED for ${FILE_NAME}"; ERRORS=$((ERRORS + 1)); }
+  done
+  [ ${ERRORS} -gt 0 ] && { log "ERROR :: ${ERRORS} upload(s) failed"; exit 1; }
+  log "INFO  ::      All files uploaded successfully"
+fi
+
+# ============================================================================
+# 11.  Remove /tmp/auditarchive and all its contents
+# ============================================================================
+log "INFO  :: [11] rm -rf ${ARCHIVE_DIR}"
+rm -rf "${ARCHIVE_DIR}"
+log "INFO  ::      Working directory removed"
+
+# ============================================================================
+# 12.  Delete the historical *.log.0.20* source files from /mnt/blumeta0/audit
+# ============================================================================
+log "INFO  :: [12] Removing historical log files from ${AUDIT_BASE}"
+
+for PATTERN in \
+  "${AUDIT_BASE}/db2audit.db.${DBNAME}.log.0.20"* \
+  "${AUDIT_BASE}/db2audit.instance.log.0.20"*
+do
+  for F in ${PATTERN}; do
+    [ -f "${F}" ] || continue
+    log "INFO  ::      [delete] $(basename "${F}")"
+    rm -f "${F}"
+  done
+done
+log "INFO  ::      Historical log files removed"
+
+# ============================================================================
+# 13.  (Optional) Delete pre-existing *.del files from /mnt/blumeta0/audit
+#       — controlled by DELETE_AUDIT_BASE_DEL; prints list before deleting
+# ============================================================================
+log "INFO  :: [13] DELETE_AUDIT_BASE_DEL=${DELETE_AUDIT_BASE_DEL}"
+
+if [ "${DELETE_AUDIT_BASE_DEL}" != "true" ]; then
+  log "INFO  ::      Skipping .del cleanup in ${AUDIT_BASE} (DELETE_AUDIT_BASE_DEL is not true)"
+else
+  AUDIT_DEL_FILES=$(ls "${AUDIT_BASE}"/*.del 2>/dev/null || true)
+  if [ -z "${AUDIT_DEL_FILES}" ]; then
+    log "INFO  ::      No .del files found in ${AUDIT_BASE} — nothing to clean"
+  else
+    log "INFO  ::      The following .del files will be deleted from ${AUDIT_BASE}:"
+    for F in ${AUDIT_DEL_FILES}; do
+      log "INFO  ::        $(basename "${F}")"
+    done
+    rm -f ${AUDIT_DEL_FILES}
+    log "INFO  ::      .del files deleted"
+  fi
+fi
+
+# ── Done ───────────────────────────────────────────────────────────────────
+log "INFO  :: ============================================================"
+log "INFO  :: Audit extraction completed successfully"
+log "INFO  :: S3 target : ${S3_TARGET}"
+log "INFO  :: ============================================================"
+exit 0