Generated secrets are regenerated on every GitOps render; ignoreDifferences does not prevent it

[MartinHell]

Summary

The chart's generated secrets cannot survive a GitOps render, and ArgoCD's
ignoreDifferences does not protect them. This blocks managing PatchMon with
ArgoCD at all — verified empirically on k8s-sto2, twice.

The mechanism

templates/secret.yaml seeds JWT_SECRET, SESSION_SECRET and AI_ENCRYPTION_KEY
with randAlphaNum, keeping them stable across upgrades via a lookup on the existing
Secret. lookup only resolves under helm upgrade/install. ArgoCD renders with
helm template, where it always returns empty — so every render mints three new values:

$ helm template patchmon patchmon/patchmon --version 0.3.6 -f values.yml | grep JWT_SECRET
  JWT_SECRET: "fAWUcEVwQRqDHzWPsBZikBELm..."      # render 1
  JWT_SECRET: "GkWMNDFpxZ0nwVvA3HUPRq7HQ..."      # render 2, same inputs

@BryanR77 documented this in #7 and added patchmon.session.secret /
patchmon.encryption.aiKey so the values can be pinned — but pinning them means
putting three secrets in git in plaintext, which is not a fix for GitOps users.

ignoreDifferences does not save you

The obvious mitigation is ignoreDifferences on the Secret's /data plus
RespectIgnoreDifferences=true. It does not work. Tested by syncing only that one
resource and hashing before/after:

Sync options	Result
ServerSideApply=true + RespectIgnoreDifferences=true	regenerated
RespectIgnoreDifferences=true (client-side apply)	regenerated

Both times all three keys were replaced. Restored from a pre-made copy of the Secret;
no pod restarted, so nothing was lost — but on an unattended selfHeal: true sync this
would be silent and permanent.

Why it matters more than it looks

• JWT_SECRET / SESSION_SECRET — every session invalidated on each sync.
• AI_ENCRYPTION_KEY — anything already encrypted with the old key becomes
  unrecoverable. There is no recovery path once the old value is gone.

A GitOps user with automated sync would rotate all three on every reconcile without
any signal that it happened.

Proposed fix

Support consuming these from an existing Secret, exactly mirroring the pattern the
chart already has for OIDC (patchmon.oidc.clientSecretFromSecret), which works well:

patchmon:
  existingSecret:
    name: ""          # when set, the chart does NOT render its own Secret
    keys:
      jwt: JWT_SECRET
      session: SESSION_SECRET
      aiKey: AI_ENCRYPTION_KEY

When name is set:

• templates/secret.yaml skips those keys entirely (or skips rendering altogether if
  OIDC is also external);
• deployment-server.yaml points each secretKeyRef at the named Secret.

That lets the secrets live in a real secret store (we use OpenBao + External Secrets)
and keeps them out of both git and the chart's render, which makes the manifest
deterministic and the ArgoCD diff clean.

Notes

• The lookup behaviour is correct for helm upgrade users and should stay as the
  default — this is purely an additional opt-in path.
• Worth a README section on GitOps deployment once implemented, since the failure mode
  is silent and expensive.
• Related: Fix postgres StatefulSet crash under readOnlyRootFilesystem #7 (added the value knobs), Expose OIDC_TRUST_UNVERIFIED_EMAIL and document the 2.1.x verified-email requirement #8 (OIDC_TRUST_UNVERIFIED_EMAIL).

[MartinHell]

The blast radius is wider than the variable name suggests

Checked the environment variables reference. AI_ENCRYPTION_KEY is not AI-specific — it is the application's general at-rest encryption key:

32+ byte random secret used to encrypt AI provider keys, bootstrap tokens, OIDC client secrets, and notification destination credentials.

If you rotate this value, every encrypted secret in the database becomes unreadable.

So a GitOps sync that regenerates it does not just break an optional AI integration. It takes out:

• bootstrap tokens — agent enrollment, i.e. the fleet
• notification destination credentials (webhooks, SMTP)
• the DB-stored OIDC client secret

SESSION_SECRET is also worth naming correctly: it is not a cookie signer but a fallback encryption key for the same data, kept "for backward compatibility with early 1.x installs." Both keys therefore need the same protection, and the proposed existingSecret should cover all three.

Related footgun the docs call out, relevant to anyone using an external database with rotating credentials:

If neither AI_ENCRYPTION_KEY nor SESSION_SECRET is set, encryption relies on the database connection string, which means any change to the DB host, port, or password rotates the encryption key and invalidates every encrypted value.

The chart always sets both, so chart users are not exposed to that — but it is another argument for making these explicitly pinnable rather than generated, and probably worth a line in the README when this lands.