Kubernetes upgrade and rollback

Upgrade planning, failed-upgrade recovery, and rollback for independent application and data releases.

Short glossary
  • Helm release: an installed and independently managed resource set from one chart.
  • Secret object: a Kubernetes resource referenced for sensitive values; the delivery charts do not own the values themselves.
  • PersistentVolumeClaim (PVC): a storage request used by a bundled data component.
  • NetworkPolicy: rules governing permitted network connections for application Pods.
  • Container Storage Interface (CSI) and Container Network Interface (CNI) capabilities are provided by the customer cluster.

Upgrade, failed-upgrade recovery, and rollback

Download the new release materials and check checksums and schema/data compatibility. Preserve local values, current Secrets and PVCs; prepare a backup and verify restoration. Upgrade the application and data release separately with Helm.

  • Do not blindly use --reuse-values: migrate local settings to the new chart schema and validate lint/template with its image overlay.
  • Do not repeat initial activation with another identity; do not rotate JWT/metering/registry Secrets outside their established procedure.
  • Before a data release upgrade separately check store version compatibility, backup/restore and StorageClass; rerun storage preflight if needed.
  • Helm rollback reverts release resources, not database migrations, PVC contents or external service data.

Upgrade and rollback procedure

  1. 1. Validate the new release

    Check the new archive, migrate local values, render and compare manifests with the existing installation. Preserve a backup and previous revision.

    bash
    sha256sum -c SHA256SUMS
helm --kube-context "$MA_CONTEXT" history massaccess -n "$MA_NAMESPACE"
helm lint charts/massaccess --strict -f work/app-values.yaml -f values/massaccess-images.json -f work/platform-palette-values.yaml
helm template massaccess charts/massaccess -n "$MA_NAMESPACE" -f work/app-values.yaml -f values/massaccess-images.json -f work/platform-palette-values.yaml > work/app-rendered.yaml

  2. 2. Upgrade the application

    Refresh the palette and apply the new massaccess version. Upgrade the data release separately using its chart, values and image overlay.

    bash
    python3 bin/palette-delivery.py --manifest activation/manifest.json --helm work/platform-palette-values.yaml
helm --kube-context "$MA_CONTEXT" upgrade massaccess charts/massaccess -n "$MA_NAMESPACE" \
  -f work/app-values.yaml -f values/massaccess-images.json -f work/platform-palette-values.yaml --rollback-on-failure --wait --timeout 15m

  3. 3. Select a rollback revision

    On failure inspect history/status and identify a compatible previous application revision. Confirm data compatibility first.

    bash
    helm --kube-context "$MA_CONTEXT" history massaccess -n "$MA_NAMESPACE"
helm --kube-context "$MA_CONTEXT" status massaccess -n "$MA_NAMESPACE"

  4. 4. Apply rollback

    Replace <previous-revision> with a revision from history. A data release rollback is a separate decision requiring store compatibility and restore checks.

    bash
    helm --kube-context "$MA_CONTEXT" rollback massaccess <previous-revision> -n "$MA_NAMESPACE" --wait --timeout 15m

  5. 5. Verify the result

    Repeat Helm tests and application checks; verify readiness, restart counts and preservation of test data.

    bash
    helm --kube-context "$MA_CONTEXT" status massaccess -n "$MA_NAMESPACE"
helm --kube-context "$MA_CONTEXT" test massaccess -n "$MA_NAMESPACE" --logs
kubectl --context "$MA_CONTEXT" -n "$MA_NAMESPACE" get deployment,statefulset,pod,service,job,pvc