Persistence, Backup and Restore

latest-snapshot 5.14 5.13 5.12 5.11 5.10 5.9 5.8 5.7 5.6 5.5

Persistence allows individual members and whole clusters to recover data by persisting map entries, JCache data, and streaming job snapshots on disk. Members can use persisted data to recover from a planned shutdown (including rolling upgrades), a sudden cluster-wide crash, or a single member failure.

This topic assumes that you know about Persistence in Hazelcast. To learn about Persistence, see the Platform documentation.

Backups can be either of the following:

  • Local: Local backups are kept in volume and never moved anywhere.

  • External: External backups are moved into buckets provided by the user.

For a working example, see this tutorial.

Enabling Persistence

Enabling Hazelcast persistence is done with the following configuration.

apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: hazelcast
spec:
  clusterSize: 3
  repository: 'docker.io/hazelcast/hazelcast-enterprise'
  licenseKeySecretName: hazelcast-license-key
  persistence:
    clusterDataRecoveryPolicy: "FullRecoveryOnly"  (1)
    pvc:
      accessModes: ["ReadWriteOnce"]
      requestStorage: 20Gi  (2)
  agent: (3)
    repository: hazelcast/platform-operator-agent
1 Cluster recovery policy.
2 Size of the PersistentVolumeClaim (PVC) where Hazelcast data is persisted.
3 Agent responsible for moving data from the local storage to external buckets. The agent configuration is optional. If you enable persistence and do not pass the agent configuration, Hazelcast Platform Operator uses the latest agent version that is compatible with its version.
Make sure to calculate the total disk space that you will use. The total used disk space may be larger than the size of in-memory data, depending on how many backups you take.
If you want to enable persistence for a map or cache you must enable it in the Map or Cache CRs by setting persistenceEnabled: true. For more information, see Configuring Map and Configuring Cache.

Triggering Local Backups

You can take local backups using the HotBackup custom resource. Local backups are kept in volume and are not moved anywhere.

apiVersion: hazelcast.com/v1alpha1
kind: HotBackup
metadata:
  name: hot-backup
spec:
  hazelcastResourceName: hazelcast

Triggering External Backups

In some cases, keeping the data only at PVC and restoring the data from it is not enough. For example, moving data between two Kubernetes clusters. You can use external storages so they will become portable.

When persistence is enabled Hazelcast pod will start with a sidecar agent which will upload the backups into an external bucket provided by the user. For external storage, AWS S3, GCP Bucket and Azure Blob storage options are supported.

To trigger an external backup, you need to configure a bucket URI and a secret to tell Hazelcast where to store backup data and how to authenticate.

  • AWS

  • GCP

  • Azure

External backup in AWS S3:

apiVersion: hazelcast.com/v1alpha1
kind: HotBackup
metadata:
  name: hot-backup
spec:
  hazelcastResourceName: hazelcast
  bucketURI: "s3://bucket-name/path/to/folder" (1)
  secretName: "secret-aws-s3" (2)
For further information about accessing resources on different cloud providers, see Authorization Methods to Access Cloud Provider Resources.

External backup in GCP Bucket:

apiVersion: hazelcast.com/v1alpha1
kind: HotBackup
metadata:
  name: hot-backup
spec:
  hazelcastResourceName: hazelcast
  bucketURI: "gs://bucket-name/path/to/folder" (1)
  secretName: "secret-gcp-bucket" (2)
For further information about accessing resources on different cloud providers, see Authorization Methods to Access Cloud Provider Resources.

External backup in Azure Blob:

apiVersion: hazelcast.com/v1alpha1
kind: HotBackup
metadata:
  name: hot-backup
spec:
  hazelcastResourceName: hazelcast
  bucketURI: "azblob://bucket-name/path/to/folder" (1)
  secretName: "secret-azure-blob" (2)
For further information about accessing resources on different cloud providers, see Authorization Methods to Access Cloud Provider Resources.
1 The bucket URI where backup data will be stored in
2 Name of the secret with credentials for accessing the given Bucket URI.

Scheduling Backups

You can schedule backups using the schedule and hotBackupTemplate fields of the CronHotBackup resource. For more information about the CronHotBackup resource, see the API Reference.

apiVersion: hazelcast.com/v1alpha1
kind: CronHotBackup
metadata:
  name: cron-hot-backup
spec:
  schedule: "* 0-23/6 * * *"
  hotBackupTemplate:
    spec:
      hazelcastResourceName: hazelcast

The schedule field takes a valid cron expression. For example, you can configure the following scheduled backups:

30 10 * * *

At 10:30 AM every day

0, 0, 1,15,25 * *

On 1st, 15th, and 25th of each month at midnight

@monthly

On the first day of each month at midnight

For a full list of supported expressions, see the library documentation.

Checking the Status of a Backup

To check the status of a local backup, run the following command:

kubectl get hotbackup hot-backup

The status of the backup is displayed in the output.

NAME         STATUS
hot-backup   Success

Restoring from Local Backups

To restore a cluster from local backups, you can directly reapply the Hazelcast resource, which gives the cluster access to the PVCs that contain the persisted data. This will restore the Hazelcast cluster from existing hot-restart folders.

Or, to restore from local backups that you have taken using the HotBackup resource, give the HotBackup resource name in the restore configuration. For the restore to work correctly, make sure the status of the HotBackup resource is Success.

When this restore mechanism is used, the Restore Agent container is deployed with the Hazelcast container in the same Pod. The agent starts as an initContainer before the Hazelcast container.

apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: hazelcast (1)
spec:
  clusterSize: 3
  repository: 'docker.io/hazelcast/hazelcast-enterprise'
  licenseKeySecretName: hazelcast-license-key
  persistence:
    clusterDataRecoveryPolicy: "FullRecoveryOnly"
    pvc:
      accessModes: ["ReadWriteOnce"]
      requestStorage: 20Gi
    restore:
      hotBackupResourceName: hot-backup (2)
  agent: (3)
    repository: hazelcast/platform-operator-agent
1 Hazelcast custom resource name for both backup and restore CRs must be the same, otherwise, the restore fails.
2 HotBackup resource name used for the restore. The backup folder name will be the name you provide here.
3 Agent responsible for restoring data from the local storage. The agent configuration is optional. If you give restore under persistence and do not pass the agent configuration, Hazelcast Platform Operator uses the latest agent version that is compatible with its version.
You can use a local backup only once to restore a cluster. We recommend you backup externally if you need to persistently restore a backup across the clusters.

Restoring from External Backups

To restore a cluster from external backups, you can either set up the bucket configuration or give the HotBackup resource name that you used to trigger the external backup. In either case, the backup is restored from the external bucket.

When this restore mechanism is used, the Restore Agent container is deployed with the Hazelcast container in the same Pod. The agent starts as an initContainer before the Hazelcast container.

If you have not created the secret, you must do so in the same way as in Triggering External Backups.

  • Bucket Configuration

  • HotBackup resource name

apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: hazelcast
spec:
  clusterSize: 3
  repository: 'docker.io/hazelcast/hazelcast-enterprise'
  licenseKeySecretName: hazelcast-license-key
  persistence:
    clusterDataRecoveryPolicy: "FullRecoveryOnly"
    pvc:
      accessModes: ["ReadWriteOnce"]
      requestStorage: 20Gi
    restore:
      bucketConfig:
        bucketURI: "s3://operator-backup?prefix=hazelcast/2022-06-08-17-01-20/" (1)
        secretName: br-secret-s3 (2)
  agent: (3)
    repository: hazelcast/platform-operator-agent
1 Bucket URI where backup data will be restored from.
2 Name of the secret with credentials for accessing the given bucket URI.
3 Agent which is responsible for restoring data from the external storage. The agent configuration is optional. If you provide restore under persistence and do not pass the agent configuration, Hazelcast Platform Operator uses the latest agent version that is compatible with its version. 
apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: hazelcast (1)
spec:
  clusterSize: 3
  repository: 'docker.io/hazelcast/hazelcast-enterprise'
  licenseKeySecretName: hazelcast-license-key
  persistence:
    clusterDataRecoveryPolicy: "FullRecoveryOnly"
    pvc:
      accessModes: ["ReadWriteOnce"]
      requestStorage: 20Gi
    restore:
      hotBackupResourceName: hot-backup (2)
  agent: (3)
    repository: hazelcast/platform-operator-agent
1 HotBackup resource name used for the restore. The bucket URI and secret are taken from the HotBackup resource.
2 Agent responsible for restoring data from external storage. The agent configuration is optional. If you provide restore under persistence and do not pass the agent configuration, Hazelcast Platform Operator uses the latest agent version that is compatible with its version.

Restoring from Persistent Volumes

To restore from local backups that are in an existing Persistent Volume, configure localConfig.

apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: hazelcast (1)
spec:
  clusterSize: 3
  repository: 'docker.io/hazelcast/hazelcast-enterprise'
  licenseKeySecretName: hazelcast-license-key
  customConfigCmName: mapCustomConfig (2)
  persistence:
    pvc:
      accessModes: ["ReadWriteOnce"]
      requestStorage: 20Gi
    restore:
      localConfig:
        pvcNamePrefix: "hot-restart-persistence" (3)
        baseDir: "base" (4)
        backupDir: "backup" (5)
        backupFolder: "backup-1709045128548" (6)
1 To successfully attach existing PVCs to the newly created cluster:

  • If the older cluster was created by the Hazelcast Platform Operator, the name of both the backup and restore Hazelcast custom resources must be the same.

  • If the older cluster was created by the Hazelcast Helm Chart, the name of the new CR must be in the following format: <name>-hazelcast-persistence. For example, assuming that the older cluster was installed with helm install hz hazelcast/hazelcast-enterprise command, the new CR name must be hz-hazelcast-enterprise.
2 If you configured maps with persistence, you must create them while creating the Hazelcast CR. To create them while creating the Hazelcast CR, you must use Custom Config. You cannot apply the Hazelcast CR and then the Map CRs.
3 pvcNamePrefix is the prefix of the existing PVCs. It can be set to persistence or hot-restart-persistence depending on the installation method of the old cluster. In different versions of Hazelcast Helm Chart and Hazelcast Platform Operator, these values are used by default. Run the kubect get pvc command to decide which setting best suits your requirements.
4 baseDir is the root directory for persistence.
5 backupDir is the directory that contains a backupFolder for each available backup.
6 backupFolder is the directory containing the specific backup for the restore.
To find the backupFolder value, you can run kubectl exec -it <hazelcast-custom-resource-name> -c hazelcast — /bin/bash and list the contents of your in your existing installation. If you already deleted your installation, you can run a simple pod that lists the contents of the PVC and checks its logs. This lists the folder structure of the specified PVC. For example: 
apiVersion: v1
kind: Pod
metadata:
  name: list-content-pod
spec:
  containers:
    - name: list-content-container
      image: busybox
      command: ["/bin/sh", "-c", "tree /data/persistence"] (1)
      volumeMounts:
        - name: pv-storage
          mountPath: /data/persistence (2)
  volumes:
    - name: pv-storage
      persistentVolumeClaim:
        claimName: hot-restart-persistence-hazelcast-0 (3)
1 Replace /data/persistence with the path to your PV, which is mounted inside the container.
2 Replace /data/persistence with the correct mountPath specified in PV.
3 Replace with the name of the one of the PVCs, which is mounted to the cluster from which the backup is taken.
Agent copies the backup to be restored from {baseDir}/{backupDir}/{backupFolder} to /data/persistence/base-dir.

Configuring Persistence

Data Recovery Timeout

To choose a data recovery timeout, you can use dataRecoveryTimeout. The field takes an integer value representing the timeout in seconds and uses this value to set validation-timeout-seconds, data-load-timeout-seconds Hazelcast Persistence options.

apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: hazelcast
spec:
  clusterSize: 3
  repository: 'docker.io/hazelcast/hazelcast-enterprise'
  licenseKeySecretName: hazelcast-license-key
  persistence:
    clusterDataRecoveryPolicy: "FullRecoveryOnly"
    dataRecoveryTimeout: 600
    pvc:
      accessModes: ["ReadWriteOnce"]
      requestStorage: 20Gi

Choosing a Cluster Recovery Policy

To decide how a cluster should behave when one or more members cannot rejoin after a cluster-wide restart, you can define one of the following cluster recovery policies. The Operator supports all the policies in the Hazelcast Platform cluster-data-recovery-policy configuration options. For complete descriptions and advice on choosing a policy, see the Platform documentation.

FullRecoveryOnly

Does not allow partial start of the cluster.

PartialRecoveryMostRecent

Allows partial start with the members that have most recent partition table.

PartialRecoveryMostComplete

Allows partial start with the member that have most complete partition table.

Configuring Force/Partial Start

To recover a cluster that has Persistence enabled after a cluster-wide restart, you can force a cluster to delete their persistence stores when one or more members fail to restart by triggering Force Start. Or you can force a cluster to start without some members by triggering Partial Start

  • The cluster loses all persisted data after a force-start if any member fails to start properly. If all members successfully start, all persisted data remains intact.

  • The cluster loses persisted data after a partial-start only for specific members that have failed to start properly. Persisted data remains intact for members that successfully start.

To trigger the cluster recover action set the startupAction:

apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: hazelcast
spec:
  clusterSize: 3
  repository: 'docker.io/hazelcast/hazelcast-enterprise'
  version: '5.5.2-slim'
  licenseKeySecretName: hazelcast-license-key
  persistence:
    clusterDataRecoveryPolicy: "FullRecoveryOnly"
    pvc:
      accessModes: ["ReadWriteOnce"]
      requestStorage: 8Gi
      storageClassName: "standard"
    startupAction: ForceStart

The PartialStart can be triggered only when clusterDataRecoveryPolicy is set to PartialRecoveryMostRecent or PartialRecoveryMostComplete.

lifebuoy

Help and support