Replicate Data between Two Hazelcast Clusters with Hazelcast Platform Operator

Learn how to keep data in sync across two Hazelcast clusters.

Context

In this tutorial, you’ll do the following:

  • Deploy two Hazelcast clusters.

  • Create two Hazelcast map configurations on one of the clusters.

  • Synchronize map data between the two Hazelcast clusters.

Before you Begin

Before starting this tutorial, make sure that you have the following:

Step 1. Start the Hazelcast Cluster

  1. Create a secret with your Hazelcast Enterprise License.

    kubectl create secret generic hazelcast-license-key --from-literal=license-key=<hz-license-key>
    shell
  2. Create the Hazelcast clusters.

    1. Run the following command to create the first cluster.

      kubectl apply -f - <<EOF
      apiVersion: hazelcast.com/v1alpha1
      kind: Hazelcast
      metadata:
        name: hazelcast-first
      spec:
        licenseKeySecretName: hazelcast-license-key
        exposeExternally:
          type: Unisocket
          discoveryServiceType: LoadBalancer
      EOF
      shell
    2. Run the following command to create the second cluster.

      kubectl apply -f - <<EOF
      apiVersion: hazelcast.com/v1alpha1
      kind: Hazelcast
      metadata:
        name: hazelcast-second
      spec:
        licenseKeySecretName: hazelcast-license-key
        exposeExternally:
          type: Unisocket
          discoveryServiceType: LoadBalancer
      EOF
      shell
  3. Check the status of the clusters to make sure that both clusters are running.

    kubectl get hazelcast
    shell
    NAME               STATUS    MEMBERS
    hazelcast-first    Running   3/3
    hazelcast-second   Running   3/3
    shell
  4. Find the addresses of the clusters.

    kubectl get hazelcastendpoint --selector="app.kubernetes.io/instance in (hazelcast-first, hazelcast-second)"
    shell
    NAME                   TYPE        ADDRESS
    hazelcast-first        Discovery   34.123.9.149:5701
    hazelcast-first-wan    WAN         34.123.9.149:5710
    hazelcast-second       Discovery   34.16.0.16:5701
    hazelcast-second-wan   WAN         34.16.0.16:5710
    shell

    The ADDRESS column displays the external addresses of the Hazelcast clusters.

Step 2. Create a WAN Replication Configuration

  1. Create two maps on the first cluster. In this example, the following maps are created:

    • map-1

    • map-2.

      kubectl apply -f - <<EOF
      apiVersion: hazelcast.com/v1alpha1
      kind: Map
      metadata:
        name: map-1
      spec:
        hazelcastResourceName: hazelcast-first
      ---
      apiVersion: hazelcast.com/v1alpha1
      kind: Map
      metadata:
        name: map-2
      spec:
        hazelcastResourceName: hazelcast-first
      EOF
      shell
  2. Create the configuration for WAN replication:

    • Use the first cluster as the source cluster by adding its name as a resource in the WAN Replication configuration. Adding the cluster name as a resource starts WAN replication for both the maps that you created earlier.

    • Add the second cluster as the target cluster to receive the WAN Replication events.

      Run the following command to apply the configuration.

      kubectl apply -f - <<EOF
      apiVersion: hazelcast.com/v1alpha1
      kind: WanReplication
      metadata:
        name: wan-replication
      spec:
        resources:
          - name: hazelcast-first
            kind: Hazelcast
        targetClusterName: dev
        endpoints: "<SECOND-CLUSTER-EXTERNAL-IP>"
      EOF
      shell

Step 3. Put Entries to the Maps on the First Cluster

In this step, you’ll fill the maps on the first, source cluster.

  1. Configure the Hazelcast client to connect to the first cluster, using its address.

    To access all sample clients, clone the following repository:

    git clone https://github.com/hazelcast-guides/hazelcast-platform-operator-wan-replication.git
    cd hazelcast-platform-operator-wan-replication
    shell

    The sample code(excluding CLC) for this tutorial is in the docs/modules/ROOT/examples/operator-wan directory.

    • CLC

    • Java

    • NodeJS

    • Go

    • Python

    • .NET

    Note Before using CLC, it should be installed in your system. Check the installation instructions for CLC: Installing the Hazelcast CLC.

    Run the following command for adding the first cluster config to the CLC.

    clc config add hz-1 cluster.name=dev cluster.address=<FIRST-CLUSTER-EXTERNAL-IP>
    bash
    package com.hazelcast;
    
    import com.hazelcast.client.HazelcastClient;
    import com.hazelcast.client.config.ClientConfig;
    import com.hazelcast.core.HazelcastInstance;
    import com.hazelcast.map.IMap;
    
    import java.util.Random;
    
    public class Main {
        public static void main(String[] args) throws Exception {
            if(args.length != 2) {
                System.out.println("You need to pass two arguments. The first argument must be `fill` or `size`. The second argument must be `mapName`.");
            } else if (!((args[0].equals("fill") || args[0].equals("size")))) {
                System.out.println("Wrong argument, you should pass: fill or size");
            } else{
                ClientConfig config = new ClientConfig();
                config.getNetworkConfig().addAddress("<EXTERNAL-IP>");
    
                HazelcastInstance client = HazelcastClient.newHazelcastClient(config);
                System.out.println("Successful connection!");
    
                String mapName = args[1];
                IMap<String, String> map = client.getMap(mapName);
    
                if (args[0].equals("fill")) {
                    System.out.printf("Starting to fill the map (%s) with random entries.\n", mapName);
    
                    Random random = new Random();
                    while (true) {
                        int randomKey = random.nextInt(100_000);
                        map.put("key-" + randomKey, "value-" + randomKey);
                        System.out.println("Current map size: " + map.size());
                    }
                } else {
                    System.out.printf("The map (%s) size: (%d)\n\n", mapName, map.size());
                    client.shutdown();
                }
            }
    
        }
    }
    java
    'use strict';
    
    const { Client } = require('hazelcast-client');
    
    const clientConfig = {
        network: {
            clusterMembers: [
                '<EXTERNAL-IP>'
            ]
        }
    };
    
    (async () => {
        try {
            if (process.argv.length !== 4) {
                console.error('You need to pass two arguments. The first argument must be `fill` or `size`. The second argument must be `mapName`.');
            } else if (!(process.argv[2] === 'fill' || process.argv[2] === 'size')) {
                console.error('Wrong argument, you should pass: fill or size');
            } else {
                const client = await Client.newHazelcastClient(clientConfig);
                const mapName = process.argv[3]
                const map = await client.getMap(mapName);
                await map.put('key', 'value');
                const res = await map.get('key');
                if (res !== 'value') {
                    throw new Error('Connection failed, check your configuration.');
                }
                console.log('Successful connection!');
                if (process.argv[2] === 'fill'){
                    console.log(`Starting to fill the map (${mapName}) with random entries.`);
                    while (true) {
                        const randomKey = Math.floor(Math.random() * 100000);
                        await map.put('key' + randomKey, 'value' + randomKey);
                        const size = await map.size();
                        console.log(`Current map size: ${size}`);
                    }
                } else {
                    const size = await map.size();
                    console.log(`The map (${mapName}) size: ${size}`);
                }
            }
        } catch (err) {
            console.error('Error occurred:', err);
        }
    })();
    javascript
    package main
    
    import (
    	"context"
    	"fmt"
    	"math/rand"
    	"os"
    
    	"github.com/hazelcast/hazelcast-go-client"
    )
    
    func main() {
    	if len(os.Args) != 3 {
    		fmt.Println("You need to pass two arguments. The first argument must be `fill` or `size`. The second argument must be `mapName`.")
    		return
    	}
    	if os.Args[1] != "fill" && os.Args[1] != "size" {
    		fmt.Println("Wrong argument, pass `fill` or `size` instead.")
    		return
    	}
    
    	config := hazelcast.Config{}
    	cc := &config.Cluster
    	cc.Network.SetAddresses("<EXTERNAL-IP>:5701")
    	cc.Unisocket = true
    	ctx := context.TODO()
    	client, err := hazelcast.StartNewClientWithConfig(ctx, config)
    	if err != nil {
    		panic(err)
    	}
    	fmt.Println("Successful connection!")
    
    	mapName := os.Args[2]
    	m, err := client.GetMap(ctx, mapName)
    	if err != nil {
    		panic(err)
    	}
    	if os.Args[1] == "fill" {
    		fmt.Printf("Starting to fill the map (%s) with random entries.\n", mapName)
    		for {
    			num := rand.Intn(100_000)
    			key := fmt.Sprintf("key-%d", num)
    			value := fmt.Sprintf("value-%d", num)
    			if _, err = m.Put(ctx, key, value); err != nil {
    				fmt.Println("ERR:", err.Error())
    				continue
    			}
    			mapSize, err := m.Size(ctx)
    			if err != nil {
    				fmt.Println("ERR:", err.Error())
    				continue
    			}
    			fmt.Println("Current map size:", mapSize)
    		}
    		return
    	}
    	mapSize, err := m.Size(ctx)
    	if err != nil {
    		fmt.Println("ERR:", err.Error())
    		return
    	}
    	fmt.Printf("The map (%s) size: %v", mapName, mapSize)
    
    }
    go
    import logging
    import random
    import sys
    
    import hazelcast
    
    logging.basicConfig(level=logging.INFO)
    
    if len(sys.argv) != 3:
        print("You need to pass two arguments. The first argument must be `fill` or `size`. The second argument must be `mapName`.")
    elif not (sys.argv[1] == "fill" or sys.argv[1] == "size"):
        print("Wrong argument, you should pass: fill or size")
    else:
        client = hazelcast.HazelcastClient(
            cluster_members=["<EXTERNAL-IP>"],
            use_public_ip=True,
        )
        print("Successful connection!", flush=True)
    
        mapName = sys.argv[2]
        m = client.get_map(mapName).blocking()
    
        if sys.argv[1] == "fill":
            print(f'Starting to fill the map ({mapName}) with random entries.', flush=True)
            while True:
                random_number = str(random.randrange(0, 100000))
                m.put("key-" + random_number, "value-" + random_number)
                print("Current map size:", m.size())
        else:
            print(f'The map ({mapName}) size: {m.size()}')
    python
    using System;
    using System.Threading.Tasks;
    using Hazelcast;
    using Microsoft.Extensions.Logging;
    
    namespace Client
    {
        public class Program
        {
            static async Task Main(string[] args)
            {
                if (args.Length != 2)
                {
                    Console.WriteLine("You need to pass two arguments. The first argument must be `fill` or `size`. The second argument must be `mapName`.");
                    return;
                }
                if (!(args[0] == "fill" || args[0] == "size"))
                {
                    Console.WriteLine("Wrong argument, you should pass: fill or size");
                    return;
                }
    
                var mapName = args[1];
                var options = new HazelcastOptionsBuilder()                
                    .With(args)                
                    .With((configuration, options) =>
                    {
                        options.LoggerFactory.Creator = () => LoggerFactory.Create(loggingBuilder =>
                            loggingBuilder
                                .AddConsole());
    
                        options.Networking.UsePublicAddresses = true;
                        options.Networking.SmartRouting = false;
                        options.Networking.Addresses.Add("<EXTERNAL-IP>:5701");
                        
                    })
                    .Build();
    
    
    
                await using var client = await HazelcastClientFactory.StartNewClientAsync(options);
                
                Console.WriteLine("Successful connection!");
                Console.WriteLine("Starting to fill the map with random entries.");
    
                var map = await client.GetMapAsync<string, string>(mapName);
                var random = new Random();
    
                if (args[0] == "fill")
                {
                    Console.WriteLine("Starting to fill the map with random entries.");
                    while (true)
                    {
                        var num = random.Next(100_000);
                        var key = $"key-{num}";
                        var value = $"value-{num}";
                        await map.PutAsync(key, value);
                        var mapSize = await map.GetSizeAsync();
                        Console.WriteLine($"Current map size: {mapSize}");
                    }
                }
                else
                {
                    var mapSize = await map.GetSizeAsync();
                    Console.WriteLine($"Current map size: {mapSize}");
                    await client.DisposeAsync();
                }
            }
        } 
    }
    cs
  2. Start to fill the maps.

    • CLC

    • Java

    • NodeJS

    • Go

    • Python

    • .NET

    Run the following command for each map, using the map name as an argument to fill each map with entries. Use the map names map-1 and map-2.

    for i in {1..10};
    do
       clc -c hz-1 map set --name <MAP-NAME> key-$i value-$i;
    done
    bash

    Run the following command for each map to check if the sizes are expected.

    clc -c hz-1 map size --name <MAP-NAME>
    bash

    Start the application for each map, using the map name as an argument to fill each map with random entries. Use the map names map-1 and map-2.

    cd java
    mvn package
    java -jar target/*jar-with-dependencies*.jar fill <MAP-NAME>
    bash

    You should see the following output.

    Successful connection!
    Starting to fill the map (<MAP-NAME>) with random entries.
    Current map size: 2
    Current map size: 3
    Current map size: 4
    ....
    ....
    shell

    Start the application for each map, using the map name as an argument to fill each map with random entries. Use the map names map-1 and map-2.

    cd nodejs
    npm install
    npm start fill <MAP-NAME>
    bash

    You should see the following output.

    Successful connection!
    Starting to fill the map (<MAP-NAME>) with random entries.
    Current map size: 2
    Current map size: 3
    Current map size: 4
    ....
    ....
    shell

    Start the application for each map, using the map name as an argument to fill each map with random entries. Use the map names map-1 and map-2.

    cd go
    go run main.go fill <MAP-NAME>
    bash

    You should see the following output.

    Successful connection!
    Starting to fill the map (<MAP-NAME>) with random entries.
    Current map size: 2
    Current map size: 3
    Current map size: 4
    ....
    ....
    shell

    Start the application for each map, using the map name as an argument to fill each map with random entries. Use the map names map-1 and map-2.

    cd python
    pip install -r requirements.txt
    python main.py fill <MAP-NAME>
    bash

    You should see the following output.

    Successful connection!
    Starting to fill the map (<MAP-NAME>) with random entries.
    Current map size: 2
    Current map size: 3
    Current map size: 4
    ....
    ....
    shell

    Start the application for each map, using the map name as an argument to fill each map with random entries. Use the map names map-1 and map-2.

    cd dotnet
    dotnet build
    dotnet run fill <MAP-NAME>
    bash

    You should see the following output.

    Successful connection!
    Starting to fill the map (<MAP-NAME>) with random entries.
    Current map size: 2
    Current map size: 3
    Current map size: 4
    ....
    ....
    shell

Step 4. Verify the Replication of Map Entries

In this step, you’ll check the sizes of the maps on the second, target cluster to make sure that WAN replication events have been received.

  1. Configure the Hazelcast client to connect to the second cluster, as you did in Configure the Hazelcast Client.

  2. Start the application for each map, using the map name as an argument to check the map size, and to check that WAN replication was successful. Use the map names map-1 and map-2.

    • CLC

    • Java

    • NodeJS

    • Go

    • Python

    • .NET

    clc -c hz-2 map size --name <MAP-NAME>
    bash
    cd clients/java
    mvn package
    java -jar target/*jar-with-dependencies*.jar size <MAP-NAME>
    bash

    You should see the following output:

    Successful connection!
    Current map (<MAP-NAME>) size: 12
    shell
    cd clients/nodejs
    npm install
    npm start size <MAP-NAME>
    bash

    You should see the following output:

    Successful connection!
    Current map (<MAP-NAME>) size: 12
    shell
    cd clients/go
    go run main.go size <MAP-NAME>
    bash

    You should see the following output:

    Successful connection!
    Current map (<MAP-NAME>) size: 12
    shell
    cd clients/python
    pip install -r requirements.txt
    python main.py size <MAP-NAME>
    bash

    You should see the following output:

    Successful connection!
    Current map (<MAP-NAME>) size: 12
    shell
    cd clients/dotnet
    dotnet build
    dotnet run size <MAP-NAME>
    bash

    You should see the following output:

    Successful connection!
    Current map (<MAP-NAME>) size: 12
    shell

Clean Up

To remove all custom resources, run the following commands:

kubectl delete secret hazelcast-license-key
kubectl delete $(kubectl get wanreplications,map,hazelcast -o name)
shell