Replicate Data between Two Hazelcast Clusters with Hazelcast Platform Operator
Learn how to keep data in sync across two Hazelcast clusters.
Context
In this tutorial, you’ll do the following:
-
Deploy two Hazelcast clusters.
-
Create two Hazelcast map configurations on one of the clusters.
-
Synchronize map data between the two Hazelcast clusters.
Before you Begin
Before starting this tutorial, make sure that you have the following:
-
A running Kubernetes cluster
-
The Kubernetes command-line tool, kubectl
-
A deployed Hazelcast Platform Operator
Step 1. Start the Hazelcast Cluster
-
Create a secret with your Hazelcast Enterprise License.
-
Create the Hazelcast clusters.
-
Run the following command to create the first cluster.
-
Run the following command to create the second cluster.
-
-
Check the status of the clusters to make sure that both clusters are running.
-
Find the addresses of the clusters.
The
ADDRESS
column displays the external addresses of the Hazelcast clusters.
Step 2. Create a WAN Replication Configuration
-
Create two maps on the first cluster. In this example, the following maps are created:
-
Create the configuration for WAN replication:
-
Use the first cluster as the source cluster by adding its name as a resource in the WAN Replication configuration. Adding the cluster name as a resource starts WAN replication for both the maps that you created earlier.
-
Add the second cluster as the target cluster to receive the WAN Replication events.
Run the following command to apply the configuration.
-
Step 3. Put Entries to the Maps on the First Cluster
In this step, you’ll fill the maps on the first, source cluster.
-
Configure the Hazelcast client to connect to the first cluster, using its address.
To access all sample clients, clone the following repository:
The sample code(excluding CLC) for this tutorial is in the
docs/modules/ROOT/examples/operator-wan
directory.Before using CLC, it should be installed in your system. Check the installation instructions for CLC: Installing the Hazelcast CLC. Run the following command for adding the first cluster config to the CLC.
package com.hazelcast; import com.hazelcast.client.HazelcastClient; import com.hazelcast.client.config.ClientConfig; import com.hazelcast.core.HazelcastInstance; import com.hazelcast.map.IMap; import java.util.Random; public class Main { public static void main(String[] args) throws Exception { if(args.length != 2) { System.out.println("You need to pass two arguments. The first argument must be `fill` or `size`. The second argument must be `mapName`."); } else if (!((args[0].equals("fill") || args[0].equals("size")))) { System.out.println("Wrong argument, you should pass: fill or size"); } else{ ClientConfig config = new ClientConfig(); config.getNetworkConfig().addAddress("<EXTERNAL-IP>"); HazelcastInstance client = HazelcastClient.newHazelcastClient(config); System.out.println("Successful connection!"); String mapName = args[1]; IMap<String, String> map = client.getMap(mapName); if (args[0].equals("fill")) { System.out.printf("Starting to fill the map (%s) with random entries.\n", mapName); Random random = new Random(); while (true) { int randomKey = random.nextInt(100_000); map.put("key-" + randomKey, "value-" + randomKey); System.out.println("Current map size: " + map.size()); } } else { System.out.printf("The map (%s) size: (%d)\n\n", mapName, map.size()); client.shutdown(); } } } }
'use strict'; const { Client } = require('hazelcast-client'); const clientConfig = { network: { clusterMembers: [ '<EXTERNAL-IP>' ] } }; (async () => { try { if (process.argv.length !== 4) { console.error('You need to pass two arguments. The first argument must be `fill` or `size`. The second argument must be `mapName`.'); } else if (!(process.argv[2] === 'fill' || process.argv[2] === 'size')) { console.error('Wrong argument, you should pass: fill or size'); } else { const client = await Client.newHazelcastClient(clientConfig); const mapName = process.argv[3] const map = await client.getMap(mapName); await map.put('key', 'value'); const res = await map.get('key'); if (res !== 'value') { throw new Error('Connection failed, check your configuration.'); } console.log('Successful connection!'); if (process.argv[2] === 'fill'){ console.log(`Starting to fill the map (${mapName}) with random entries.`); while (true) { const randomKey = Math.floor(Math.random() * 100000); await map.put('key' + randomKey, 'value' + randomKey); const size = await map.size(); console.log(`Current map size: ${size}`); } } else { const size = await map.size(); console.log(`The map (${mapName}) size: ${size}`); } } } catch (err) { console.error('Error occurred:', err); } })();
package main import ( "context" "fmt" "math/rand" "os" "github.com/hazelcast/hazelcast-go-client" ) func main() { if len(os.Args) != 3 { fmt.Println("You need to pass two arguments. The first argument must be `fill` or `size`. The second argument must be `mapName`.") return } if os.Args[1] != "fill" && os.Args[1] != "size" { fmt.Println("Wrong argument, pass `fill` or `size` instead.") return } config := hazelcast.Config{} cc := &config.Cluster cc.Network.SetAddresses("<EXTERNAL-IP>:5701") cc.Unisocket = true ctx := context.TODO() client, err := hazelcast.StartNewClientWithConfig(ctx, config) if err != nil { panic(err) } fmt.Println("Successful connection!") mapName := os.Args[2] m, err := client.GetMap(ctx, mapName) if err != nil { panic(err) } if os.Args[1] == "fill" { fmt.Printf("Starting to fill the map (%s) with random entries.\n", mapName) for { num := rand.Intn(100_000) key := fmt.Sprintf("key-%d", num) value := fmt.Sprintf("value-%d", num) if _, err = m.Put(ctx, key, value); err != nil { fmt.Println("ERR:", err.Error()) continue } mapSize, err := m.Size(ctx) if err != nil { fmt.Println("ERR:", err.Error()) continue } fmt.Println("Current map size:", mapSize) } return } mapSize, err := m.Size(ctx) if err != nil { fmt.Println("ERR:", err.Error()) return } fmt.Printf("The map (%s) size: %v", mapName, mapSize) }
import logging import random import sys import hazelcast logging.basicConfig(level=logging.INFO) if len(sys.argv) != 3: print("You need to pass two arguments. The first argument must be `fill` or `size`. The second argument must be `mapName`.") elif not (sys.argv[1] == "fill" or sys.argv[1] == "size"): print("Wrong argument, you should pass: fill or size") else: client = hazelcast.HazelcastClient( cluster_members=["<EXTERNAL-IP>"], use_public_ip=True, ) print("Successful connection!", flush=True) mapName = sys.argv[2] m = client.get_map(mapName).blocking() if sys.argv[1] == "fill": print(f'Starting to fill the map ({mapName}) with random entries.', flush=True) while True: random_number = str(random.randrange(0, 100000)) m.put("key-" + random_number, "value-" + random_number) print("Current map size:", m.size()) else: print(f'The map ({mapName}) size: {m.size()}')
using System; using System.Threading.Tasks; using Hazelcast; using Microsoft.Extensions.Logging; namespace Client { public class Program { static async Task Main(string[] args) { if (args.Length != 2) { Console.WriteLine("You need to pass two arguments. The first argument must be `fill` or `size`. The second argument must be `mapName`."); return; } if (!(args[0] == "fill" || args[0] == "size")) { Console.WriteLine("Wrong argument, you should pass: fill or size"); return; } var mapName = args[1]; var options = new HazelcastOptionsBuilder() .With(args) .With((configuration, options) => { options.LoggerFactory.Creator = () => LoggerFactory.Create(loggingBuilder => loggingBuilder .AddConsole()); options.Networking.UsePublicAddresses = true; options.Networking.SmartRouting = false; options.Networking.Addresses.Add("<EXTERNAL-IP>:5701"); }) .Build(); await using var client = await HazelcastClientFactory.StartNewClientAsync(options); Console.WriteLine("Successful connection!"); Console.WriteLine("Starting to fill the map with random entries."); var map = await client.GetMapAsync<string, string>(mapName); var random = new Random(); if (args[0] == "fill") { Console.WriteLine("Starting to fill the map with random entries."); while (true) { var num = random.Next(100_000); var key = $"key-{num}"; var value = $"value-{num}"; await map.PutAsync(key, value); var mapSize = await map.GetSizeAsync(); Console.WriteLine($"Current map size: {mapSize}"); } } else { var mapSize = await map.GetSizeAsync(); Console.WriteLine($"Current map size: {mapSize}"); await client.DisposeAsync(); } } } }
-
Start to fill the maps.
Run the following command for each map, using the map name as an argument to fill each map with entries. Use the map names
map-1
andmap-2
.Run the following command for each map to check if the sizes are expected.
Start the application for each map, using the map name as an argument to fill each map with random entries. Use the map names
map-1
andmap-2
.You should see the following output.
Start the application for each map, using the map name as an argument to fill each map with random entries. Use the map names
map-1
andmap-2
.You should see the following output.
Start the application for each map, using the map name as an argument to fill each map with random entries. Use the map names
map-1
andmap-2
.You should see the following output.
Start the application for each map, using the map name as an argument to fill each map with random entries. Use the map names
map-1
andmap-2
.You should see the following output.
Start the application for each map, using the map name as an argument to fill each map with random entries. Use the map names
map-1
andmap-2
.You should see the following output.
Step 4. Verify the Replication of Map Entries
In this step, you’ll check the sizes of the maps on the second, target cluster to make sure that WAN replication events have been received.
-
Configure the Hazelcast client to connect to the second cluster, as you did in Configure the Hazelcast Client.
-
Start the application for each map, using the map name as an argument to check the map size, and to check that WAN replication was successful. Use the map names
map-1
andmap-2
.You should see the following output:
You should see the following output:
You should see the following output:
You should see the following output: