This is a prerelease version.

View latest

Prometheus Monitoring

6.0-SNAPSHOT 5.6 5.5.2 5.5 5.4.1 5.4 5.3.4 5.3 5.2 5.1 5.0

You can export cluster metrics to a scrape endpoint in Management Center that Prometheus can consume. By collecting cluster metrics in Prometheus you can monitor Hazelcast along with your other applications in the same monitoring tool.

Enabling the Prometheus Exporter

To start exposing metrics to Prometheus, set the hazelcast.mc.prometheusExporter.enabled system property to true.

Metrics are exposed to the /metrics endpoint, which is the default Prometheus metrics path.

All metrics are exported with the hz_ prefix.

Disabling Timestamp

Management Center collects metrics with the timestamp of individual members, and sends this timestamp to Prometheus each time. Prometheus discards metrics with the old timestamps. You can disable sending the timestamp to Prometheus so that your monitoring system continues to operate when clock shifts happen in the cluster.

To disable sending timestamp to Prometheus, set the hazelcast.mc.prometheusExporter.timestamp.enabled property to false.

Configuring Prometheus

After enabling the Prometheus exporter, you can configure your Prometheus YAML file to scrape metrics from Management Center.

For guidance, see the Prometheus configuration docs.

To allow Prometheus to scrape metrics from Management Center, set the static_configs.targets parameter to the URL of your Management Center instance.

scrape_configs:
  - job_name: 'HZ MC'
    # scheme defaults to HTTP.
    static_configs:
    - targets: ['localhost:8080'] # replace this address with the network address of Hazelcast Management Center

Authenticating the Scrape Endpoint

To secure the scrape endpoint, you can set up basic authentication.

To set up basic authentication, use the set prometheus-auth task of the hz-mc conf tool. Replace the <my-user> and <my-pass> placeholders with your chosen username and password.

  • Linux and Mac

  • Windows

hz-mc conf set prometheus-auth <my-user> <my-pass>
mc-conf.bat set prometheus-auth <my-user> <my-pass>

Then, add the credentials to the basic_auth parameter of the Prometheus YAML file:

  - job_name: 'HZ MC'
    static_configs:
    - targets: ['localhost:8080']
    basic_auth:
      username: my-user
      password: my-pass

Resetting the Authentication Credentials

To reset authentication credentials for the scrape endpoint, use the security reset task of the hz-mc conf tool.

This task will also delete all users from the database.

  • Linux and Mac

  • Windows

hz-mc conf security reset
mc-conf.bat security reset

Filtering Metrics

By default, Management Center exports all metrics that are reported by the cluster.

To filter the metrics that are exposed to the scrape endpoint, use the hazelcast.mc.prometheusExporter.filter.metrics.included and hazelcast.mc.prometheusExporter.filter.metrics.excluded system properties.

For example, this configuation exposes only the following metrics:

  • hz_topic_totalReceivedMessages

  • hz_map_totalPutLatency

  • Linux and Mac

  • Windows

hz-mc start -Dhazelcast.mc.prometheusExporter.enabled=true \
  -Dhazelcast.mc.prometheusExporter.filter.metrics.included=hz_topic_totalReceivedMessages,hz_map_totalPutLatency
mc-start.cmd -Dhazelcast.mc.prometheusExporter.enabled=true ^
  -Dhazelcast.mc.prometheusExporter.filter.metrics.included=hz_topic_totalReceivedMessages,hz_map_totalPutLatency

This configuation exposes all metrics except the following:

  • hz_os_systemLoadAverage,hz_memory_freeHeap

  • Linux and Mac

  • Windows

hz-mc start -Dhazelcast.mc.prometheusExporter.enabled=true \
  -Dhazelcast.mc.prometheusExporter.filter.metrics.excluded=hz_os_systemLoadAverage,hz_memory_freeHeap
mc-start.cmd -Dhazelcast.mc.prometheusExporter.enabled=true ^
  -Dhazelcast.mc.prometheusExporter.filter.metrics.excluded=hz_os_systemLoadAverage,hz_memory_freeHeap

Overriding the Default Port

By default, metrics are exposed on the same port as the Management Center web interface. To override the port number, use the -Dhazelcast.mc.prometheusExporter.port system property.

  • Linux and Mac

  • Windows

hz-mc start -Dhazelcast.mc.prometheusExporter.enabled=true \
  -Dhazelcast.mc.prometheusExporter.port=2222
mc-start.cmd -Dhazelcast.mc.prometheusExporter.enabled=true ^
  -Dhazelcast.mc.prometheusExporter.port=2222

In this example, the Prometheus endpoint will be available at http://localhost:2222/metrics. To allow Prometheus to scrape metrics from Management Center, make sure to set the static_configs.targets parameter to this port.

scrape_configs:
  - job_name: 'HZ MC'
    static_configs:
    - targets: ['localhost:2222']

Visualizing Metrics in Grafana

To visualize Prometheus metrics in Grafana, you can start with this dashboard.

Next Steps

Learn more about the hz-mc conf tool.

Explore system properties and their definitions.

lifebuoy

Help and support