Configure Service Replication

Whether service registry events should be streamed and published across a CAS cluster. One typical workflow is to enable the publisher on one master node and have others consume definitions and changes from the upstream master node in order to avoid overrides and timing issues as changes may step over each other if the service registry schedule is not timed correctly.

The instance name.

AWS access key.

AWS secret key.

The Azure Active Directory Service Principal client ID.

The Azure Active Directory Service Principal client secret.

The name of the tag on the hazelcast vm resources. With every Hazelcast Virtual Machine you deploy in your resource group, you need to ensure that each VM is tagged with the value of cluster-id defined in your Hazelcast configuration. The only requirement is that every VM can access each other either by private or public IP address.

The Azure resource group name of the cluster. You can find this in the Azure portal or CLI.

The Azure subscription ID.

The Azure Active Directory tenant ID.

Cloud Provider credential, can be thought of as a password for cloud services.

Cloud Provider identity, can be thought of as a user name for cloud services.

String value that is used to identify ComputeService provider. For example, "google-compute-engine" is used for Google Cloud services. See here for more info.

Name of this Hazelcast cluster. You can have multiple distinct clusters to use the same ZooKeeper installation

Path in zookeeper to be used for auto-discovery of members where members are tracked.

Zookeeper url address typically in the format of ip-address:port.

Sets the well known members. If members is empty, calling this method will have the same effect as calling clear(). A member can be a comma separated string, e..g 10.11.12.1,10.11.12.2 which indicates multiple members are going to be added. The list of members must include ALL CAS server node, including the current node that owns this configuration.

You can specify the ports which Hazelcast will use to communicate between cluster members. The name of the parameter for this is port and its default value is 5701. By default, Hazelcast will try 100 ports to bind. Meaning that, if you set the value of port as 5701, as members are joining to the cluster, Hazelcast tries to find ports between 5701 and 5801.

Hazelcast supports both synchronous and asynchronous backups. By default, backup operations are synchronous. In this case, backup operations block operations until backups are successfully copied to backup members (or deleted from backup members in case of remove) and acknowledgements are received. Therefore, backups are updated before a put operation is completed, provided that the cluster is stable. Asynchronous backups, on the other hand, do not block operations. They are fire and forget and do not require acknowledgements; the backup operations are performed at some point in time.

Used when replication is turned on with #isReplicated().

To provide data safety, Hazelcast allows you to specify the number of backup copies you want to have. That way, data on a cluster member will be copied onto other member(s). To create synchronous backups, select the number of backup copies. When this count is 1, a map entry will have its backup on one other member in the cluster. If you set it to 2, then a map entry will have its backup on two other members. You can set it to 0 if you do not want your entries to be backed up, e.g., if performance is more important than backing up. The maximum value for the backup count is 6. Sync backup operations have a blocking cost which may lead to latency issues.

CP Subsystem is a component of a Hazelcast cluster that builds a strongly consistent layer for a set of distributed data structures. Its data structures are CP with respect to the CAP principle, i.e., they always maintain linearizability and prefer consistency over availability during network partitions. Besides network partitions, CP Subsystem withstands server and client failures. All members of a Hazelcast cluster do not necessarily take part in CP Subsystem. The number of Hazelcast members that take part in CP Subsystem is specified here. CP Subsystem must have at least 3 CP members.

Hazelcast supports policy-based eviction for distributed maps. Currently supported policies are LRU (Least Recently Used) and LFU (Least Frequently Used) and NONE.

Max timeout of heartbeat in seconds for a node to assume it is dead.

Sets the maximum size of the map.

A Replicated Map is a distributed key-value data structure where the data is replicated to all members in the cluster. It provides full replication of entries to all members for high speed access. A Replicated Map does not partition data (it does not spread data to different cluster members); instead, it replicates the data to all members. Replication leads to higher memory consumption. However, a Replicated Map has faster read and write access since the data is available on all members. Writes could take place on local/remote members in order to provide write-order, eventually being replicated to all other members.

If you have a large cluster or very high occurrences of updates, the Replicated Map may not scale linearly as expected since it has to replicate update operations to all members in the cluster. Since the replication of updates is performed in an asynchronous manner, Hazelcast recommends you enable back pressure in case your system has high occurrences of updates.

Note that Replicated Map does not guarantee eventual consistency because there are some edge cases that fail to provide consistency.

Replicated Map uses the internal partition system of Hazelcast in order to serialize updates happening on the same key at the same time. This happens by sending updates of the same key to the same Hazelcast member in the cluster.

Due to the asynchronous nature of replication, a Hazelcast member could die before successfully replicating a "write" operation to other members after sending the "write completed" response to its caller during the write process. In this scenario, Hazelcast’s internal partition system promotes one of the replicas of the partition as the primary one. The new primary partition does not have the latest "write" since the dead member could not successfully replicate the update.

Connection timeout in seconds for the TCP/IP config and members joining the cluster.

ECS cluster short name or ARN; default is the current cluster.

The maximum amount of time Hazelcast will try to connect to a well known member before giving up. Setting this value too low could mean that a member is not able to connect to a cluster. Setting the value too high means that member startup could slow down because of longer timeouts (for example, when a well known member is not up). Increasing this value is recommended if you have many IPs listed and the members cannot properly build up the cluster. Its default value is 5.

Filter to look only for ECS tasks with the given family name; mutually exclusive with #getServiceName().

Host header. i.e. ec2.amazonaws.com. The URL that is the entry point for a web service.

If you do not want to use access key and secret key, you can specify iam-role. Hazelcast fetches your credentials by using your IAM role. This setting only affects deployments on Amazon EC2. If you are deploying CAS in an Amazon ECS environment, the role should not be specified. The role is fetched from the task definition that is assigned to run CAS.

Hazelcast port. Typically may be set to 5701. You can set searching for other ports rather than 5701 if you've members on different ports.

AWS region. i.e. us-east-1. The region where your members are running.

If a security group is configured, only instances within that security group are selected.

Filter to look only for ECS tasks from the given service; mutually exclusive with #getFamily().

If a tag key/value is set, only instances with that tag key/value will be selected.

If a tag key/value is set, only instances with that tag key/value will be selected.

Whether discovery should be enabled via the configured strategies below.

A range of ports where the plugin looks for Hazelcast members.

A filter to look only for instances labeled as specified; property format: key=value.

A filesystem path to the private key for GCP service account in the JSON format; if not set, the access token is fetched from the GCP VM instance.

A list of projects where the plugin looks for instances; if not set, the current project is used.

A region where the plugin looks for instances; if not set, the #getZones() property is used; if it and #getZones() property not set, all zones of the current region are used.

A list of zones where the plugin looks for instances; if not set, all zones of the current region are used.

Used for cloud providers which require an extra JSON or P12 key file. This denotes the path of that file. Only tested with Google Compute Engine. (Required if Google Compute Engine is used.)

Defines the endpoint for a generic API such as OpenStack or CloudStack (optional).

Filters instance groups (optional). When used with AWS it maps to security group.

Port which the hazelcast instance service uses on the cluster member. Default value is 5701. (optional)

Defines region for a cloud service (optional). Can be used with comma separated values for multiple values.

Used for IAM role support specific to AWS (optional, but if defined, no identity or credential should be defined in the configuration).

Filters cloud instances with tags (optional). Can be used with comma separated values for multiple values.

Filters cloud instances with tags (optional) Can be used with comma separated values for multiple values.

Defines zone for a cloud service (optional). Can be used with comma separated values for multiple values.

Defines the number of retries to Kubernetes API. Defaults to: 3.

Defines an oauth token for the kubernetes client to access the kubernetes REST API. Defaults to reading the token from the auto-injected file at: /var/run/secrets/kubernetes.io/serviceaccount/token.

CA Authority certificate from Kubernetes Master. Defaults to reading the certificate from the auto-injected file at: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt.

Defines an alternative address for the kubernetes master. Defaults to: https://kubernetes.default.svc

Defines the namespace of the application POD through the Service Discovery REST API of Kubernetes.

Defines the pod label to lookup through the Service Discovery REST API of Kubernetes.

Defines the pod label value to lookup through the Service Discovery REST API of Kubernetes.

Defines if not ready addresses should be evaluated to be discovered on startup.

Defines the DNS service lookup domain. This is defined as something similar to my-svc.my-namespace.svc.cluster.local.

Defines the DNS service lookup timeout in seconds. Defaults to 5 secs.

Defines the service label to lookup through the Service Discovery REST API of Kubernetes.

Defines the service label value to lookup through the Service Discovery REST API of Kubernetes.

Defines the service name of the POD to lookup through the Service Discovery REST API of Kubernetes.

If specified with a value greater than 0, its value defines the endpoint port of the service (overriding the default).

Defines if the node name should be used as external address, instead of looking up the external IP using the /nodes resource. Default is false.

Enables a multicast configuration using a group address and port. Contains the configuration for the multicast discovery mechanism. With the multicast discovery mechanism Hazelcast allows Hazelcast members to find each other using multicast. So Hazelcast members do not need to know concrete addresses of members, they just multicast to everyone listening. It depends on your environment if multicast is possible or allowed; otherwise you need to have a look at the tcp/ip cluster

The multicast group address used for discovery. With the multicast auto-discovery mechanism, Hazelcast allows cluster members to find each other using multicast communication. The cluster members do not need to know the concrete addresses of the other members, as they just multicast to all the other members for listening. Whether multicast is possible or allowed depends on your environment.

The multicast port used for discovery.

Gets the time to live for the multicast package in seconds. This is the default time-to-live for multicast packets sent out on the socket

specifies the time in seconds that a member should wait for a valid multicast response from another member running in the network before declaring itself the leader member (the first member joined to the cluster) and creating its own cluster. This only applies to the startup of members where no leader has been assigned yet. If you specify a high value, such as 60 seconds, it means that until a leader is selected, each member will wait 60 seconds before moving on. Be careful when providing a high value. Also, be careful not to set the value too low, or the members might give up too early and create their own cluster.

Multicast trusted interfaces for discovery. With the multicast auto-discovery mechanism, Hazelcast allows cluster members to find each other using multicast communication. The cluster members do not need to know the concrete addresses of the other members, as they just multicast to all the other members for listening. Whether multicast is possible or allowed depends on your environment.

IPv6 support has been switched off by default, since some platforms have issues in use of IPv6 stack. And some other platforms such as Amazon AWS have no support at all. To enable IPv6 support set this setting to false.

If this property is set, then this is the address where the server socket is bound to.

You can specify which network interfaces that Hazelcast should use. Servers mostly have more than one network interface, so you may want to list the valid IPs. Range characters ('*' and '-') can be used for simplicity. For instance, 10.3.10.* refers to IPs between 10.3.10.0 and 10.3.10.255. Interface 10.3.10.4-18 refers to IPs between 10.3.10.4 and 10.3.10.18 (4 and 18 included). If network interface configuration is enabled (it is disabled by default) and if Hazelcast cannot find an matching interface, then it will print a message on the console and will not start on that node.

Interfaces can be separated by a comma.

The outbound ports for the Hazelcast configuration.

You may also want to choose to use only one port. In that case, you can disable the auto-increment feature of port.

The default public address to be advertised to other cluster members and clients.

Comma-separated list of cipher suite names allowed to be used. Its default value are all supported suites in your Java runtime.

Name of the algorithm based on which the authentication keys are provided.

Type of the keystore. Its default value is JKS. Another commonly used type is the PKCS12. Available keystore/truststore types depend on your Operating system and the Java runtime. Only needed when the mutual authentication is used.

Path of your keystore file. Only needed when the mutual authentication is used.

Password to access the key from your keystore file. Only needed when the mutual authentication is used.

Name of the algorithm which is used in your TLS/SSL. For the protocol property, we recommend you to provide TLS with its version information, e.g., TLSv1.2. Note that if you write only TLS, your application chooses the TLS version according to your Java version.

Name of the algorithm based on which the trust managers are provided.

Path of your truststore file. The file truststore is a keystore file that contains a collection of certificates trusted by your application.

Password to unlock the truststore file.

Type of the truststore. Its default value is JKS. Another commonly used type is the PKCS12. Available keystore/truststore types depend on your Operating system and the Java runtime.

Flag which allows enabling endpoint identity validation. It means, during the TLS handshake client verifies if the server’s hostname (or IP address) matches the information in X.509 certificate (Subject Alternative Name extension).

Enable TCP/IP config. Contains the configuration for the Tcp/Ip join mechanism. The Tcp/Ip join mechanism relies on one or more well known members. So when a new member wants to join a cluster, it will try to connect to one of the well known members. If it is able to connect, it will now about all members in the cluster and doesn't rely on these well known members anymore.

Whether WAN should be enabled.

Name of this replication group.

List of target clusters to be used for synchronization and replication.

Maximum amount of time, in milliseconds, to be waited before sending a batch of events in case batch.size is not reached.

Maximum size of events that are sent to the target cluster in a single batch.

Sets the cluster name used as an endpoint group password for authentication on the target endpoint. If there is no separate publisher ID property defined, this cluster name will also be used as a WAN publisher ID. This ID is then used for identifying the publisher.

Strategy for checking the consistency of data between replicas.

Comma separated list of endpoints in this replication group. IP addresses and ports of the cluster members for which the WAN replication is implemented. These endpoints are not necessarily the entire target cluster and WAN does not perform the discovery of other members in the target cluster. It only expects that these IP addresses (or at least some of them) are available.

The number of threads that the replication executor will have. The executor is used to send WAN events to the endpoints and ideally you want to have one thread per endpoint. If this property is omitted and you have specified the endpoints property, this will be the case. If necessary you can manually define the number of threads that the executor will use. Once the executor has been initialized there is thread affinity between the discovered endpoints and the executor threads - all events for a single endpoint will go through a single executor thread, preserving event order. It is important to determine which number of executor threads is a good value. Failure to do so can lead to performance issues - either contention on a too small number of threads or wasted threads that will not be performing any work.

The WAN publisher properties.

Publisher class name for WAN replication.

Returns the publisher ID used for identifying the publisher.

For huge clusters or high data mutation rates, you might need to increase the replication queue size. The default queue size for replication queues is 10,000. This means, if you have heavy put/update/remove rates, you might exceed the queue size so that the oldest, not yet replicated, updates might get lost.

Time, in milliseconds, to be waited for the acknowledgment of a sent WAN event to target cluster.

When set to true, only the latest events (based on key) are selected and sent in a batch.

Enables compression when default java serialization is used.

Enable Jet configuration/service on the hazelcast instance. Hazelcast Jet is a distributed batch and stream processing system that can do stateful computations over massive amounts of data with consistent low latency. Jet service is required when executing SQL queries with the SQL service.

Enables scripting from Management Center.

Hazelcast enterprise license key.

Duration that indicates how long should items be kept in the hazelcast cache. Note that generally this number needs to be short as once an item is delivered to a target, it is explicitly removed from the cache/queue. This duration needs to be adjusted if the latency between the CAS nodes in the cluster is too large. Having too short a value will cause the record to expire before it reaches other members of the cluster.

Kafka bootstrapping server address (i.e. localhost:9092).

Specify the final compression type for a given topic. This configuration accepts the standard compression codecs ('gzip', 'snappy', 'lz4', 'zstd'). It additionally accepts 'uncompressed' which is equivalent to no compression; and 'producer' which means retain the original compression codec set by the producer.

Additional configuration options, as pointed out by TopicConfig.

Set the name of the topic.

Set the number of partitions (default 1).

Set the number of replicas (default 1).