Boosting Elasticsearch Cluster Performance: 3 Proven Tips

Tuning Elasticsearch configuration for improved resource utilization

Photo by Shubham Dhage on Unsplash

This article will explain what the application does better from the underlying Elasticsearch implementation, but I’m not going to dive into too much detail in a very obscure way, so I’ll try to use a simplified process.

Before we get started, let’s go over a few important Elasticsearch terminologies.

• Index: This is like the table in RDBMS or the collection in MongoDB, not to be confused with index in RDBMS.
• Shard (aka Primary Shard): The access point where the data is written to the index, and of course the data can be read.
• Replica: Access point to read data, can not be used to write data.

These terminologies are closely related to today’s three tips.

So, what is the role of these components in Elasticsearch?

Let me illustrate with an example. Suppose we have a two-node Elasticsearch cluster with two indexes, A and B, and they are configured as follows.

A index

• number_of_shards = 2
• number_of_replicas = 1

B index

• number_of_shards = 1
• number_of_replicas = 2

These are the index settings, which clearly indicate how many shards and replica are required, and the following diagram shows how Elasticsearch looks internally.

The red block represents the shard, and the normal block is the replica; the user, or application, will access both nodes to manipulate the corresponding data.

Tip #1: The number of shards should align to the number of nodes

This tip is limited to those large indexes, because we know the more shards are used, the more evenly the data is distributed, and if the data in each shard is only a little bit and distributed in many nodes, it will cause more search overhead.

According to the official recommendation, the data volume of a shard should be less than 50GB.

• 20–40 GB
• 10–50 GB

Why such a tip? Let’s use a counter-example to observe. Suppose we have many nodes, but each index has only one shard.

A index

• number_of_shards = 1
• number_of_replicas = 3

Then the user’s read operations will be evenly distributed on each node, but the write operations will be concentrated on a single node.

In the case of a large volume of writes, such a setting can create a performance bottleneck on a single node and affect every operation on that node.

Since Elasticsearch 5.x started to support hot warm architecture, if a cluster has hot warm architecture enabled, then the number of shards should align to the number of hot nodes.

Tip #2: The document should be routed

Let’s continue with the top example, assuming a two-node cluster, and index A has two shards and one replica each.

When a user writes to a document, which node does he write to?

This is determined by Elasticsearch’s built-in routing rules, briefly, by the following formula.

shard = hash(_id) % number_of_shards

The _id is automatically dispatched by Elasticsearch, but can of course be specified by the user. As for the hash algorithm, it is murmur3, which is not a consistent hash, so a specific shard will be calculated.

So, the worst case would look like the following diagram.

When one user starts a bulk operation, writing a large volume of documents to the cluster, and at the same time, one user is searching, the two users’ operations interfere with each other.

If there is a way to keep the data of a single user in a single node, then the above case can be effectively avoided. At most, one user’s search performance will be affected when he writes a lot. This is called custom routing.

Elasticsearch is able to add a routing parameter to read and write operations, so that the formula mentioned earlier becomes

shard = hash(routing) % number_of_shards

Nevertheless, it is also the need to pay attention to whether there will be a hotspot problem, i.e., data is overly concentrated in certain nodes.

I have an article to explain the design of sharding keys. Although this article is about MongoDB, the concepts are the same.

Tip #3: Turn off replica and refresh when doing a large volume of writes

When doing a large volume of writes, stop replication first to reduce resource consumption, and then turn on replication when the large volume of writes is complete. This can significantly reduce write consumption and improve the performance of the cluster.

Disabling replica is easy to understand, but what about disabling refresh?

Before we explain, let’s understand the mechanism of refresh.

When a user writes data to the shard, it is first written to the memory buffer, and the data is invisible to the search operation at this time. Then, Elasticsearch writes the data in the memory buffer to the hard drive via refresh and converts it into Lucene format, and then it is really searchable.

There are two ways to refresh.

1. refresh_interval in Index setting, refresh will be triggered automatically when the time is up.
2. Explicitly call Refresh API.

We don’t actively invoke the Refresh API during heavy writes, so by turning it off, we mean turning refresh_interval longer or even off.

Of course, either turning off replica or turning off refresh is to reduce resource consumption so that all resources can be dedicated to processing large amounts of writes.

So, you may wonder, won’t turning off refresh cause data loss?

No, it won’t.

The reason is that Elasticsearch’s persistence model relies on more than just refresh. Let’s dig a little deeper.

From the above diagram, we can see even the process of writing from the memory buffer to the hard drive is not really on the hard drive yet, it still needs to go through flush before it is really persisted.

However, this is a pretty long path, which is totally unreliable for a database, so in fact, when writing data, it will write to two places at the same time: the memory buffer and the transaction log.

In the event of a disaster, the entire written dataset can still be recovered through the transaction log. But of course, there is a price to pay for this. There is no insurance mechanism, and if the transaction log is corrupted, the data will really be lost. Nevertheless, I believe the chance is definitely much lower, and the loss is only for the data during this bulk writing period.

Conclusion

This time, we introduce three ways to significantly improve the performance of Elasticsearch clusters.

I have to say that in the big data world, it is obviously not enough to use all kinds of data storage. Without a deep understanding of the underlying implementation of data storage, there is a high risk of wasting resources. This will not only result in lower cluster performance, but also in higher hardware costs as more hardware is used to solve these overheads.

In this article, we briefly explain a few terminologies behind Elasticsearch, and then use this knowledge to learn more about optimization techniques. I believe you may be able to learn more details through this process.

If you have any good tips, please feel free to share them with me.