Amazon has recently announced the Auto-Tune feature in Amazon Elasticsearch Service, a closed-loop control system that adapts the Elasticsearch cluster to the running workload. The new automated memory management provides better ingestion throughput for log analytics workloads and reduced tail latencies for search queries.
Amazon Elasticsearch Service uses performance and usage metrics from the Elasticsearch cluster to suggest optional changes to the Java virtual machine (JVM) settings on the nodes with the goal of improving cluster speed and stability. Based on the domain's performance metrics, Auto-Tune can increase or decrease the following cluster resources according to the application workloads: search and write thread pool queues, field data cache, shard request cache, JVM generation sizing, JVM heap sizing and clear cache.
Source: https://aws.amazon.com/blogs/big-data/introducing-auto-tune-in-amazon-es/
An article on the AWS Big Data blog explains why adaptive resource management is important:
Elasticsearch powers diverse use cases such as curated search and relevance ranking, log analytics, incident response (SIEM), time series data stores, or metrics for telemetry. These use cases have different performance and scaling characteristics, and use Elasticsearch features in multiple ways. Although heuristics and best practices provide a good starting point, there is no one size that fits all. With constantly evolving workloads, tuning your configuration to the specific needs of a cluster has significant performance and stability implications. However, Elasticsearch has several settings that are highly interdependent, and sometimes difficult to tune manually. Auto-Tune for Amazon ES addresses this.
Auto-Tune has two categories of changes, nondisruptive changes that it applies as the cluster runs, and changes requiring a blue/green deployment. If it is enabled without a maintenance window, Auto-Tune only applies nondisruptive changes that bring smaller performance benefits over time but avoid the overhead of a blue/green deployment. Ankit Malpani, senior software engineer at AWS, comments:
Auto tune and self driving DB is the new norm and this is a good first step towards getting #elasticsearch there.
The new functionality is enabled by default on new domains but it can be disabled using the command lines or the console. To enable Auto-Tune on existing domains, Amazon recommends using the console.
Source: https://aws.amazon.com/blogs/big-data/introducing-auto-tune-in-amazon-es/
One of the basis of the new service, Performance Analyzer, is open source and exposes a REST API that allows querying numerous performance metrics for a cluster, including aggregations of those metrics, independent of the Java Virtual Machine (JVM). PerfTop is the default command line interface (CLI) for displaying those metrics.
Alolita Sharma, principal technologist at AWS, tweets:
Excited to see years of open source development in #OpenDistroforElasticsearch PerformanceAnalyzer powering a new feature in Amazon Elasticsearch Service - AutoTune.
The relation between the open source community, Elasticsearch and AWS has been controversial in the latest months, a topic that InfoQ covered in a separate article.
Auto-Tune is available on domains running Elasticsearch 6.7 or later with a supported instance type.