Elastic recently announced the release of Elastic Stack 7.6. This release contains a number of security improvements including a new Security Information and Event Management (SIEM) detection engine and a redesigned SIEM overview dashboard page. This release also includes performance improvements to queries that are sorted by date, enhanced supervised machine learning capabilities, and support for ingesting Jaeger trace data.
With the release of Elastic Stack 7.6, Elastic has introduced a curated set of approximately 100 out-of-the-box detection rules to their SIEM detection engine. These rules are created and maintained by the security team at Elastic and are aligned with the MITRE ATT&CK Knowledge Base. MITRE ATT&CK is a knowledge base of techniques and approaches used in cyber-attacks. These documented techniques have been observed to be in use in real-world compromises.
The signals detected by these new rules are displayed within the SIEM detections interface. This interface reports on the risk and severity scores of the signals. These rules consume data in the Elastic Common Schema (ECS) format. This includes data from the OS level (with support for Windows, macOS, and Linux systems) as well as network information from other sources.
The SIEM overview page was redesigned to provide an enhanced timeline view and better integrate data from a number of sources. This includes curated visibility into HTTP data and the ability to view APM data directing within the SIEM application. Additional integrations include support for ingesting AWS CloudTrail data and enhanced support for monitoring services in Google Cloud Platform via StackDriver.
The primary performance improvement is the usage of the block-max WAND optimizations to queries that are sorted by date or by a field that is stored in the long datatype. According to Daniel Courcy, product marketing manager at Elastic, and Matt Davis, principal product manager at Elastic, this change led to their standard nightly performance test showing "an improvement of between several times faster to 35(!) times faster for certain tests." This algorithm was previously leveraged in the 7.0 release to improve the performance of top hits aggregations.
This release also introduces end-to-end supervised machine learning capabilities. The addition of data frame analytics allows for the use of labelled data to train and test models, storing of the models as Elasticsearch indices, and using inference to add predicted values to the indices based on the trained models. Data frame analytics jobs exist as a persistent Elasticsearch task. This process does not affect the source index, instead creating a new index that contains a copy of the source and annotated data. Data analysis techniques that are supported include outlier detection for locating unusual data points, regression for making predictions on the data set, classification for categorizing the data, and inference for using the model to assess new data points on ingestion.
On the observability side, Elastic APM can now be used as a Jaeger intake allowing for the ingesting of Jaeger traces directly into Elasticsearch without the need for adjusting Jaeger instrumented code. Jaeger, a CNCF graduate project, is a distributed tracing tool that is compatible with the OpenTracing standard.
Additional changes included in the 7.6 release can be found on the Elastic blog. Elastic Stack 7.6 is available either via the Elasticsearch Service on Elastic Cloud or via the self-hosted downloads.
