logz.io released a new feature to help identify recurring patterns within log data. This new feature, entitled Log Patterns, automatically analyzes millions of log messages in real-time to reduce the volume of logs and potentially reveal common trends. This feature is their third feature to provide automated interpretation and filtering of log messages under the label of AIOps.
Log Patterns uses clustering algorithms to condense logs into a consolidated view. The output of this is viewable directly through Kibana. Within this view, the logs are automatically sorted with the pattern that occurs most being displayed first. As with a typical Kibana interface, patterns can be excluded to filter them out from the search results.
The filtering functionality allows for zoning in on rare or specific patterns. Filtering can also be used to remove patterns that are irrelevant to the investigation. These can also be outright discarded, allowing for a reduction in storage costs associated with storing the log data. As Asaf Yigal, VP of product at logz.io, notes, "engineers will save valuable time by filtering the millions of lines of data and analyzing only the specific patterns that need investigating."
This new release builds upon the previous released cognitive insights and application insights features. Cognitive insights, released in 2016, combines machine learning algorithms to piece together human interactions with the log data. This is coupled with social threads, discussion forums, and open source repos to create a data pool of insights that may contain information pertinent to resolving issues.
Application insights is a machine learning analytics tool to identify new exceptions and errors being logged within the environment. These are displayed within a seperate portion of the logz.io UI, the Insights page. The Insights page provides a single view for results from both cognitive and application insights. This includes a storyline view which depicts both the insights identified by logz.io and also key user-identified points. By default, this shows the five more frequent events from the past seven days.
The log pattern feature is available in all logz.io plans with no additional charge. More details can be found in the logz.io blog post.