Logz.io offers a hosted service which performs intelligent log analysis by using machine learning to derive insights from human interactions with log data that include discussions on tech forums and public code repositories.
Modern applications churn out logs and metrics at a huge rate. DevOps teams utilize tools like the ELK stack to centralize the log aggregation and indexing process. However, the sheer amount of logs is not easy to wade through. The amount of metrics collected has also grown tremendously so that traditional monitoring tools cannot do much more than display it. This increase in data volume has also led to a situation where thresholds-based alerting is not viable, thus leading to the rise of anomaly detection systems like Anomaly.io, Datadog and SumoLogic, and also open source ones like Etsy’s Skyline.
Logz.io takes this further by offering insights into error conditions, which are derived by using machine learning on data gathered from human interactions on the internet. These interactions include discussions about similar error conditions on sites like StackOverflow and Github. Since a simple Google search for most error conditions usually throws up many results, how does Logz.io filter out the relevant ones? InfoQ got in touch with Tomer Levy, CEO and Co-Founder at Logz.io, to find out more:
Logz.io uses machine learning to understand the context of the thread, the product to which the thread is referring, whether there was a resolution, and if it was satisfactory to the person asking the question. We also cross-reference these threads with data we’ve gathered from our own users.
In addition, Levy says, “after we bring an insight to a user’s attention, we assess the relevance based on his or her interaction with the insight.“ This completes a feedback loop so that the system can improve with time.
Searching for an error or exception for a product on the internet would lead to results in most cases since most modern products are built using open source libraries. In the rare cases that a product is built completely from scratch without using any publicly available code, Logz.io plans to release “private insights” in the future that will “enable organizations to define their own exceptions and the reasons behind them”, according to Levy.
Logz.io uses the Elasticsearch-Logstash-Kibana (ELK) stack under the hood. It’s a service hosted on Logz.io’s own cloud, which means that log data has to be piped there from the application. In cases where the organization runs its own ELK stack, Logz.io can integrate with the log shippers that reside on-premises and help securely transfer the data to the Logz.io cloud.
In response to how they address enterprise concerns about data security, Levy said:
We’ve invested in security from data encryption to comprehensive security processes. Logz.io has hired Earnest & Young to audit its security infrastrcture and process and was recently passed their audit for SOC-2 Type 2 and ISO-27001 compliance. In addition, the company hosts data in the United States, and EU-based companies can choose to store data in Frankfurt to ensure compliance with EU regulations.
