Hortonworks and Apache announced Metron graduating to a top-level project. Metron is the latest evolution of an all-in-one security telemetry data capture, streaming analytics and response platform whose lineage started at Cisco with the OpenSOC project, an open-source security framework for big data systems. Metron provides log aggregation, full packet capture indexing, storage, advanced behavioral analytics and data enrichment and applies current threat intelligence information to security telemetry, and does so with a single platform.
Conceptually, Metron is comprised of four components: Data capture and ingest, real-time processing, guaranteed data-persistence and storage, and machine learning models as a service that drive monitoring and alerts around risk.
Metron is at its core a Kappa architecture, a variant of Lambda architecture implemented with Apache Kafka as its unified data bus, and Apache Storm as the processing component. A Bro plugin provides the ability to forward Bro logs to Kafka. This lets Metron capture data that are specifically useful for deep packet inspection, capture and reconstruction while taking advantage of Kafka's guarantees and integration with the rest of the big data ecosystem.
For data capture, telemetry data can be posted to Metron's message-bus, and persisted or processed in real-time via Storm to HBase. Once the data is captured a number of options are available for search indexing and real-time, and near real-time processing. Metron provides interfaces to optionally interface HBase with ElasticSearch, or Lucene and Solr. Default management and dashboarding interfaces are built on Kibana.
There are a few features that make Metron different from an emerging standard in data pipelines. First is its integration with a set of data transformation utilities and API's via Stellar, a threat intelligence triage and field transformations language, that operate as functions deployed and executed via Metron's RESTful modeling-as-a-service (MaaS). MaaS functions are managed via Yarn and are designed for implementing as real-time or near-real-time threat detection and response mechanisms. A data-enrichment tool set provides the ability to manage and load various enrichment and threat intelligence sources into Metron's HBase data sink. The machine learning models deployed through the MaaS are meant to augment the behavior of this data enrichment step. A set of profilers. Lastly, a profiler mechanism executes feature extraction and windowing over the real-time and near-real-time telemetry data as it's coming in off the message bus, and as it gets persisted to HBase.
Owen O'Malley is the lead on the project at Apache and is responsible for initially porting Metron over from OpenSOC.