Mesosphere have released an alpha version of Mesos-DNS, an open source service discovery framework for cloud environments running applications on the Apache Mesos cluster manager.
Apache Mesos is an open source cluster manager that abstracts CPU, memory, storage and other compute resources away from machines (physical or virtual), and supports the execution of fault-tolerant and elastic distributed systems. A Mesos kernel runs on each machine in the cluster and provides applications with APIs for resource management and scheduling across the entire cluster.
Apache Mesos consists of a master daemon that manages slave daemons running on each cluster node, and Mesos applications (also called frameworks) that run tasks on these slaves. Example frameworks include Apache Spark and Hadoop for large-scale data processing, and Mesosphere’s Marathon and Apache’s Aurora for long running applications, such as those utilised in a microservice architecture. Service discovery is difficult in this environment due to the volatility of compute resources within the cluster and the dynamic allocation of resources to tasks.
The new open source Mesos-DNS framework supports service discovery in Apache Mesos clusters by allowing applications and services running on Mesos to locate each other using the domain name system (DNS). Mesos-DNS acts as the primary directory service for the cluster in much the same way DNS is used as a directory service within the Internet.
Applications or long-running services launched by the Marathon or Aurora frameworks are assigned names like search.marathon.mesos or log-aggregator.aurora.mesos, and Mesos-DNS translates these names to the IP address and port on the machine currently running each application. This enables a connection to a service running on an arbitrary (and potentially dynamically changing) machine in the cluster to be made by issuing DNS lookup requests.
Mesos-DNS has been designed to be a simple and stateless, and it does not require consensus mechanisms, persistent storage, or a replicated log. This is possible because Mesos-DNS does not implement heartbeats, health monitoring, or lifetime management for applications. This functionality is already available from the Mesos master, slaves, and frameworks.
Architectural diagram of Mesos-DNS taken from the Mesos-DNS Github README.md
Frameworks running within Mesos are not required to communicate with Mesos-DNS directly. Mesos-DNS periodically queries the Mesos master(s), retrieves the state of all running tasks from all running frameworks, and generates DNS records for these tasks (A and SRV records). As tasks start, finish, fail, or restart on the Mesos cluster, Mesos-DNS updates the DNS records to reflect the latest state.
Mesos-DNS is configured by specifying the Mesos master(s) at launch, and can be made fault-tolerant by launching with a framework like Marathon that can monitor application health and re-launch it on failure. On restart after a failure, Mesos-DNS retrieves the latest state from the Mesos master(s) and serves DNS requests without further coordination. It can be replicated to improve availability or to load balance DNS requests in clusters with large numbers of slaves.
Christos Kozyrakis, a core contributor to Mesos-DNS has stated that there were two motivations for implementing Mesos-DNS instead of using another DNS system, such as SkyDNS or Consul:
First, we needed a DNS system that closely integrates with Mesos. Instead of having every user or framework describe tasks twice (once to Mesos for execution and once to a DNS system), it easier and cleaner to automatically pass task information from Mesos to DNS. Second, we wanted a simple solution. Mesos and its frameworks already implement fault tolerance and life cycle management. We did not want to force Mesos users to deploy another set of consensus mechanisms, persistent storage, or a replicated logs just for DNS.
The alpha release of Mesos-DNS supports a basic scheme for application naming. The Mesosphere blog states that the upcoming enhancements to Mesos for service discovery will support flexible naming schemes that capture the service discovery needs for a wide set of deployments.
Mesos-DNS will also be extended to track security and networking features introduced in Mesos, and integrated into the upcoming Mesosphere Datacenter Operating System (DCOS) to support service discovery in public cloud, private datacenter, and hybrid deployments.
Mesos-DNS can be built and run on a private cluster by following the instructions on the Mesosphere Mesos-DNS configuration parameter Github page, or on the Google Compute Engine (GCE) platform by following the instructions on the Mesos-DNS on Google Compute Platform Github page.