Weaveworks, creators of the Weave Docker virtual networking solution, have released a pre-alpha version of Weave Scope, an open source developer-focused container monitoring tool. Scope automatically generates a map of containers, enabling developers to visualise, monitor, and control applications by using the information exposed to drive deployment and operational decisions. Weaveworks are asking for feedback and collaboration from the community in order to drive forward the development of Scope.
The Weaveworks blog post announcing the release of Scope proposes that current application and infrastructure monitoring solutions are primarily aimed at operators, and do not address the requirements of developers. With the rise of DevOps and the 'you write it, you run it' philosophy, there is now a need for developer-focused monitoring tooling. Scope seeks to provide monitoring applications that are understandable by both developers and operators, with a particular focus on Docker-based deployments.
Weave Scope builds upon Weaveworks' existing product Weave, which offers a layer-2 software-defined network (SDN) that connects Docker containers that are potentially deployed across multiple hosts, and enables automated discovery. Applications running within the Docker containers use the network as if the containers were all plugged into the same network switch, with no requirements to configure port mappings or container links etc. The Weaveworks blog suggests that one key use case for Weave is to help customers build 'cloud-native' applications, which are typically container-packaged, dynamically scheduled and microservice-oriented.
The idea is [cloud-native microservice applications] improve developer productivity, and that ops can get big cost reductions. Efficiencies come from automation, and from having more flexible apps in relation to resources.
The Weaveworks blog proposes two key reasons that more organisations are not embracing the development of cloud-native applications. The first challenge is operational, for example, ensuring application services are healthy, gathering data for capacity planning and infrastructure management. The second challenge relates to scale, such as the inability to handle application and infrastructure elasticity. Weaveworks believe that their previous experience with developing Weave will enable them to deliver a monitoring solution that overcomes the above problems.
Weave Scope is run by executing a probe process with elevated privileges on each physical host that is to be monitored e.g. 'sudo probe'. A Scope application processes can then be executed on a host, and configured to talk to the probes e.g. 'app -probes="probe-host-1:4030, probe-host-2:4030"'. The Scope user interfaces can be viewed by pointing a browser at a web application running on port 4040 on the Scope host e.g. http://scope-host:4040. An example container/application monitoring graph can be seen below:
The Weave Scope code can be found in the associated Weaveworks Github repository, and is in the process of being fully integrated into Weave. The Weaveworks blog states that this should be completed over the next few weeks. Currently no other SDN implementation is supported by Weave Scope, but the Weaveworks team are looking for contributions:
If you are an SDN vendor or using some network other than Weave, do not despair. We'd love for you to use Scope too, and have no plans to make that awkward for you. Indeed, it should be very easy. So get started or get in touch. With Weave's growing adoption among Docker developers, and friendly features like automated service discovery and address allocation, it is an ideal way to combine your favourite network with cloud native application development.
InfoQ caught up with Alexis Richardson, founder and CEO of Weaveworks, and asked several questions about the release of Weave Scope:
InfoQ: In the Weave blog post announcing Weave Scope, you make an interesting observation that not many monitoring solutions are aimed as developers. With the increase of collaboration between developers and operations (a'la DevOps), do you think this is still a problem? And what are the ramifications?
Richardson: Yes, absolutely it is still a problem. Being developer friendly is fundamental. Scope enables application developers to take an existing application, that is using containers, and then monitor and visualize it.
As for DevOps: an important part of it is this: Developers take more ops responsibility, and integrate ops actions into their workflow. But that does not mean that Ops staff start to think about applications the same way that developers do, or that monitoring products all suddenly become developer-friendly.
Our goal is to enable a dialogue between "dev" and "ops". Let's call this "devops monitoring": a combination of technology and practice. Consider the classic example: it's 2am, and ops have been trying to resolve an issue for several hours. Maybe the developers are best qualified to understand and troubleshoot this issue, so they are called in. At this point they're often stymied by existing tools that reflect a fundamentally different view of their infrastructure, with different idioms baked in. With Scope, we intend to bridge the gap between the developer's logical view of the application, and the operator's physical view of the infrastructure. We think we can build something equally useful to both!
InfoQ: There are a few monitoring-as-a-service companies emerging now, such as Datadog and Dataloop.IO. Will your tooling be complementary to this, or would you see these companies as competitors?
Richardson: We see Weave Scope as primarily complementary, but with some competitive aspects due to its innovative approach. We also believe that organisations will benefit from leveraging standalone monitoring products, like InfluxDB and Prometheus.
In this context, Weave Scope may generate new feeds for those existing monitoring tools (Datadog, etc.) to consume and process, in conjunction with an entire application portfolio. In other words: these tools are aggregators, pulling in multiple sources of data to enable correlation, insight and decision. So in that sense Weave is complementary.
At the same time, we are trying to do something new, and in that sense we are competing with an existing set of assumptions about the limits of monitoring. We intend to put those assumptions to the test
InfoQ: Adrian Cockcroft of Battery Ventures (and ex-Netflix/eBay/Sun) is doing some interesting work with visualising microservices in his spigo/simianviz application, the output of which is similar to Weave Scope. What are your thoughts on Adrian's work?
Richardson: We very much admire Adrian's analysis of the microservices "way" and how it relates to the adoption of DevOps. If you are interested in this, I recommend the video of his talk at Dockercon Amsterdam 2014, plus the preceding talk from the CIO of ING. The latter is simply a fantastic example of how corporate adoption can benefit a company beyond mere technology metrics like 'velocity'.
One major difference between Scope and Spigo must be understood though. As mentioned above, Scope enables application developers to take an existing application, that is using containers, and then monitor and visualize it. This is done without changing your app - Scope just figures it out for you. Whereas in Spigo, you use the tool to simulate applications that do not (yet) exist, by modelling them as a set of interacting agents and associated protocols.
InfoQ: Several organisations are now stating publicly that they are building microservice platforms on top of cluster scheduling and orchestration technology, such as Mesos and Kubernetes, which often contain built-in basic monitoring and health-checking. Do you think Weave Scope will still add value here?
Richardson: Yes, we think adding Weave Scope would be an obvious win for Mesos and Kubernetes users and developers. Please note that it is difficult to compare our product with things that don't exist yet. Much has yet to happen. But whether others adopt Weave or do something else, we have ideas for where to take this that we think are genuinely new and of course highly valuable. Some examples are mentioned on our blog post.
InfoQ: Your blog contains several good points about the lack of developer tooling support for building microservice applications. Is this a hint that Weave is about to pivot away from offering only networking-related applications, to instead include a wider-range of developer tooling?
Richardson: "Yes and no".
Yes - we always wanted to do monitoring as we think it is central to understanding applications in a dynamic setting. We want to offer other capabilities that help developers succeed here.
But, no - we don't want to do this by offering them "build tools" like orchestration frameworks and platforms. The value of such tools comes from increasing developer productivity by supporting a particular model, or "opinion", of how an application should be constructed. For example Heroku makes it easy to build web apps on EC2, if you want to build web apps on EC2 using the 12-factor approach. But, we don't want Weave to be prescriptive in that way.
In summary -- We think developers already know how to write apps, and our tools should work for any application, anywhere.
InfoQ: At the end of your blog post you mention that Weave would like to encourage support for Weave Scope to be capable of monitoring other vendor networking solutions. Is this something you are going to drive, or will you look to encourage community-driven development?
Richardson: It is very easy to implement support for Weave Scope, and so we strongly encourage the community (and other vendors) to get involved! We do not believe in offering tightly coupled "monolithic" tools, and strongly prefer the "Unix philosophy". That is to say that any developer should be able, ideally, to use any part of weave freely, usefully and quickly. No developer should be obliged or forced to use every capability that we offer. So weave itself is trying to be closer to the ideal of "microservices".
InfoQ: Finally, Docker Inc, recently announced their strategic acquisition of Socketplane to improve networking in their container platform offerings. CoreOS have also been talking recently about gaining support from other vendors for the development of their App Container Spec and rkt implementation - would you consider teaming up with CoreOS to help drive this work forward?
Richardson: We can't do everything, and our customers are currently screaming for Docker support/integration. We've actually been working very closely with the former Socketplane team, now the Docker networking team. It's been very positive and Docker are doing their best to enable an "open platform".
With appc: we would welcome closer ties between the Docker spec and appc spec. We think the market wants more standardisation here, instead of two or more packaging standards like .deb and .rpm, but for containers. Arguably, a single spec is better for containers than multiple specs: If there is one standard container spec, then there will necessarily be multiple implementations that compete. Surely this is healthy since no one implementation can be 'one size fits all'.
Concerning rkt: Currently "competition to be the leading implementation of a container spec" is what is happening with appc and rkt. This is good for both appc and rkt, and could create threats for Docker. Currently Docker is the leading and only implementation of its own spec, which is only sustainable if it can bring the market and ecosystem with it. We would argue therefore that in a world of only one spec, Docker does better than in a 'bilateral' world of two specs. Docker would be still the clear leader in this scenario if multiple competitors have to fight for second place or occupy niches.
The Weaveworks Scope Github repository README.md states that the current release is pre-alpha, and users should not be surprised if bugs or missing features are encountered. Users are encouraged to experiment with Weave Scope, and to raise issues via Github or email.