Nine months after acquiring consulting firm, BoldRadius, Lightbend announced their acquisition of OpsClarity, a company specializing in monitoring reactive applications.
Lightbend, founded in 2011 as TypeSafe, rebranded themselves as Lightbend last year. With the acquisition of BoldRadius and OpsClarity, they now have over 130 employees, which is double since December 2015. When asked about their growth, Mark Brewer, president and CEO at Lightbend, told InfoQ:
Mark Brewer: It's been good, it’s been fun. I love growing this way where we get to bring on a dedicated team that knows what they're working on, knows what their charter is, literally hitting the ground running. We get a product immediately out of this. It’s not just the team, but we get a product that we take to market. Customers are already actively using it and excited about using it.
Alan Ngai, now VP of Cloud Services at Lightbend, co-founded OpsClarity at the end of 2012. Their expertise in monitoring reactive applications adds value to Lightbend and their customers that will address four challenge areas for DevOps:
- Containers
- Microservices and reactive frameworks
- Continuous integration
- Technologies in the SMACK stack
InfoQ spoke to Mark Brewer and Alan Ngai to learn more about this new partnership.
InfoQ: What are your current responsibilities at Lightbend, that is, what you do on a day-to-day basis?
Alan Ngai: The acquisition happened just recently, so my near-term focus is to get OpsClarity well integrated into Lightbend and offer additional insight in the OpsClarity roadmap for Lightbend technologies and, in general, find better ways of integrating the products that Lightbend offers. On a day-to-day basis, I'm focused more on providing cloud services for Lightbend.
Mark Brewer: Alan’s role, of course, is to continue to manage the OpsClarity team. He and his team bring some experience that we didn't previously have here at Lightbend. We didn't have any operations of cloud services experience. Certainly our customers are building applications frequently that are running in the cloud. So we have knowledge of how to do that. We haven’t operated a cloud service, or we hadn't, until OpsClarity became part of this company. So we're excited about that - the experience - because there will be more. You'll see more cloud services from this company.
So my role as CEO of the company is to run the company. I've been here now five years and the company is about a year older than that. It was founded in February 2011, exactly six years ago. I joined a year later to be the professional CEO, the guy to run the company and figure out what to take to market. I’ve done this a couple times before. I knew about the company before that, TypeSafe was the name, because they were starting to show up with Scala and Akka at some of our customers at VMware and SpringSource.
InfoQ: Were you involved in Spring moving over to Pivotal?
Brewer: I was part of SpringSource, the company that we sold in 2009 to VMware. Pivotal was the spinout, which happened in 2013. I was involved in the planning for the spinout of Pivotal.
InfoQ: What is the history and timeline of OpsClarity?
Ngai: My co-founders and I started the company at the end of 2012 and, at that time, we were trying to figure out what can we do in this space that can add value. I used to work at eBay and Yahoo in the search and the cloud teams. My co-founders, one worked at Google and Yahoo, and other one worked at Cisco in their data operations business. So we had a lot of understanding of what the space was like. What we saw was the trends of open source and cloud. And what we thought we could bring to the table was our domain expertise and experience with data systems - taking a lot of data and making sense out of the data in a way that's easily digestible and usable. So for 2013-2014, we were essentially building a platform. We had some beta customers throughout, but since we were a very data intensive application, it took a bit of time to fine tune that and collect enough data to have some confidence that the platform would work. We were in a beta cycle for a while, maybe six months, almost a year, and in 2016 is when we went GA. At that time, we had a more refined focus on data applications because we saw the same trends that Lightbend saw, which was more and more people were using Kafka and Spark and building data-centric applications. And in fact, that's where we first met - at a Kafka summit where we were talking about monitoring in the space. Brad Murdoch at Lightbend was looking at the same space and they came to talk to us. That's when we started engaging with each other and then realized that we saw the same vision going forward.
InfoQ: What general comments or background information would you like to share about OpsClarity joining Lightbend?
Brewer: The market opportunities that we're going after at Lightbend are a combination of new kinds of applications as well as legacy apps that are being modernized. And in this new world, it's either microservices architecture as the way you program and design these new systems or it could be somebody who is just taking an old monolithic app and breaking it down into something that's a little more understandable, a little more scalable, and manageable; otherwise known as microservices. And the another interesting trend in the market that we see is companies coming to us for applications that require streaming. The applications themselves are dealing with data in motion as opposed to data at rest. So these two market trends, microservices and data streaming, happen to be one of the hottest things going right now, and honestly, they drive most people to our platform. These trends are encapsulated by the Reactive Manifesto that describes the characteristic that modern applications need to have to meet the requirements of today’s realities.
So the acquisition of OpsClarity came about because two things: one - because they were working closely with us to enable people to see inside of those applications that were created using Lightbend technology, see inside of Akka, and see the messages going between Akka mailboxes or clusters of Akka services that are being run; as well as monitoring things like streaming technology, things like Kafka. We were working on a partnership to bundle their product in with our platform and the opportunity came up to acquire the company. It just made sense to put the two together and now our customers get both; our product and the OpsClarity product in a single offering.
InfoQ: That actually answers one of our questions. We were going to ask about OpsClarity's heritage and what their customers were using before joining Lightbend. It seems like OpsClarity joining Lightbend was a natural fit.
Brewer: It was a natural fit. And something else I should tell you is that OpsClarity designed and built their system using Lightbend technology. So a lot of it is written in Scala and Akka. That helped make the integration a lot easier, but also just made the decision to bring them on board a lot easier.
Ngai: The only color I would add is the landscape that we saw was very much the same that Lightbend saw, in that more and more companies were building their software stack on open source technologies in a distributed fashion. And what we saw on the operational side was that for DevOps and engineers, it was becoming harder and harder for people to understand what's actually going on in their environment, much less monitor it. Whereas Lightbend was helping customers build such distributed systems, OpsClarity was designed for helping people monitor such systems, so it was a pretty natural fit.
InfoQ: For our readers that may not be familiar with OpsClarity and what it means to monitor reactive applications, can you provide a brief overview of monitoring and what technologies were used, assuming that's not proprietary?
Ngai: I think the crux of it is that people are using more and more variety of open source technologies to build their applications and, even within the Lightbend stack, you have Play, you have Akka, you have Lagom, and you have Scala. And most applications these days require you to have other pieces such as Kafka for a message bus, or some NoSQL databases, or some data processors such as Spark or Flink. That's really the environment that we were designed for because we realized that it would be impossible for any engineer or DevOps to be an expert in all these systems, much less monitor them, much less know what to monitor. And so the crux of the platform of OpsClarity is that the information about such systems was built into the product. Rather than having to have a Kafka expert in your team, and a Cassandra expert, and a Spark expert, and an Akka expert, we've included a lot of kind of the knowledge about how to monitor these systems, what the failure patterns are, what are the key metrics, and how to monitor them into the product itself. And I think that's where OpsClarity provides the most value for our customers in the distributed world.
Brewer: One thing that we saw, and I'm sure OpsClarity was seeing, is in this new world where you have containers and functionality of microservices that just live for very short periods of time, it's not like the old monolithic app server days of WebSphere and WebLogic where the server is always up and running in the application. Parts of it were used all the time and parts were used only sparingly. In this new distributed and deconstructed world of microservices, you could have a service running on a Docker container or standalone that only persists, or only lives, for a matter of seconds. And having monitoring, or being able to monitor that is, first off - very difficult; second - just not possible with the old monitoring solutions. That just wouldn't work in this new world.
InfoQ: Microservices has been an interesting topic over the past few years. Based on the Enterprise Developers Trends 2016 report, there seems to be an even distribution of organizations (a) using microservices in production, (b) considering using them, (c) experimenting with microservices in a sandbox environment, or (d) no interest at all. There is also plenty of literature warning developers on using microservices including, “The Seven Deadly Sins of Microservices.” The takeaway message seems to be “use microservices with care.” Is there anything Lightbend can do to address those particular concerns?
Brewer: We're certainly trying. One - it's just education - teach people how to architect, build and design these systems the right way. There are a number of books that we've published. They’re generally really short, easy reads defining microservices, what's the right way for you to think about your system and how you would architect it the microservices way. And then the second is provide the right tools for developers, give them tools that help them as they build these new systems. So we launched a project last year in February called Lagom, and that's our microservices framework. It's a Swedish word - it means “just right” or “just the right size.” That's intentional. Microservices aren't necessarily micro or tiny, it's not so much about the size, it's more about what it does. It does one thing very well and it does it in isolation, meaning that it can scale and fail in isolation without bringing down the whole system if something bad occurs. The Lagom framework is designed purely to help people get started building systems or microservices the right way. There's more we can do, of course, but that's a good start.
Ngai: To the point of monitoring the systems, Lagom and Lightbend technology is leading the effort in trying to help the industry build such architectures: streaming, asynchronous, message-driven, etc. In addition to helping developers build and monitor these systems automatically, will be a need to inform the market about the key operational concerns for these new types of distributed systems. For example, if you look at, historically, what people care about in terms of web servers: request latencies and request throughput, in the asynchronous message-driven world the metrics would be different. For example, back pressure is a big thing in Lagom that operators will actually care about in the system. So how do we service that in terms of monitoring and kind of the metrics that we provide to such architectures?
InfoQ: Alan stated in the press release, “By joining forces with Lightbend, we bring advanced monitoring capabilities to the Reactive Platform for real-time collection and analysis metrics, automatic discovery of dynamic components across distributed systems, and built-in intelligence about the specific open source frameworks.” Can you expand upon that for our readers?
Ngai: Monitoring is more than just collecting metrics. It's not just talking to an API or talking to a JMX endpoint and just collecting metrics and drawing them to a dashboard. I think most monitoring tools can do that. I think what we want to provide to the developer and operational engineers is basically the insight. Given a service, what do they actually care about? Every service exports hundreds of metrics these days - the types of various dimensions like topic name, partition name, or things like that. But which metrics, which performance characteristics about these systems actually matter? What should the DevOps and the engineer keep an eye on? What are the four or five metrics that they should pay attention to? And then for each of those metrics, what kind of characteristics do they exhibit? Is it a group of metrics, a latency metric, a back-pressure metric, etc. And based on what type of metric it is, you would monitor them differently. So the intelligence that OpsClarity provides is basically an understanding of what these metrics are, how to monitor them, and just provide them out of the box for the user. I think going forward we're going to be focused more on the Lightbend components and really understand, in addition to how customers should build kind of the architecture of using Lagom, Play, Akka, etc., what are the important performance and operational characteristics of the systems that people need to pay attention to.
InfoQ: The Enterprise Development Trends 2016 report also stated, “If you’re still releasing software every 12-18 months, you might as well go back to the Waterfall model.” In the press release, one of the four common use cases states, “Those running a dynamic environment with continuous integration, which implies new code is pushed frequently, often multiple times per week, if not days.” What do you feel is an optimum development cycle time for software updates?
Brewer: I think it varies, obviously, by organization and both their expertise or experience in doing agile development as well as the kind of project they’re building. Some demand much more frequent updates or releases. You've got companies like LinkedIn that does a release at least once an hour, sometimes multiple times an hour, where they’re constantly pushing updates. That’s just their practice and they've been doing that for years and they operate that way. The teams know that they're going to be pushing new code out virtually every hour. That doesn't mean everybody has a change to their code every hour. So if they want to catch one of those cycles, they can get their commits in by then. And then you have companies, some of our customers, who are really pushing it to do a release once every month. They're used to one release every 18-24 months. I don't know that there is an ideal, but we haven't seen a consistent, “here's the ideal agile development practice” when it comes to how often somebody pushes code out.
Ngai: It’s pretty much all along the spectrum. I think it really depends on the technology stack teams build on. So if you're more legacy, kind of the previous generation, then the tendency would be longer cycles. If you start using Docker and Mesos, and these kind of technologies, almost certainly the cycles will be much shorter because you can spin up whole environments with a command in these kind of environments.
InfoQ: Once an hour, that's pretty intense. What kind of code is LinkedIn pushing? Is it more than just bug fixes?
Brewer: It's functionality and it could be bug fixes as well. Twitter is another one. A lot of these web property businesses, that's their practice. Once a day would be slow for most of them. Many of them are doing it multiple times a day.
InfoQ: Actually, that's pretty enlightening. Are most people aware of this?
Brewer: There's some content out there, but you’d be surprised at some of these teams. They’re just used to operating that way where there are constant updates being pushed. Even here at Lightbend, some of the teams operate very rapidly and the way their agile development works versus others that are a little slower.
InfoQ: We would hope that waterfall is a thing of the past.
Brewer: There's still an argument that if you're building a monolithic app, something that's just a big old application, and still using the same technologies of yesterday, there’s no reason to switch or force yourself into agile, just keep using Waterfall. I’m not an advocate of it, but it's possible for you to do that.
InfoQ: It's a big commitment to move towards agile. There seemed to be a lot of resistance in doing so, say, 15 years ago with the advent of pair programming and other attributes to eXtreme Programming. We hope that, today, most companies are moving towards agile especially with all the existing J2EE apps and Oracle’s seemingly lack of support for J2EE 8. We’d be interested to know how many organizations are still using J2EE.
Brewer: It's a lot more than you think. I would say that many of those companies that are using J2EE are also embarking on newer technologies. They’re using Lightbend technologies and others for new projects, but they still have a lot of applications in production. A lot of those that need to be maintained that are written in the old style, running on old app server, running on WebSphere or WebLogic.
Ngai: I think there’s certainly an understanding of the continuous development, continuous integration kind of movement. Most of the engineers I've talked to, that I interact with, even if they don't practice daily release, hourly release, or even weekly release, they're certainly aware of that. And there's an understanding that's where everybody wants to go, wherever you are right now, that's where you want to go. So there's a drive towards that.
InfoQ: SMACK - Spark, Mesos, Akka, Cassandra, Kafka. Interestingly, they are all part of Apache with the exception of Akka. As your website describes, Lightbend leverages all of those aforementioned Apache products.
Brewer: Akka is not Apache. We license it under the Apache license, but it is owned by Lightbend so it's not an Apache project, but you're right, the rest of those are all Apache projects. I think the commonality across them is not just that they’re Apache projects, but they are frequently used together for some of these new kind of streaming based applications. I would say the product that OpsClarity has, and now Lightbend has, is largely built on that SMACK stack. They took the same technologies pulled them together to create what is now the next generation monitoring platform. It's the same thing we're seeing in our customers especially those that are dealing with lots of data, data that's in motion. They want to be able to deal with that data in the application in real time, not wait for it to be written to disk, and then read later. The other kind of commonality across all of these, which is not very obvious, they are mostly written in Scala. So you know that we’re the company behind Scala.
InfoQ: What's on the horizon for Lightbend moving forward now that BoldRadius and OpsClarity have joined the team?
Brewer: I think the combination of these two companies that we brought in help us deliver on our promise. One - being a services team that can help people get trained and implement and take advantage of these technologies. We're not a consulting shop where we do the development work for people, however. We now have expertise to provide knowledge transfer and enablement services like best practices and training. And then with OpsClarity we provide a commercial product that we can help somebody get better insight into their applications, how they're performing, and plan for how that application is going to operate going forward. So, what's next? I don't have anything on the docket that we're actively pursuing. But you will see, as I mentioned earlier, you will see Lightbend provide more cloud based services. Monitoring is the first one, but you'll see others from Lightbend that are higher value services for companies who build applications with our technology. Our job is to make them successful but also to make sure that when they build these applications, they get the most out of it.
Related Resources
- Monitoring and Troubleshooting Real-Time Data Pipelines video by Alan Ngai and Premal Shah.
- Lightbend Acquires OpsClarity: Interview with CEO Mark Brewer video by Oliver White.