Marc Andreessen has famously said software is eating the world, and we’re seeing that in spades in the data center. Software is taking over what have always been hardware functions, and that shift is causing dramatic realignments in how people work together.
Fundamentally, the whole cloud movement is about hardware being replaced or abstracted by software. It started with virtualization, which has been around for decades, but only in the last 10 years or so has it become widespread on commodity hardware.
Decoupling hardware from the services that ran on top of it allowed massive server consolidation, which accelerated the commoditization of hardware and further reduced the bottom lines of manufacturers that were already struggling to maintain profit margins. While this wasn’t the best outcome for hardware manufacturers, it has been great for innovative software vendors like VMware. It has also been great for technology buyers.
Pretty quickly, people realized virtualization wasn’t just about saving money — it provided new possibilities. The rate of change of physical infrastructure is limited by the supply chain, along with the time it takes to do things like power on a server or update firmware, but virtual infrastructure generally works via APIs, and thus is dramatically faster. What might take weeks or months to do in physical infrastructure can often be done in minutes or hours with virtual servers and networks.
In 2009, we finally saw more virtual servers than physical. That means most people working with infrastructure today never even make it to the hardware. Ironically, this can create a new silo that didn’t previously exist: a group dedicated to maintenance from the hardware up to the virtual machine. Meanwhile, the rest of the world operates in pure software.
Well, kind of. In truth, most organizations have virtualized their computing resources, and they’ve probably moved to commodity hardware for firewalls and load balancers (basically, x86 boxes running a Linux or BSD with some special software on top). But their storage and network resources are still very much hardware, and even the commodity IP devices living on top of the hardware act as much as possible like devices, rather than behaving like software.
The next big movement is abstracting not just compute to the software level, but also storage and networking. This shift to a completely software-defined infrastructure is opening up great possibilities for the organizations that take advantage of it.
DevOps and Software-Defined Infrastructure
DevOps is neither the cause nor the result of cloud computing, yet it's a closely related movement. It is fundamentally a cultural change that’s directed at overcoming the divide between developers and IT systems administrators, with the goal of deploying better software more frequently and better aligning to the needs of the business.
That divide exists because the interests of developers and operations people are not only different, they can actually collide in a typical workplace. Systems administrators are tasked with keeping IT running smoothly and reliably — no hiccups, no downtime, and the ability to scale on demand, as the business requires it. Devs, on the other hand, are rewarded for releasing new code frequently, to stay out in front of the competition. IT operations people see developers who release buggy code and leave IT to pick up the pieces as making it hard for them to do their jobs. Devs see IT operations people who insist on process and procedures as impeding them from doing their jobs.
When you view infrastructure as software, you can get sysadmins and developers aligned and working together harmoniously by treating the infrastructure as a running application and building software development workflows around operations like application updates and security patches. If you think of infrastructure as servers, switches and routers, this is impossible. But once you see infrastructure as a set of services, treating and operating it as software becomes obvious.
While DevOps is supposed to be about culture, cloud is supposed to be about technology. In reality, though, cloud has to be about both. Technology can’t solve the political problems, but it can help mitigate them. The technology of the cloud enables application developers to work around IT departments that can’t provide a software view of the world. But for those who can acquire a complete software-defined infrastructure, the traditional silos of system, network, and storage are being broken down. A single team can build and deploy new applications, allowing greater agility, higher velocity, and much less time between software being ready and actually delivering value to its intended users.
A New Kind of IT, and a New Kind of System Administrator
We are seeing the rise of a new kind of administrator, one who thinks in terms of infrastructure capabilities and services — not in terms of devices. You already see this thinking in administrators of applications that run in places like Amazon’s AWS. These administrators have no need to think about network or storage, in most cases — they just build their infrastructure around software that Amazon provides.
This reframing of infrastructure as software is now making it into the enterprise itself, usually through private clouds built on technologies such as OpenStack. Even though modern private clouds don’t quite replace the traditional device-based corporate infrastructure, they are pushing organizations very quickly towards treating infrastructure as software. When it’s possible to spin up 1,000 virtual machines in an hour — get them up and running and doing real work — but it takes 10 times as long to get new vlans built or storage allocated because the people who do those things exist in different silos, the organization quickly realizes it needs to change.
As the cloud infiltrates the enterprise, you’ll see a shift to horizontal teams, instead of vertical. Your cloud team will see its job as maintaining an API and an SLA that comprehend all of your infrastructure needs. Your application teams will speak about infrastructure in terms of those same APIs and SLAs, rather than talking about operating systems or hardware needs. The abstraction from hardware to software will move from your technology interfaces to your team interfaces — and your business will see the benefits. You’ll launch new products more frequently, get customer feedback more quickly, and be able to respond to the changing market more quickly. Harnessing infrastructure-as-software and making it a core competency will give you a significant advantage over your competitors.
There’s no question that ultimately, we’ll find all of our applications running in service-level, on-demand infrastructure. The software running on commoditized hardware will itself be commoditized. The vast majority of our applications will no longer be unique snowflakes with highly specific infrastructure requirements — instead, they’ll be able to run on common software platforms that eliminate the need at the application layer to worry about infrastructure at all.
In this world, the vertical silos of storage, network and system teams (not to mention VMware, firewall and load balancer teams) that cause so much suffering — and slow down business — will make no sense. In their place, we’ll have teams that are responsible for reliable delivery of business services. After all, that’s what IT is: a business service.
Is your team working hard to embrace this change, or is it fighting change tooth and nail?
About the Author
Luke Kanies founded Puppet and Puppet Labs in 2005 out of fear and desperation, with the goal of producing better operations tools and changing how we manage systems. He has been publishing and speaking on his work in system administration since 1997, focusing on development since 2001. He has developed and published multiple simple sysadmin tools and contributed to established products like Cfengine, and has presented on Puppet and other tools around the world, including at OSCON, LISA,Linux.Conf.au, and FOSS.in. His work with Puppet has been an important part of DevOps and delivering on the promise of cloud computing. He serves as a board member for the Software Association of Oregon.