Jenkins recently announced a partnership with Microsoft to run its infrastructure on Azure. This will include, among other things, Jenkins developer infrastructure - its developer wiki, issue tracker, databases and static content. Moving to Azure will enable elastic workloads as well as additional resources for Jenkins services.
The current infrastructure is composed of a mix of physical and virtual machines, with the core machines hosted at the OSUOSL and the remaining split between AWS, Rackspace and a physical datacenter. InfoQ reached out to R. Tyler Croy, Jenkins Community Lead, for more details on what this partnership means for Jenkins.
As is usually the case with projects that grow organically, the Jenkins infrastructure has grown in bits and pieces and consolidating to a single cloud platform will benefit several areas, according to Croy. Would this consolidation lead to a major redesign in the way the infra services work? Croy says that they are “re-evaluating the infrastructure design most in our distribution/download center and build/test services that we offer to core and plugin developers (i.e. Jenkins on Jenkins).” He further adds that:
The migration itself will bring a number of benefits, by using scale-sets by default in a number of services, hosted/scalable database backends for various DB-backed services and having the ability to define our actual infrastructure topologies in code, since everything will be an API away. Leveraging ready-made services (e.g. CDN, Azure Container Service, SQL Server) also removes portions of our infrastructure, lightening the burden of what we have to operate.
Moving to the cloud will also enable better handling of unpredictable traffic. In case of Jenkins, the traffic depends on the service. Some services such as the developer infrastructure (wiki, issue tracker, miscellaneous services) grow predictably over time as the community grows, whereas other services like the distribution network or the build/release infrastructure have much more elastic workloads, says Croy. This elasticity applies to both compute and network throughput requirements. The recently launched Jenkins 2 caused a huge increase in requests to the distribution network.
Security is another aspect that needs to be considered when moving workloads to the cloud. Jenkins reported a possible security incident reported last month. The attack vector for that incident was due to too many services running on a single asset in the current infrastructure.
“Under-resourcing of project infrastructure in Jenkins leads to machines having too many services co-located on them. Balkanizing our infrastructure in Azure with a minimum (appropriately sized) instance-per-service helps, plus the ability to layer multiple virtual networks together will help isolate and prevent any potential future issues”, added Croy, elaborating on this point.