At QCon New York 2017 Michael Fisher presented "Presidential Campaigns & Immutable Infrastructure" and discussed the implementation and challenges of provisioning infrastructure for the Hillary for America (HFA) campaign that ran during the 2015-2016 US regional and national elections. Immutable infrastructure was key to the technical success of the campaign - the team moved quickly, but were resilient against failure for the majority of the time. It can take more effort to apply the principle of immutability to everything being deployed, but it is beneficial and developers "like the handshake between SRE and dev".
Fisher, currently a Software Engineer at the Democratic National Committee (DNC) and acting as a Site Reliability Engineer (SRE) Manager during the HFA campaign, began the talk by demonstrating how variable unexpected web traffic can be when running an election campaign. Although the HFA team had requested that high-ranking politicians notify the campaign team when a public recommendation to visit the HFA website was to be made, not all did. For example, when Senator Al Franken made an unexpected announcement on US national television, a spike in traffic was seen, resulting in up to 148k requests per minute over a duration of ten minutes. This traffic was served successfully due to the implementation of extensive caching, the reliance on static websites, and autoscaling of immutable stateless applications.
The HFA technical team consisted 80+ technical staff, and provided infrastructure that supported 60 elections running over 577 days. Ultimately over 100 backend services powered the campaign websites, and 150 serverless frontends were deployed. The team chose to provision and run all backend services as immutable infrastructure, creating Amazon Machine Images (AMIs) using HashiCorp Packer and Travic CI for deployment onto AWS EC2, and Ansible to orchestrate the underlying infrastructure such as Varnish caching, networking and AWS RDS Aurora MySQL databases. Using immutable infrastructure increased reliability and repeatability, and although conceptually this was simple to implement, making it work for everything and getting all members of the team on board was challenging.
Next, Fisher discussed the inherent security challenges of running a political campaign website, and stated that "everyone is coming for you". This included "script kiddies" and bad actors attempting to issue denial-of-service (DoS) attacks on the websites. The scriptable capabilities of the Varnish edge cache, Varnish Configuration Language (VCL), was used to deflect incoming requests for geographically targeted areas, prevent Wordpress pingback attacks, and provide obfuscated error messages that made it appear attacks had been successful when in fact they had not.
Many of the campaign websites were served via as static pages deployed via Amazon S3 and fronted with the Fastly CDN, which scaled very effectively and also provides local points-of-presence (PoPs). Additional edge caching was provided extensively via the integration of Varnish within Fastly. The remainder of the requests were served via an API Gateway that distributed load to autoscaled EC2 instances running application services.
Fisher also described how business stakeholders had required WordPress instances to be running later in the campaign, as they were familiar with the CMS technology and workflow, but the technical team did not want to expose this as a service directly due to the challenges of scaling and securing this stateful application. Instead new articles published to a backend WordPress instance were extracted in JSON format, pushed into an ElasticCache Redis cluster, and displayed to users via a hardened stateless Node.js-based application. This Node.js service was named "Secretary of Stateless".
Summarising the talk, Fisher stated that immutable infrastructure was key to the technical success of the campaign. On a presidential campaign innovation is a necessity, and "there aren't any hard and fast rules in technology" - the team moved quickly, but were resilient against failure (most of the time). It can take more effort to apply the principle of immutability to everything being deployed, but it is beneficial and developers "like the handshake between SRE and dev". The final closing remark from Fisher was that is it possible to "build cool technology and work in public service".
Additional information on Michael Fisher’s presentation "Presidential Campaigns & Immutable Infrastructure" can be found on the QCon New York website, and the video recording of the talk will be released on InfoQ over the coming months.
