At the fifth HashiConf US conference, held in Seattle, the HashiCorp team made several new feature announcements for their Terraform and Consul products, including the full release of the SaaS-based Terraform Cloud workflow management platform and a private beta release of HashiCorp Consul Service on Azure, a fully-managed Consul (service mesh) platform. Key takeaways from the event included: the software delivery world is becoming multi-cloud, multi-platform, and multi-service, and engineers have to adapt to this; focus on implementing effective engineering workflows, not specific tooling; and there is still much that operations teams can learn from developers, and vice versa.
The conference opened with the HashiCorp co-founders, Armon Dadgar and Mitchell Hashimoto, setting the context of the shift within software delivery to the “multi-*” world and discussing the related HashiCorp Cloud Operating Model. Covering the topics of provisioning, securing, connecting, and running, Dadgar described the changing workflows and requirements that teams must acknowledge when embracing multiple clouds and platforms. He then moved on to outline related new features added to Terraform, HashiCorp’s cloud-agnostic infrastructure as code (IaC) offering.
Earlier in the year HashiCorp announced Terraform Cloud Remote State Management, a free SaaS-based platform that enabled users to collaborate on Terraform configurations via remotely stored, versioned, and shared Terraform state files. This release complemented the commercial Terraform Enterprise product, which is a self-hosted platform with additional features that allows users to collaborate via shared workspaces, plan and apply infrastructure modifications, and enforce policies remotely, in addition to storing state.
At HashiConf US the complete version of Terraform Cloud was announced. This release adds new features to the SaaS platform that brings this into close parity with Terraform Enterprise, and enables up to five users to collaborate on and automate Terraform workflows without the need to manage a self-hosted platform. Terraform Cloud is offered in three tiers: Free, Team, and Team and Governance. Team provides support for additional users over the allowance in the Free tier, and Team and Governance adds support for Sentinel-defined policy-as-code and cost estimation on top of this.
Also announced was the beta version of clustering for HashiCorp Terraform Enterprise, which enables organisations to install and manage a scalable cluster that can meet additional performance and availability requirements. The beta version of Clustering supports installations of Terraform Enterprise in AWS, Azure, and GCP.
Led by Hashimoto, a second series of big announcements focused on HashiCorp’s Consul product. HashiCorp Consul Service (HCS) on Azure, was released in private beta, and this is effectively a fully-managed Consul-as-a-service offering.
Over the past several years, Consul has evolved into somewhat of a communication-focused chimera, as it now provides a distributed key/value store (and other associated distributed systems primitives like locks and watches), a service discovery mechanism, and service mesh functionality via Connect. This is the first time the product has been provided as-a-service by any cloud vendor.
As a fully managed service, HCS on Azure lowers the barrier to entry for an organization to leverage Consul for service discovery or service mesh across a mix of VM, hybrid/on-premises, and Kubernetes environments while offloading the operational burden to the site reliability engineering (SRE) experts at HashiCorp.
Additional Consul announcements included Consul Enterprise supporting VMware NSX Service Mesh Federation and Consul Connect integration with the HashiCorp Nomad workload scheduling framework.
VMware announced a new open specification for Service Mesh Federation earlier in the year at VMworld, and this defines a common standard to facilitate secure communication between different service mesh solutions. The new partnership will provide a service mesh federation between Consul Enterprise and VMware NSX Service Mesh (NSX-SM).
The beta release of Nomad 0.10 extends the workload scheduling framework’s Consul integration capabilities to include native Connect (service mesh) integration. This enables services being managed by Nomad to easily enable mutual-TLS (mTLS) communication between services, without developers having to make additional code changes to their application. Prior to Nomad 0.10, job specification authors would have to directly run and manage Connect proxies (like Envoy Proxy), and did not get network level isolation between tasks. This new feature is implemented by the introduction of two new stanzas to Nomad’s job specification: connect and sidecar_service.
David McJannet, HashiCorp CEO, also took to the keynote stage and emphasised the importance to HashiCorp of continuing to support the ecosystem in relation to their tools, partners and customers, and also the wider open source community. He stated that "cloud is a generational shift in the way applications are built and delivered", and empathised that training must be provided for engineers to embrace new workflows and tooling. In relation to this there was much discussion at the event about the new discuss.hashicorp.com and learn.hashicorp.com websites, which aim to support engineers in their learning journey. The topic of learning was also explored in more detail in a previous InfoQ podcast with Armon Dadgar.
The two-day conference ran from 9am to 5pm and provided four tracks of sessions, with a mix of presentations by technical staff from HashiCorp and engineers from end user organisations. Your correspondent attended several sessions, and although the general quality of talks was high, two sessions stood out for their focus on solving real world problems, narrative, and technical depth.
In the talk “Infrastructure as Code for Software Engineers”, Mike Gee and Ryan Hild, lead and senior software engineer at Starbucks, respectively, shared their journey of how they leveraged their experience as developers to evolve the organisation’s web platform infrastructure code, tooling, and processes. Key takeaways for this session included: embrace modularisation of declarative infrastructure code; define standardised workflows, tooling, and scripts, and codify enforcement of these; iteratively create effective delivery pipelines; invest in testing (following the test pyramid); and expose programmatically consumable metrics for architecture, operations, and business performance.
The talk “Inversion of Control With Consul”, Pierre Souchay, member of the SWAT team at Criteo, a real-time bidding advertising platform, shared the experiences of his team as they moved from VM-based infrastructure to a container-based Apache Mesos platform. In this project the ops team worked alongside developers to simplify the management of application metadata, track service updates, and improve application maintenance. A key lesson he learned was to store metadata within the application’s codebase, rather than in a key-value store or a separate configuration repository, as this led to easier to understand applications, less configuration drift, and easier maintenance.
Key takeaways from this talk included: engage with and support the development teams to ensure good operational practices are followed; keep metadata close to where you will use it; invest resources to carefully define and effectively manage metadata, as this is essential for understanding your system; automating the rollout, and clean up, of application configuration is essential when operating within a highly dynamic environment or at large scale; and focus on defining configuration that is business- or semantically-relevant, rather than tool-specific.
Recordings of all the conference talks will be uploaded to the HashiCorp YouTube channel over the coming weeks, and slides for the hallway track talks can be found in a thread on the HashiCorp discuss website.