A new platform for automating cloud infrastructure, System Initiative, has now reached general availability.
System Initiative aims to tackle some long-standing problems in conventional industry practices for deploying and managing cloud infrastructure. Specifically, existing solutions requiring static configurations for cloud infrastructure cannot reliably reflect fast-changing and dynamic cloud services, leading to "stack drift": unpredictability when engineers apply changes to infrastructure.
In the press release announcing the tool, DevOps Luminary John Willis explains how attempting to map static definitions onto dynamic and frequently changing infrastructure has caused organisations to build cloud systems that are brittle and opaque. In a blog post announcing the platform, Adam Jacob (CEO and Co-Founder of System Initiative) explains how the feedback loops for deploying infrastructure are too slow and restricted by the implementation details of the platforms beneath. System Initiative's core concept is one of "digital twins," which essentially decouples the monitoring of infrastructure status from potential modifications made by cloud engineers. This methodology enhances feedback loops significantly, allowing engineers to verify configurations without waiting for physical provisioning to happen.
"The path we have been on, where we build infrastructure with the same approaches and tools we use to build applications - is a technological dead end." - Adam Jacob, CEO and Co-Founder, System Initiative.
Speaking to InfoQ News Editor Daniel Bryant on an InfoQ podcast last year, Adam Jacob explained the need for this platform.
"The way we've decided to work and the way we've decided to put the systems together to allow us to do this work, that's what's holding us back. And if we want an order of magnitude better experience, then we need to design a system for that experience and we need to take everything we learned in the last 14 years plus of doing this kind of work" - Adam Jacob
The platform intends to solve the "200% problem" often seen in automation abstractions, a description given to the scenario where cloud engineers must still understand the infrastructure underneath an abstraction, where that abstraction is supposed to mask the complexity of the underlying platform.
In explaining the motivation for the product, Jacob continues, "the number one thing holding us back was actually that the canonical representation of what we wanted wasn't code. What we need to collaborate on is the intersection between the model of an application and the infrastructure that runs it".
System Initiative's design uses a network of functions that make digital replicas of infrastructure easily customisable and programmable. This network makes creating and adjusting models simple while facilitating policy implementation and linking elements together. There's also a comprehensive version control mechanism with a "Change Sets" feature that allows experimentation without disrupting operational environments. This mechanism automatically adjusts to mirror real-life updates to infrastructure.
Jacob went on to describe how modelling the functionality of AWS services in System Initiative enables tighter and more reliable feedback loops for infrastructure deployment.
"What if we said the canonical representation was a simulation and what you did was program the simulator to behave the way you wanted it to behave?" - Adam Jacob
System Initiative currently works only on AWS assets. In a follow-up to the launch post on LinkedIn, System Initiative explained that they had prioritised AWS to ensure that the core engine worked correctly and that the system was extensible to other providers, such as GCP and Fastly. System Initiative also has a centralised module index, so community sharing of assets is a single-click operation.
System Initiative aims to improve the experience of deploying cloud architecture with a Living Architecture diagram - allowing engineers to understand complex configurations visually. It has fine-grained controls over making infrastructure changes, allowing engineers to recover quickly from failures and adapt to infrastructure changes. It maintains current change sets for teams to propose, validate, and implement updates securely as environments evolve. The tool claims to need no other platforms or state file management.
"System Initiative looks great for being able to validate config and stitch together configurations with visualisation which looks great for complex platforms. Change proposal system looks like another strong addition with a Figma-esque approach." - George Tunnicliffe on LinkedIn
Some community reaction has focused on the tool's positioning, with several commenters erroneously suggesting that launching a visual tool conflicts with the commonly understood best practice of using configuration-as-code. These reactions indicate the tool's positioning in improving feedback loops in deploying cloud infrastructure is yet to be well understood.
"I have no clue if System Initiative is really a viable idea. It is an incredibly audacious vision, which means that it is simultaneously more likely to fail and more likely to succeed. Few would care about a Terraform that is 5% better at managing infrastructure, but a system that is possibly 10x better? Now you have my attention." - Jonathan Yu on LinkedIn
The system is open source as part of System Initiative's support for community-driven development efforts. This encourages engineers to expand and customise the system's capabilities to suit their requirements.
Since launching an early access programme in June 2023, System Initiative has assembled almost 3000 registrations on the platform, with 1600 engaged developers using it. The pricing strategy is based on usage, with a free tier described as substantial enough to sustain an organisation's operational infrastructure for actual production purposes. System Initiative is available now.