Zesty recently released Zesty Disk, an automated scaling solution for AWS EBS. Zesty Disk monitors EBS performance metrics and can automatically scale the cluster size based on those metrics. This is done by creating a cluster of EBS volumes that can be attached or detached as needed based on system usage.
The solution requires the installation of an agent on the selected devices. The agent requires an outbound connection to Zesty's platform over port 443. At the time of release there are a select number of Linux OS machine types supported.
The solution leverages Btrfs, a file system that combines the copy-on-write principle with a logical volume manager. One of the key features of Btrfs that Zesty is leveraging is the multi-device spanning within Linux file systems. Chris Mason, one of the principal developers of Btrfs, stated that the goal of Btrfs was to:
let [Linux] scale for the storage that will be available. Scaling is not just about addressing the storage but also means being able to administer and to manage it with a clean interface that lets people see what's being used and makes it more reliable
As Zesty notes:
Leveraging BTRFS advanced file system provides the means to perform online resizing and defragmentation to EBS volumes and enables Zesty to merge or detach storage volumes while the file system remains online. BTRFS unlocks decrease and increase of storage, in real-time, while maintaining desired workload performance.
The multi-device spanning capabilities of Btrfs enable the usage of a cluster of EBS volumes. The agent monitors both system performance in terms of disk utilization and read/write metrics to determine how to scale the cluster. While the system is able to automatically add or remove volumes as needed, there is support for the creation of policies to indicate how events should be handled. This includes setting a ceiling on the amount of volumes that can be added in the event of additional scale being required.
Expanding the drive involves adding additional volumes to the provisioned clustered storage. The addition of new volumes also improves the available IOPS to the system providing a performance boost. The shrinking activity can have a negative performance impact as prior to detaching drives, files need to be moved around to free up the sectors on the cluster to be dropped. These events are automatically scheduled during low usage periods. The Zesty agent monitors the disk metadata to determine where the data blocks reside and which can be released to allow for detaching the disk.
Resizing events can be performed automatically or require manual approval to proceed. During an automated resizing event, notifications can be sent to Slack or Microsoft Teams. Manual approvals can also be sent to Slack or Teams for review and human approval.
As the solution is built on top of AWS EBS, all data-at-rest resides within the customer's account. The agent only requires access to the disk metadata to determine the correct course of action.
The team is investigating adding Windows support in a future release. More information about Zesty Disk can be found on the Zesty website.