Spotify Open Sources cstar: its Cassandra Orchestration Tool

Spotify recently open sourced cstar, its Cassandra orchestration tool. Cstar is a command line tool to runs scripts on all hosts in a Cassandra cluster.

Cstar emerged from the necessity of running shell commands in Cassandra nodes. These shell commands are usually related to performance, security, consistency, upgrade, etc.

According to Spotify, during 2017 the Spotify Cassandra fleet reached 3000 nodes; from there, the necessity of having a safe and efficient solution for running shell commands became increasingly more urgent. An example of a task nightmare was a scheduled upgrade of their entire Cassandra fleet, which required the following steps:

• Clear all snapshots (to have enough disk space to finish the upgrade)
• Take a new snapshot (to allow a rollback)
• Disable automated puppet runs
• Stop the Cassandra process
• Run puppet from a custom branch of our git repo in order to upgrade the package
• Start the Cassandra process again
• Update system.schema_columnfamilies to the JSON format
• Run `nodetool upgradesstables`, which depending on the amount of data on the node, could take hours to complete
• Remove the rollback snapshot

Furthermore, there are a series of other related problems when dealing with Cassandra fleet, such as network failures, ssh connection interruptions, CPU-intensive operations, Cassandra process/nodes restarting without impacting performance and availability, or partially finished executions handling (tasks succeed only in some nodes).

In the early Spotify days, the most common way of running commands on Cassandra nodes was to be typing the command in multiple terminals.

Cstar aims to address these problems. To use cstar, all involved machines should be using some sort of UNIX-like system like OSX or Linux, and the machine running cstar must have python 3.

CStar is run through the cstar command, as follows:

# cstar COMMAND [HOST-SPEC] [PARAMETERS]

The HOST-SPEC specifies what nodes to run the script on. There are three ways to specify a HOST-SPEC:

• --seed-host tells cstar to connect to a specific host and fetch the full ring topology from there, and then run the script on all nodes in the cluster
• --host specifies an exact list of hosts to use
• --host-file points to a file name containing a newline separated list of hosts

Cstar automatically saves standard output, error, and exit status of each command run against a Cassandra host. The results (job status) are available under the user's home directory in .cstar/jobs/JOB_ID/HOSTNAME on the machine where cstar is running.

Below are some examples of cli commands on a cluster:

# cstar run --command='service cassandra restart' --seed-host some-host

# cstar puppet-upgrade-cassandra --seed-host some-host --puppet-branch=cass-2.2-upgrade

Here is an example of a script that upgrades a Cassandra cluster by running puppet on a different branch, then restarts the node, and finally upgrades the stables:

# !/usr/bin/env bash
# C* cluster-parallel: true                                                                                                                                                                                    
# C* dc-parallel: true                                                                                                                                                                                         
# C* strategy: topology                                                                                                                                                                                        
# C* description: Upgrade one or more clusters by switching to a different puppet branch                                                                                                                       
# C* argument: {"option":"--snapshot-name", "name":"SNAPSHOT_NAME", "description":"Name of pre-upgrade snapshot", "default":"preupgrade"}                                                                      
# C* argument: {"option":"--puppet-branch", "name":"PUPPET_BRANCH", "description":"Name of puppet branch to switch to", "required":true}                                                                       

nodetool snapshot -t $SNAPSHOT_NAME
sudo puppet --branch $PUPPET_BRANCH
sudo service cassandra restart
nodetool upgradesstables

The full list of cstar commands and more details about can be found at cstar repo.