GameAnalytics, maker of a free analytics platform, has recently open sourced gascheduler an Erlang library that provides a generic scheduler for parallel execution of distributed tasks. InfoQ has spoken to Chris de Vries, one of gascheduler’s creators.
Gascheduler’s design is based on a few key ideas:
- The scheduler accepts requests from clients and handles them through a pending queue and running queue.
- A client sends the scheduler a callback, that goes into one of the two queues.
- When a node is available to handle a request, the scheduler moves it from the pending queue to the running queue and dispatches it to the node.
- Nodes have a pool of processes available to run the callbacks.
- When a node is done, the task status is sent back to the client asynchronously.
- In case of failure, a worker node will do a retry, possibly an infinite number of time unless the failure is of permanent type.
The diagram below depicts the flow of messages between the client, the scheduler, and nodes.
Once a scheduler is created and started, a client will tipically use it as in the following code snippet:
%% Execute hello:world(1) asynchronously.
%% In the hello module exists, world(N) -> N.
ok = gascheduler:execute(Name, {hello, world, [1]}),
receive
{Name, {ok, Result}, Node, MFA = {Mod, Fun, Args}} ->
io:format(“hello world ~p from ~p~n”, [Result, Node]);
{Name, {error, Reason}, Node, MFA = {Mod, Fun, Args}} ->
io:format(“task ~p failed on ~p because ~p”, [MFA, Node, Reason])
end
The callback that is passed to the scheduler should be side-effect free or idempotent.
InfoQ has spoken with GameAnalytics to learn more about gascheduler.
Can you shortly explain what GameAnalytics does?
GameAnalytics is a free analytics service created by and for game developers. We have designed event types and metrics that are specific to gaming, such as player engagement, monetization, and player progress, besides more general metrics such as user acquisition, error tracking etc.
We also provide SDKs for developers to make integration easy. It is possible to get started in 5 minutes.
How does Erlang fit in the picture?
GameAnalytics uses Erlang to build soft real-time fault tolerant systems that can handle large-scale web traffic. Erlang is also an advantage in a startup because we can ship code quickly, perform hot code loads, and easily handle all the moving parts of complicated distributed systems with minimal effort.
We have also used Erlang’s integration with modules written in C via Native Interface Functions (NIFs). In another one of our open source projects implementing the HyperLogLog data structure, we made significant performance gains by using a native implementation. Erlang allows quick prototyping in a dynamic language with great safety features for parallel and distributed processing, but it also allows us to easily optimize the hottest parts of code in a native language like C.
How do you use gascheduler inside of your products?
The gascheduler library was refactored from existing code where the separation of infrastructure code and business logic was not so well-defined. Initially it was used for processing small batches of events where we interact with a few other distributed systems to annotate the events. We have now reused the code in other backend systems that require a queue of tasks. At times we can queue many thousands of tasks in the scheduler but we only want a limited number running at one time due to limits on system resources.
We also want these tasks to be restarted if they fail. Things can go horribly wrong all the time when dealing with distributed systems. The scheduler also allows us to execute tasks on many machines. We can scale the compute capacity as needed. This is important for us because we are always seeing more traffic in an expanding industry like gaming. Erlang provides great support for monitoring such activities. Being a C++ programmer for many years, I can certainly say the Erlang solutions to these problems are much more concise, readable and understandable. The entire functionality fits in one module and is quite easy to digest in a single code review session. We are leveraging the wise design choices of the Erlang VM.
Could you describe the rationale behind gascheduler’s design?
The design rationale of gascheduler is simplicity. We have some tasks that take some input and produce some output. We want to run these on multiple machines in a compute cluster and have them restarted if they fail. The library can be seen as a simple implementation of the map in MapReduce where we do not do any of distributed file system work. This is left up to the application implementing the scheduler. For example, a developer may have their data stored on a distributed file system such as AWS’s S3 and chose to use the s3erl library. We essentially manage a queue of callbacks and make sure they are executed in the case of failure such as when a compute node goes down or an exception is thrown. This also brings us to another design rationale. Tasks must be able to be run multiple times without causing consistency issues. We generally deal with this problem by making our operations idempotent, where applying the task to the state reaches the same answer, even if it is run more than once.
In the future we may consider using a multi-master design where there is more than one master node in the cluster. But for the moment we have found a single-master design to be sufficient. However, it does result in a single point of failure.
What drove you to create it as it stands?
The separate library was created from real use cases we have in processing event data. The main driver for the creation of this project was code reuse. We wanted the same functionality as we already had but the code was not as reusable as we would have liked. So, we thought we should also open source the library in case anyone else can make use of the code.
We use tasks instead of processes because we have limited resources on these systems such as memory, disk IO, and network IO. If we start everything at the same time we can run into contention problems or even exceed the available physical memory on a machine. The tasks we run are generally fairly long running in Erlang terms and take seconds to process, rather than the millisecond response times it is well known for in applications such as messaging.
A gascheduler task is a thin abstraction over an Erlang process. As a user does not have to think about how and where the work is done. Gascheduler automates the assignment of work to available compute resources. Each task just ends up being executed in its own Erlang process, but all the boilerplate fault tolerance stuff is done by gascheduler through this abstraction of a task. It is this logic which we found we can reuse.