AirFlow recently joined the Apache Incubator program. AirFlow is a workflow and scheduling system designed to manage data pipelines. Developed by AirBnb for their internal usage, it was open sourced last September, as previously reported by InfoQ.
The project has expanded outside Airbnb’s usage to the industry, adding 2 new companies using it every week, around 10 new contributors and 20 new users. At the moment with 30 companies using it in production, 8 committers and 135 contributors, it has a solid base for expansion.
Stacking up against Azkaban, Luigui and Oozie, Airflow has the key differentiator that pipelines are defined as code and tasks are instantiated dynamically. Ease to setup and being developer friendly are also among the design considerations for Airflow’s development.
Joining the incubation program, Airflow is on its way to enter the Apache Software Foundation (ASF) list of projects, along with projects like Impala, DataFu, AsterixDB, Zeppelin and others. Airflow’s code is available at Github and the project is always looking for new contributors.