Earlier last month in Las Vegas, at IBM Insight 2015, IBM announced a major commitment to the Apache Spark project. Referring to it as “potentially the most significant open source project of the next decade” tells a lot about how important IBM believes Apache Spark is. With IDC reporting that 80% of cloud applications in the future will be data intensive, Apache Spark with its real time analytics capabilities can unlock previously unusable data into into useful insights. IBM had already announced an investment of $300 million over the next few years on Apache Spark last June at Spark Summit. This investment included 3,500 IBM developers working on Apache Spark and related projects over the next few years.
The newest announcement expands on the previous offering as IBM doubles down on Apache Spark and betting in its adoption for IBM’s cloud success. IBM announced Spark as a Service as a core offering in its BlueMix platform. This means that developers and data scientists can get real-time analytics in their apps. IBM’s dedication to Apache Spark means that it will rewrite some of its applications to include Apache Spark. These include BigInsights, Data Science Workbench and SPSS Analytics Server and Modeler.
IBM is also open sourcing its SystemML machine learning technology and will collaborate with Databricks in Spark machine learning development. Finally, IBM plans to participate in design of classes both onsite and MOOC through partnerships with AMPLab, DataCamp, MetiStream, Galvanize and Big Data University, aiming to educate more than 1 million data engineers and scientists.
IBM analytics for Apache Spark is available as free trial and starting from $0.7/instance hour.