InfoQ Homepage Data Pipelines Content on InfoQ
-
Future of Data Engineering
Chris Riccomini talks about the current state-of-the-art in data pipelines & data warehousing, and shares some of the solutions to current problems dealing with data streaming & warehousing.
/presentations/data-engineering-pipelines-warehouses/en/mediumimage/Chisbig-1575474871986.JPG)
-
Building and Operating a Serverless Data Pipeline
Will Norman discusses the motivations of switching to a serverless infrastructure, and lessons learned while building and operating such a system at scale.
/presentations/serverless-data-pipeline/en/mediumimage/Will-Norman-m-1571264580667.jpg)
-
Announcing Broadway
José Valim discusses how Broadway connects multiple stages and producers, how it leverages GenStage to provide back-pressure, and other features such as batching, rate-limiting, partitioning and more.
/presentations/broadway/en/mediumimage/Jobig-1568591143171.JPG)
-
Cloud-Native Data Pipelines with Apache Kafka
Gwen Shapira discusses how data engineering requirements changed in a cloud-native world, and how the solutions change with them.
/presentations/cloud-native-data-pipeline-kafka/en/mediumimage/Gwebig-1564871370264.JPG)
-
Event-Driven Architectures with Apache Geode and Spring Integration
Charlie Black deploys Spring Integration pipelines to react to changes of the data stored in Apache Geode.
/presentations/apache-geode-spring-integration-pipelines/en/mediumimage/Chabig-1553140295208.JPG)
-
Designing Automated Pipelines for Unseen Custom Data
Kevin Moore discusses some challenges in designing automated machine learning pipelines that can deal with custom user data that it has never seen before, as well as some of Salesforce’s solutions.
/presentations/design-automatic-pipelines-ml/en/mediumimage/Kevin-Moore-m-1547037948084.jpeg)
-
Zero to Multi-Cloud
Marcin Grzejszczak, Jon Schneider discuss using Spring Cloud Pipelines and Spinnaker together.
/presentations/spring-cloud-pipelines-spinnaker/en/mediumimage/Jobig-1544244214507.JPG)
-
ML Data Pipelines for Real-Time Fraud Prevention @PayPal
Mikhail Kourjanski focuses on the architectural approach towards PayPal’s real-time service platform that leverages ML models, delivers performance and quality of decisions.
/presentations/paypal-ml-fraud-prevention-2018/en/mediumimage/Mikbig-1534991790754.JPG)
-
Simplifying ML Workflows with Apache Beam
Tyler Akidau discusses how Apache Beam is simplifying pre- and post-processing for ML pipelines.
/presentations/ml-workflow-apache-beam/en/mediumimage/Tybig-1529542935186.JPG)
-
Orchestrating Data Microservices with Spring Cloud Data Flow
Mark Pollack discusses how to create data integration and real-time data processing pipelines using Spring Cloud Data Flow and deploy them to multiple platforms – Cloud Foundry, Kubernetes, and YARN.
/presentations/microservices-spring-cloud-data-flow/en/mediumimage/Marbig-1528670572828.JPG)
-
Data Pipelines for Real-Time Fraud Prevention at Scale
Mikhail Kourjanski discusses the architecture of PayPal’s data service which combines a Big Data approach with providing data in real time for decision making in fraud detection.
/presentations/paypal-data-service-fraud/en/mediumimage/Mikbig-1527121529188.JPG)
-
Developing Data and ML Pipelines at Stitch Fix
Jeff Magnusson discusses thoughts and guidelines on how Stitch Fix develops, schedules, and maintains their data and ML pipelines.
/presentations/data-ml-pipelines-stitchfix/en/mediumimage/Jefbig-1526475400409.JPG)