AWS announced Amazon AppFlow a "no-code" or "low-code" service to integrate AWS platform services with data from other frequently-used software as a service (SaaS)-based applications such as Salesforce, Zendesk, ServiceNow etc.
It's a reality that enterprise data is often strewn across multiple "SaaS islands", and this decentralized data needs to be gathered in a centralized location for comprehensive reporting and other reasons. Manually extracting and merging information. or writing custom connectors. is often expensive and cumbersome to maintain.
Amazon AppFlow is a bi-directional integration between the AWS platform and a variety of SaaS platforms. It's intended for SaaS application admins, business analysts, BI specialists, and other similar roles to be able to implement these integrations with a few clicks, and also to be capable of scheduling these flows with minimum use of code. In addition, data can be encrypted in motion by leveraging AWS PrivateLink.
It's worth noting that Amazon AppFlow is in the same league as tools referred to as integration platform as a service (iPaaS), offering a linear data processing pipeline without orchestration and workflow that is a feature of many of the other services.
InfoQ caught up with Kurt Kufield, VP AWS Platform, Amazon Web Services (AWS), regarding details about the announcement.
Kurt Kufield talked about how the integration service is expected to alleviate the pain behind gathering data manually or writing custom connectors that are usually expensive and error prone. He highlighted the audience for the service: primarily low-code and no-code developers. He also touched upon some of the technical and data requirements for the integration of the different SaaS platforms with AWS.
InfoQ: What is Amazon AppFlow, and what pain points does this solve?
Kurt Kufield: Amazon AppFlow is a fully managed integration service that enables customers to securely transfer data between Software-as-a-Service (SaaS) applications like Salesforce, Marketo, Slack, and ServiceNow, and AWS services like Amazon S3 and Amazon Redshift, in just a few clicks.
Customers often also have data stored in dozens of SaaS applications, resulting in silos that are disconnected. Organizations want to be able to combine their data from these sources, but that requires customers to spend days writing code to build custom connectors and data transformations to convert disparate data types and formats across different SaaS applications.
Customers with multiple SaaS applications end up with a sprawl of connectors and complex code that is time-consuming and expensive to maintain. Further, custom connectors are often difficult to scale for large volumes of data or near real-time transfer, causing delays between when data is available in SaaS and when other systems access the data.
In large enterprises, business users wait months for skilled developers to build custom connectors. In firms with limited in-house developer skills, users resort to manually uploading and downloading data between systems, which is tedious, error-prone and risks data leakage. Amazon AppFlow addresses these pain points specifically.
InfoQ: Is this service aimed at “no code” or “low code” developers? Should traditional developers/architects be interested, also?
Kufield: Amazon AppFlow has a no-code interface that makes it super easy for users in lines of business to build their own integrations. Traditional developers can also benefit from the automation and ease of use - this will allow them to improve productivity by leveraging out of the box integration capabilities offered by AppFlow, allowing them to focus on building applications. Likewise, architects can leverage AppFlow to easily connect different applications and services and avoid the undifferentiated heavy lifting needed to build custom connectors.
InfoQ: What are the endpoint and data requirements (if any) to be able to integrate with other SaaS apps.? i.e. does it need to have REST endpoints and data to be available in JSON format, for example?
Kufield: Our connectors are quite flexible. The majority of connectors we support today interface with REST API endpoints of the applications and services we support. However, we connect to data warehouses (Amazon Redshift and Snowflake) over JDBC connectors. We can handle both JSON and CSV formats for data and are constantly expanding these capabilities.
InfoQ: In a related question, do you make any recommendations/best practices for the integration, since this may be subject to performance/threshold concerns?
Kufield: While AppFlow easily scales to support data flows at any scale, the applications we support as sources and destinations often impose limits on the maximum volume that can be transferred and/or can have volume or rate limits on API calls. We recommend our customers be aware of data transfer limits imposed by the applications that they integrate using AppFlow since AppFlow data transfers will count against these limits. Generally, you must also ensure that all data transfers comply with access policies set by your organization.
InfoQ: There are a number of integrations right off the box. Can you mention other planned integrations and a roadmap for Amazon AppFlow?
Kufield: We’re unable to discuss our upcoming roadmap plans at this time.
In summary, Kufield talked about the pain points that Amazon AppFlow is expected to solve, the intended audience for the integration service and some of the requirements for integration. He specifically did not address the roadmap or planned future integrations for the service -- it's unclear if 3rd party developers or integrators can develop integrations to other SaaS platforms as part of the AppFlow service.
Further details are in the docs and the associated AWS blog post walks through an example of creating data flow from Slack to an S3 bucket on a schedule, and also examples of how to leverage other AWS services to produce a word cloud of the Slack channel data.