ETL services on AWS
There is no single ETL service on AWS. A dozen services do part of the job, and the choice is usually settled by two questions: is the hard part the transformation or the orchestration, and does the work have to run somewhere AWS does not manage.
For most new data integration work the answer is AWS Glue, with Step Functions or Amazon MWAA above it when the pipeline has many steps that have to be coordinated.
| Service | Type | What it contributes | Best for |
|---|---|---|---|
| AWS Glue | Serverless data integration | Spark and Python ETL jobs, crawlers, a central data catalogue, visual authoring | Large-scale data integration and cataloguing; the default starting point |
| Amazon MWAA | Managed Apache Airflow | Python DAGs, backfills, retries, a large operator library, portable definitions | Complex multi-step workflows, and teams already using Airflow |
| AWS Step Functions | Serverless orchestration | Coordinates other services, handles errors, retries and throttling | Workflows spanning several AWS services, including Glue and EMR jobs |
| Amazon EMR | Managed big data platform | Spark, Hive and the wider Hadoop ecosystem on clusters you size | Very large processing jobs, and workloads tied to a specific Hadoop application |
| AWS Batch | Managed batch computing | Queues and runs containerised jobs on managed compute | Compute-intensive batch transformation |
| AWS Lambda | Serverless functions | Short, event-driven transformations | Small per-object or per-record work, triggered by an event |
| Amazon Redshift | Data warehouse | COPY for bulk loading, stored procedures for in-warehouse transformation | Transformation that belongs next to the warehouse data (ELT) |
| AWS Database Migration Service (AWS DMS) | Database migration and replication | One-off and continuous replication, including between different engines | Getting data out of operational databases and keeping it current |
| AWS Lake Formation | Data lake governance | Automates data lake setup, permissions and access control | Building and securing a lake that several teams share |
| Amazon AppFlow | SaaS integration | Managed connectors to SaaS applications, no code required | Pulling data from SaaS products into AWS on a schedule or a trigger |
| AWS Transfer Family | Managed file transfer | SFTP, FTPS, FTP and AS2 endpoints writing into S3 and EFS | Partner file feeds that must keep speaking a legacy protocol |
| Amazon EventBridge Pipes | Point-to-point integration | Connects a source to a target with optional filtering and enrichment | Event-driven movement between services without glue code |
| AWS Data Pipeline | Legacy workflow service | Closed to new customers and in maintenance mode | Nothing new — see below |
Glue and the service it replaced
Section titled “Glue and the service it replaced”AWS Data Pipeline is closed to new customers and receives no new features or Region expansions. Existing pipelines still run, but it cannot be chosen for new work, and AWS names three replacements depending on what the pipeline actually did:
- Glue for data integration — moving and transforming data between stores, with a catalogue.
- Step Functions for orchestration — coordinating services, with error handling and retries.
- MWAA for teams that want Airflow’s DAGs, operators and portability.
Glue’s advantages over the older service are that it is serverless, so there are no task runners or EC2 resources to define, and that the Data Catalog gives a single place where table definitions live, shared with Athena, EMR and Redshift Spectrum.
The one thing Data Pipeline did that none of the three does directly is run activities on machines outside AWS through an installed task runner. The equivalent today is to register those machines with Systems Manager and invoke the work with Run Command from a state machine or a DAG.
Where to start
Section titled “Where to start”- If data has to be extracted from an operational database, start with DMS.
- If the work is transforming and cataloguing data at rest, start with Glue.
- If the work is mostly sequencing other services, start with Step Functions, or MWAA when the sequencing is complex enough to want Airflow.
- If the transformation is small and event-driven, a Lambda function is often the whole pipeline.