AWS Data Pipeline
AWS Data Pipeline is closed to new customers. AWS states that it “is no longer available to new customers” and that the service “is in maintenance mode and no new features or region expansions are planned”. Accounts that already use it can continue to, but nothing new can be built on it, so it is not a choice for any design starting today. It is documented here because existing pipelines still run and older runbooks still refer to it.
AWS launched the service in 2012 to move data between sources on a schedule using a variety of compute options. The services that replaced it are listed at the end of this page.
What it does
Section titled “What it does”Data Pipeline runs data-driven workflows in which a task can depend on the successful completion of an earlier one. A pipeline definition, written in JSON, describes the whole workflow; the service schedules it, provisions the compute, runs the work, retries failures and can notify through SNS.
The definition is built from a handful of object types:
- Activities — the work itself: copying data, running an EMR job, running a SQL statement or executing a custom script.
- Data nodes — where input comes from and where output goes: S3 locations, RDS or Redshift databases, DynamoDB tables.
- Preconditions — checks that must pass before an activity starts, such as a file existing.
- Schedules — when and how often activities run, one-off or recurring.
- Resources — the compute that performs an activity. The service creates EC2 instances or EMR clusters for the duration, or uses machines you already have.
- Task runners — the agent that polls for work and performs it. Data Pipeline installs and runs one automatically on the resources it creates; a task runner can also be installed on an on-premises server so that the pipeline reaches machines outside AWS. A custom runner can be written if the supplied one does not fit.
A task is the combination of an activity, a schedule and a resource. Each run creates task instances, which are the actual executions.
Typical workloads were archiving web server logs to S3 and running a weekly EMR cluster over them, importing and exporting DynamoDB tables, copying files between S3 buckets, exporting RDS data to S3, and loading Redshift for analysis.
What replaces it
Section titled “What replaces it”AWS names three successors, and which one fits depends on the workload rather than on preference.
AWS Glue is the replacement for data integration work — serverless ETL jobs in Python or Spark, crawlers, a central data catalogue, and visual authoring. It is the right target when the pipeline is fundamentally about moving and transforming data between stores, and when it does not depend on a specific Hadoop ecosystem application or on orchestrating on-premises servers.
Step Functions is the replacement for orchestration — coordinating other AWS services, handling errors and retries, and controlling throttling. Like Data Pipeline it defines workflows in JSON, so definitions stay in source control. For activities that ran on on-premises servers or self-managed EC2 instances, the equivalent is to install the SSM Agent and invoke the work through Systems Manager Run Command from a state machine, scheduled by EventBridge.
Amazon MWAA (Managed Workflows for Apache Airflow) is the replacement when the team wants an open-source orchestrator with Python-defined DAGs, backfills, retries and a large operator library — and when portability away from a single vendor’s workflow service matters.
AWS documentation calls the service AWS Data Pipeline; the console and some AWS posts call it Amazon Data Pipeline. They are the same service. Neither should be confused with a “data pipeline” as a general architectural term, which is covered under ETL, ELT and where a data lake sits.