Big Data

This section covers the AWS services used to land, move, process and query data at a scale where a single machine is not an option, and the pipeline patterns that connect them.

Start with ELT and the data lake below for the shape of a typical pipeline, then read Amazon EMR for cluster-based processing, Athena and AWS Glue for the serverless equivalent, and Big data storage solutions for choosing where the data lives.

ELT: the modern pipeline shape

ELT postpones transformation until after the load, which inverts the older ETL order. Raw data is landed in a data lake staging zone first and transformed only when a specific question needs answering.

The trade is deliberate. ETL transforms before loading, which reduces storage and produces a tidy warehouse, but it discards whatever the transformation did not keep. ELT preserves the original data, defers transformation cost until there is a reason to pay it, and lets the same raw data serve use cases that were not anticipated when it was collected — including machine learning training sets, which usually want more of the original signal than a reporting schema retains.

This suits cloud economics, where object storage is cheap relative to compute. Transformations are then expressed as views, query-time projections or scheduled jobs, and their output either used directly or written back into the lake as a curated dataset.

An ELT pipeline: data is ingested and loaded into the lake before any transformation happens.

The diagram names AWS Data Pipeline as the orchestrator and Amazon QuickSight as the reporting tool. Both labels are out of date: Data Pipeline has been closed to new customers since July 2024 and is in maintenance mode, and QuickSight became Amazon Quick Suite in October 2025. Use AWS Step Functions, AWS Glue workflows, Amazon MWAA or EventBridge Pipes for orchestration.

The following diagram shows the same shape with current service names against each stage.

flowchart TD subgraph Sources [Data Sources] Analytics["Analytics Systems (Managed Service for Apache Flink)"] BI["Business Intelligence (Amazon Quick Suite)"] ERP["ERP Systems (Database Migration Service)"] CRM["CRM Systems (AppFlow, DMS)"] IOT["IoT (IoT Core)"] Social["Social Media (Kinesis, AppFlow)"] end subgraph DataLake [Data Lake Infrastructure] Raw["Raw Zone (S3 + Lake Formation)"] Process["Processing Layer (Glue, EMR, Lambda)"] Curated["Curated Zone (S3 + Athena/Redshift)"] Catalog["Data Catalog (Glue Data Catalog)"] end subgraph Consumers [Data Consumers] ML["Machine Learning (SageMaker AI)"] DataScience["Data Science (SageMaker Unified Studio)"] Reporting["Enterprise Reporting (Amazon Quick Suite)"] Apps["Applications (ECS, Lambda)"] end Analytics --> Raw BI --> Raw ERP --> Raw CRM --> Raw IOT --> Raw Social --> Raw Raw --> Process Process --> Curated Process --> Catalog Curated --> Catalog Curated --> ML Curated --> DataScience Curated --> Reporting Curated --> Apps classDef source fill:#97c2fc,stroke:#2d6aa5,stroke-width:2px classDef lake fill:#f8cecc,stroke:#b85450,stroke-width:2px classDef consumer fill:#d5e8d4,stroke:#82b366,stroke-width:2px class Analytics,BI,ERP,CRM,IOT,Social source class Raw,Process,Curated,Catalog lake class ML,DataScience,Reporting,Apps consumer

The service map

Storage

• Amazon S3 for object storage and as the foundation of a data lake
• Amazon S3 Tables for tabular data stored as Apache Iceberg tables, with automatic compaction and snapshot maintenance
• Amazon EFS and Amazon FSx for shared file systems
• Amazon EBS for block storage attached to EC2 instances

Processing and analytics

• Amazon EMR for Spark, Hive, Flink, HBase, Trino and the rest of the Hadoop ecosystem
• AWS Glue for serverless Spark-based ETL, crawling and cataloguing
• Amazon Athena for SQL over data in S3 and, through federated query, over external sources
• Amazon Redshift for data warehousing, provisioned or serverless

Ingestion and movement

• Amazon Kinesis Data Streams for real-time streaming ingest
• Amazon Data Firehose (formerly Kinesis Data Firehose) for delivery of streaming data into S3, Redshift, OpenSearch and elsewhere
• Amazon MSK for managed Apache Kafka
• AWS Database Migration Service for database replication and migration
• AWS Transfer Family for SFTP, FTPS and AS2 transfers

Orchestration

• AWS Step Functions for state-machine workflows
• AWS Glue workflows for Glue-native crawl-then-job sequences
• Amazon Managed Workflows for Apache Airflow (MWAA)
• Amazon EventBridge Pipes for point-to-point event routing

AWS Data Pipeline, which older material recommends here, has been closed to new customers since 25 July 2024 and receives no new features.

Databases

• Amazon RDS and Amazon Aurora for relational workloads
• Amazon DynamoDB for key-value and document workloads
• Amazon DocumentDB for MongoDB-compatible workloads
• Amazon Neptune for graph workloads
• Amazon Timestream for InfluxDB for time series; Timestream for LiveAnalytics has been closed to new customers since 20 June 2025

Query and search

• Amazon OpenSearch Service (formerly Amazon Elasticsearch Service)
• Amazon Redshift Spectrum for querying S3 data from RA3 and DC2 provisioned clusters

Visualisation

• Amazon Quick Suite, the product formerly named Amazon QuickSight

Security and governance

• AWS Lake Formation for data lake permissions and setup
• Amazon Macie for sensitive data discovery
• AWS KMS for key management