Amazon EMR

Amazon EMR (originally Elastic MapReduce) is a managed cluster platform for running open-source big data frameworks on AWS. It is not a single product but a packaged, deployable collection of open-source projects — Apache Spark, Hive, HBase, Flink, Trino and others — with the cluster provisioning, configuration and scaling handled by AWS.

EMR runs on EC2 instances, on EKS, or serverless. Clusters can be transient, created for one job and terminated when it finishes, or long-running.

Cluster architecture

An EMR cluster on EC2 has three node types.

Primary node — coordinates the cluster: distributes work, tracks job status and runs the cluster management software. Every cluster has one, or three in a high-availability configuration. (AWS previously called this the master node; the documentation now uses “primary”.)

Core nodes — run tasks and hold HDFS storage. Because they carry data, removing core nodes risks data loss, so they scale more conservatively than task nodes.

Task nodes — run tasks only, with no HDFS. They are the natural place to use Spot capacity and to scale up and down aggressively, because losing one costs only the work in flight.

Persistent output should be written to S3 rather than HDFS. HDFS on core nodes is cluster-local storage and disappears when the cluster terminates; using S3 as the durable layer is what makes transient clusters practical. See Glue and EMR storage patterns.

Steps

Work is submitted to a cluster as steps — ordered units of work, each one a Hive script, a Spark application, a custom JAR, a Pig script or a shell command. A typical sequence might run a Hive query over raw data, then a custom JAR for domain-specific processing, then a Pig script for a final transformation, writing results to S3.

Steps run in order and each either succeeds or fails, so design them to be independent and re-runnable rather than relying on state left behind by the previous step.

Applications in the bundle

The EMR 7.x releases include, among others:

• Hadoop — HDFS for distributed storage and MapReduce for distributed batch processing. MapReduce gives EMR its original name, though most new work uses Spark instead.
• Spark — the general-purpose distributed processing engine used for most batch and interactive work on EMR today. Spark is also the engine underneath AWS Glue ETL jobs. Its core abstractions are the driver and cluster manager that plan work, executors on worker nodes that run tasks, and resilient distributed datasets (RDDs): immutable, fault-tolerant collections with lazily evaluated transformations and eager actions.
• Hive — a data warehousing layer over Hadoop that exposes HiveQL, a SQL-like language, so that tabular queries can be written without MapReduce code. Hive keeps the table-and-database model, including partitioning and bucketing.
• Flink — a stream and batch processing engine. Flink’s watermarks handle out-of-order events by tracking event time rather than arrival time, so a computation’s state can be resolved correctly even when records arrive late. Its runtime is a client, a job manager and task managers.
• HBase — a wide-column store for random read/write access over large tables.
• Trino and Presto — distributed SQL query engines. Amazon Athena is built on this lineage.
• Hudi, Iceberg and Delta — open table formats providing transactions, schema evolution and time travel over data in S3.
• Pig — a scripting layer for data manipulation.
• Oozie — workflow scheduling.
• ZooKeeper — distributed coordination.
• Hue, Zeppelin, JupyterHub and Livy — notebook and submission interfaces.

Older material lists Mahout, Ambari, Flume and Ganglia as part of the EMR bundle. None of them ships in any EMR 7.x release. Sqoop shipped up to EMR 7.4.0 and was removed in 7.5.0.

Common use cases

• Log analysis at scale
• ETL over large datasets
• Financial and clickstream data processing
• Anomaly detection across large volumes
• Feature preparation for machine learning

Practical notes

• Use S3 as the system of record and treat clusters as replaceable.
• Match node type to workload: core nodes for capacity that must hold data, task nodes (often on Spot) for burst compute.
• Prefer columnar formats (Parquet, ORC) and partitioning for anything that will be queried repeatedly.
• Choose the application set at cluster creation — adding frameworks later means a new cluster.