Glue and EMR Storage Patterns
Output from Glue and EMR jobs, and the intermediate data they produce along the way, generally belongs in S3. The diagram below shows the common storage patterns.
flowchart LR
subgraph "Data Sources"
Raw[Raw Data]
Inter[Intermediate Data]
Results[Process Results]
end
subgraph "Storage Locations"
S3Raw[S3 Raw Data Bucket]
S3Stage[S3 Staging Bucket]
S3Results[S3 Results Bucket]
EMRHDFS[EMR HDFS]
GlueTemp[Glue Temporary Storage]
end
subgraph "Best Practices"
Life[Lifecycle Policies]
Comp[Compression]
Part[Partitioning]
Format[File Format Selection]
end
Raw --> S3Raw
Inter --> S3Stage
Inter --> EMRHDFS
Inter --> GlueTemp
Results --> S3Results
S3Stage --> |Temporary| Life
S3Results --> |Management| Life
S3Results --> |Optimization| Comp
S3Results --> |Organization| Part
S3Results --> |Performance| Format
classDef source fill:#e1f3d8,stroke:#333,stroke-width:2px;
classDef storage fill:#dae8fc,stroke:#333,stroke-width:2px;
classDef practice fill:#fff2cc,stroke:#333,stroke-width:2px;
class Raw,Inter,Results source;
class S3Raw,S3Stage,S3Results,EMRHDFS,GlueTemp storage;
class Life,Comp,Part,Format practice;
Raw data
Section titled “Raw data”- Store in a dedicated raw bucket
- Keep the original format — do not transform on the way in
- Apply retention policies appropriate to the source and to any regulatory obligation
Intermediate and temporary data
Section titled “Intermediate and temporary data”- EMR: use HDFS on core nodes for shuffle and intermediate data during the job, then write final results to S3. HDFS disappears with the cluster.
- Glue: temporary storage in S3 is allocated automatically
- Use staging buckets with short lifecycle rules so intermediate data does not accumulate
Results
Section titled “Results”- Store in a dedicated results bucket, separate from raw and staging
- Write columnar formats — Parquet or ORC — for anything that will be queried
- Partition on the columns that queries filter by
- Compress
- Let the expected access pattern drive the storage class
Practical notes
Section titled “Practical notes”- Keep bucket organisation explicit: raw, staging, processed
- Lifecycle rules are the main control on storage cost; without them, staging buckets grow indefinitely
- Format and partitioning determine query cost as well as query speed, particularly with Athena, which bills by data scanned
- Apply access controls at each stage — raw data often carries sensitivity that the curated output does not