Big Data Storage Solutions
AWS offers several storage services suited to big data workloads. The choice is usually driven by access pattern — sequential bulk reads, random point lookups, analytical scans, or archival — rather than by volume alone.
1. Amazon S3
Section titled “1. Amazon S3”- Object storage with effectively unlimited capacity
- The default foundation for a data lake
- Multiple storage classes so that access frequency can be traded against cost
- Durable and highly available, with lifecycle rules to move objects between classes automatically
Use for: data lakes, analytics source data, backup, archive, static content distribution.
2. Amazon S3 Tables
Section titled “2. Amazon S3 Tables”- S3 storage purpose-built for tabular data, held in a distinct bucket type called a table bucket
- Tables are stored in Apache Iceberg format and queried with standard SQL from engines that support Iceberg, including Athena, Redshift and Spark
- S3 performs compaction, snapshot management and unreferenced-file cleanup automatically, which is otherwise manual work on self-managed Iceberg tables in general-purpose buckets
Use for: analytics tables that are appended to continuously — transactions, sensor readings, impressions.
3. Amazon EBS
Section titled “3. Amazon EBS”- Block volumes attached to a single EC2 instance
- Low latency, with volume types tuned for throughput or IOPS
Use for: databases running on EC2, HDFS on EMR core nodes, any workload needing a filesystem on one host.
4. Amazon EFS
Section titled “4. Amazon EFS”- Managed NFS file storage that many EC2 instances can mount concurrently
- Grows and shrinks automatically as files are added and removed
Use for: shared working directories across a fleet, content management, workloads that need POSIX file semantics from several machines at once.
5. Amazon Redshift
Section titled “5. Amazon Redshift”- Managed data warehouse with columnar storage and massively parallel query execution
- Available as provisioned clusters or as Redshift Serverless, which has no cluster or node concept and bills per second of compute
- Integrates with most BI tools
Use for: business intelligence, repeated complex analytical queries, enterprise data warehousing. See Athena vs Redshift.
6. Amazon DynamoDB
Section titled “6. Amazon DynamoDB”- Managed NoSQL key-value and document database
- Scales horizontally without capacity planning; single-digit millisecond latency for point reads
Use for: high-volume application state, session stores, IoT telemetry sinks, anything with a known access key.
7. Amazon EMR and HDFS
Section titled “7. Amazon EMR and HDFS”- HDFS on EMR core nodes gives cluster-local storage for intermediate data during a job
- It is not durable beyond the life of the cluster
Use for: shuffle and intermediate data only. Final results belong in S3.
8. Amazon S3 Glacier storage classes
Section titled “8. Amazon S3 Glacier storage classes”“Amazon Glacier” was once a standalone service. It is now a set of S3 storage classes, so archival data lives in ordinary S3 buckets and is addressed with ordinary S3 APIs:
- S3 Glacier Instant Retrieval — millisecond retrieval, for archives that are rarely accessed but needed immediately when they are.
- S3 Glacier Flexible Retrieval — expedited retrievals typically complete in 1–5 minutes, standard retrievals in 3–5 hours, and free bulk retrievals in 5–12 hours.
- S3 Glacier Deep Archive — the lowest-cost class, with retrieval options in the 12–48 hour range, for data accessed less than once a year.
Use for: long-term backups, compliance retention, archives of raw data that must be kept but will rarely be read.
9. Amazon Neptune
Section titled “9. Amazon Neptune”- Managed graph database supporting both property graph (Gremlin, openCypher) and RDF/SPARQL
- Read replicas for high availability
Use for: relationship-heavy queries — recommendation graphs, fraud rings, knowledge graphs, network topology.
Options that are no longer available
Section titled “Options that are no longer available”Older architecture material recommends two services here that a new reader cannot adopt.
Amazon Timestream for LiveAnalytics was closed to new customers on 20 June 2025. Existing workloads continue to run. AWS points new time-series workloads at Amazon Timestream for InfluxDB, which offers similar functionality with single-digit millisecond query response.
Amazon QLDB (Quantum Ledger Database), the immutable cryptographically verifiable ledger, reached end of support and its documentation and product pages have been withdrawn. AWS directs ledger workloads to Amazon Aurora PostgreSQL, where the verifiable-history properties are implemented in the schema and application rather than by the database engine. Do not design new financial-audit or supply-chain-provenance systems around QLDB.