AWS DataSync

AWS DataSync moves file and object data to, from and between AWS storage services. It is the service AWS now names first for bulk data movement, including as the replacement for offline transfer on Snow devices.

Key features

• Agent-based for on-premises sources. A DataSync agent runs as a virtual machine next to the source storage and speaks to the service over a public endpoint or a VPC endpoint. Cloud- to-cloud and AWS-to-AWS transfers need no agent.
• One-off migrations and scheduled recurring transfers. A task can run once or on a schedule, and each run copies only what changed, so DataSync serves both an initial migration and ongoing replication.
• Verification built in. Transfers are encrypted in flight and integrity-checked on arrival, and file permissions and metadata are preserved across supported pairings.
• Purpose-built transfer protocol. A parallel, multi-threaded protocol with in-line compression and sparse-file detection; a single task can saturate a 10 Gbps link.

Supported sources

On-premises and self-managed

• Network File System (NFS)
• Server Message Block (SMB)
• Hadoop Distributed File System (HDFS)
• Self-managed object storage that speaks the Amazon S3 API

Other clouds

Google Cloud Storage, Azure Blob Storage, Azure Files, and a range of S3-compatible providers including Wasabi, DigitalOcean Spaces, Oracle Cloud Infrastructure Object Storage, Cloudflare R2, Backblaze B2 and IBM Cloud Object Storage.

Supported AWS destinations

• Amazon S3, including the archive storage classes
• Amazon EFS
• Amazon FSx for Windows File Server, FSx for Lustre, FSx for OpenZFS and FSx for NetApp ONTAP

Use cases

• Migrate an active dataset over the network into AWS storage.
• Archive cold data straight into S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive, freeing on-premises capacity and letting a legacy filer be switched off.
• Replicate into a standby file system or a second storage class.
• Move data in and out for processing, where a hybrid workflow computes in the cloud on data that originates on-premises.

flowchart LR subgraph Source["Source storage"] NFS[NFS share] SMB[SMB share] HDFS[HDFS cluster] OBJ[S3-compatible object store] end Agent[DataSync agent] subgraph AWS["AWS Cloud"] S3[(Amazon S3)] EFS[(Amazon EFS)] FSx[(Amazon FSx)] end NFS --> Agent SMB --> Agent HDFS --> Agent OBJ --> Agent Agent -->|Encrypted, verified transfer| S3 Agent --> EFS Agent --> FSx style Agent fill:#FF9900,stroke:#FF9900,color:white style S3 fill:#3F8624,stroke:#3F8624,color:white style EFS fill:#3F8624,stroke:#3F8624,color:white style FSx fill:#3F8624,stroke:#3F8624,color:white

Networking

DataSync does not require a VPN. It connects over a public service endpoint or a VPC endpoint, and where bandwidth or reliability is the constraint the usual answer is a Direct Connect hosted connection taken for the duration of the project rather than a permanent circuit.

What it is not

DataSync is a transfer service, not a storage layer. It does not present a mount point and it does not cache — applications that need to keep reading and writing cloud data as a local file share want S3 File Gateway instead. The side-by-side comparison sets out when to use which, and why the two are often used together.