Data lake vs AWS Lake Formation
A data lake is a concept — an architecture. AWS Lake Formation is a service that manages and secures one. Confusing the two leads to plans that budget for a service when the work is architectural, or the reverse.
The data lake
Section titled “The data lake”- Raw data storage, on AWS almost always in Amazon S3
- Holds structured and unstructured data side by side
- Accepts a variety of file formats
- Exists to make large-scale analytics possible over data that was never modelled for it
- Requires security, governance and cataloguing to be useful rather than a dumping ground
That last point is the one that gets skipped. A bucket full of files is only a data lake once somebody can find out what is in it and who is allowed to read it.
AWS Lake Formation
Section titled “AWS Lake Formation”- A management layer over the storage
- Central security and access control
- Manages data ingestion
- Manages permissions, including fine-grained controls down to table, column and row
- Integrates with the AWS Glue Data Catalog
The distinction
Section titled “The distinction”The lake is a library of books; Lake Formation is the library management system that controls who may read which book, keeps the catalogue, governs how new books are added, and records who borrowed what.
Lake Formation is what makes a data lake enterprise-ready, by adding management, security and governance on top of raw storage. It is not a prerequisite for storing data in S3 — it is a prerequisite for letting more than one team share that data safely.
For where a data lake sits relative to a warehouse and the transformation steps around it, see ETL, ELT and where a data lake sits.