Delta Lake 101: How Databricks Made S3 Behave Like a Warehouse

Data lakes promised cheap, flexible storage — but they paid for it in chaos. No transactions, no reliable schema, no performance guarantees. Databricks' Delta Lake is the protocol that fixed that without throwing out the lake. Here's a quick walkthrough.

The Core Idea: Don't Replace the Lake, Make It Smarter

The pitch from Databricks was simple: the data lake got one thing right — cheap object storage like S3. Data being in S3 was never the problem. The problem was that the lake was just files and folders. No reliability, no rules, no good performance — just a dump.

So instead of migrating everything into a warehouse, what if you kept the data in S3 and added a layer on top that gave you the reliability and performance of a warehouse? That layer is the Delta Lake.

Delta Lake Is a Protocol, Not a Database

This is the key misconception Delta Lake clears up. It is not a separate system. It is not a database you migrate to. It is a protocol — a specific way of organizing files in S3 so they behave like a reliable database instead of a "file swamp."

That distinction matters. Delta Lake doesn't ask you to give up S3 economics. It ships the warehouse semantics (ACID transactions, time travel, schema enforcement) down to the storage layer itself.

What a Delta Table Actually Looks Like

When you write to a Delta table, two things get created in S3:

Component	What It Is	Why It Matters
Parquet files	Columnar, compressed data files	Efficient scans and compression
Delta log	Transaction log folder (_delta_log/)	Enables ACID, time travel, schema evolution

The Parquet + transaction log combination is what makes it a Delta table. The log is a sequence of JSON commit files that records every change — inserts, updates, deletes, schema changes. Readers replay the log to reconstruct a consistent view of the table at any point in time. That's how you get atomic writes, concurrent readers, and snapshot isolation on top of plain object storage.

Why This Pattern Won

Delta Lake became the blueprint for what the industry now calls the lakehouse: lake-class storage economics with warehouse-class reliability. The same idea — an open table format layered over object storage — is now the core of three competing formats:

• Delta Lake (Databricks)
• Apache Iceberg (originated at Netflix)
• Apache Hudi (originated at Uber)

All three solve the same core problem: file-level chaos on object storage. All three use the pattern of data files plus a metadata/transaction layer. The winner is still being decided, but the architectural bet — keep the data in S3, add a smarter layer on top — has clearly paid off.

Key Takeaways

• The problem with data lakes was never where the data lived; it was the lack of reliability and performance guarantees on top of the files
• Delta Lake is a protocol/format, not a separate system — a Delta table is just Parquet files plus a transaction log in S3
• The transaction log (_delta_log/) is what gives you ACID, time travel, and schema enforcement on object storage
• The "open table format on object storage" pattern (Delta, Iceberg, Hudi) is now the dominant architecture for analytical data

Resources

• Delta Lake Documentation — Official docs for the Delta Lake protocol
• Delta Lake GitHub — Open-source reference implementation
• Databricks Delta Lake Page — Databricks' managed Delta offering