How Databricks Turns a Messy Bucket of Parquet Files Into a Reliable Database

If you’ve ever built a data lake on AWS S3, Azure Data Lake, or Google Cloud Storage, you know the dirty secret: under the hood, it’s just a massive, dumb pile of Parquet or CSV files dumped into a bucket.

For a long time, this architecture meant you had zero safeguards. If a Spark job crashed halfway through writing a 500GB dataset, you were left with partial, corrupted data that downstream analysts might accidentally query. If two pipelines tried to update the same table simultaneously, they quietly overwrote each other.

Building a data lake was cheap and scalable, but it lacked the guarantees of a traditional relational database.

Delta Lake, created by Databricks and now fully open-source, solves this entirely. The secret to turning that messy bucket of files into a highly reliable, ACID-compliant database isn't a complex distributed locking server.

It’s a single, hidden subdirectory called _delta_log.

The Git of Data

Think of the _delta_log as a Git commit history for your data. Every modification to a Delta table—whether it's an insert, update, delete, or schema change—is recorded as an ordered, atomic commit.

When a compute engine (like Spark, Trino, or DuckDB) queries a Delta table, it doesn't blindly scan your S3 bucket to see what Parquet files exist. Instead, it reads the transaction log first. The log acts as the ultimate source of truth, dictating exactly which Parquet files make up the valid, current state of the table.

Inside this hidden folder, you’ll find two main types of files:

• JSON Commit Files: Every transaction appends a sequentially numbered JSON file, starting from 00000000000000000000.json. These files record exactly what changed during that specific transaction.
• Checkpoint Files: Reading thousands of JSON files to figure out a table's current state would be brutally slow. To fix this, Delta Lake automatically aggregates the JSON log entries into a single Parquet checkpoint file (e.g., 00000000000000000010.checkpoint.parquet) every 10 commits. This acts as a consolidated "save state."

The 5 Core Actions

Parquet files are inherently immutable—you cannot edit a row inside an existing Parquet file. Therefore, Delta Lake handles transactions through a simple set of metadata actions recorded inside the JSON logs:

1. add: Tells the reader, "Here is a newly written Parquet file. Include it in the table."
2. remove: Tells the reader, "Mark this older Parquet file as deleted." The file stays in your storage bucket for historical queries, but active readers will ignore it. (This is called tombstoning).
3. metaData: Defines the schema, partitioning columns, and table properties.
4. protocol: Ensures the reader or writer library is compatible with the table's version, preventing an outdated engine from accidentally corrupting a newer format.
5. commitInfo: Captures the audit trail—who made the change, when, and what operation was run.

See It In Action (Interactive Visualizer)

To truly understand how this works, we need to see it in motion. Below is an interactive visualization of the Delta Lake transaction log. Step through the animation to see how Delta handles initial writes, concurrent conflicts, and file tombstoning.

Surviving the Traffic Jam: Optimistic Concurrency Control

What happens when a massive traffic jam of queries hits the table at once? Delta Lake uses Optimistic Concurrency Control (OCC) to handle conflicts safely without locking down the whole table.

When a pipeline wants to write data, it performs a three-step dance:

1. Read: The writing job checks the latest log version (let's say, version 5) to see the current state of the table.
2. Execute: It processes the data and writes new Parquet files into the bucket. Crucially, even if the job crashes here, no one sees the files because they aren't registered in the log yet.
3. Commit: It tries to write 00000000000000000006.json.

• If no one else has claimed version 6, it succeeds. The atomic commit is complete.
• If another job sneaked in and wrote version 6 first, the transaction is rejected. Delta Lake will then silently check if the intervening changes conflict with its own data. If they don't (e.g., they updated entirely different partitions), it automatically retries and commits as version 7.

This mutual exclusion guarantees that no partial, corrupted states ever leak to your end users.

Time Travel and Taking Out the Trash

Because every remove and add action is meticulously preserved in the log, querying historical data requires zero extra infrastructure.

When you run a time travel query like SELECT * FROM table VERSION AS OF 42, the engine simply reads the log up to version 42, applies those rules, and ignores everything that happened in versions 43 and beyond. You can instantly see the data exactly as it existed last Tuesday.

Eventually, keeping every historical file gets expensive. That’s where the VACUUM command comes in. Running VACUUM scans the log for any remove actions older than your retention threshold (usually 7 days) and physically deletes those underlying Parquet files from your cloud storage, keeping your AWS or Azure bill in check.

The Takeaway

By completely abstracting away the physical storage layer, the _delta_log design allows cheap, infinitely scalable cloud storage to achieve the transactional reliability of a traditional data warehouse. It's an incredibly elegant solution hiding in plain sight—and it's the reason the modern data lakehouse actually works.