Delta Lake Common Format (UniForm) for Iceberg compatibility, now in GA

Delta Lake has confirmed to be the hottest and quickest lakehouse format through the years. Delta Lake Common Format (UniForm), now obtainable in GA, builds on Delta Lake’s wealthy connector ecosystem to mix Delta Lake’s superior price-performance with entry to each software in your stack. With Delta Lake UniForm, you possibly can write a single copy of your knowledge and make it obtainable to any engine that helps any of the first open desk codecs: Linux Basis Delta Lake, Apache Iceberg, and Apache Hudi (coming quickly). On this weblog, we cowl the next:

 

• Constructing the open knowledge Lakehouse with Delta Lake UniForm
• Getting quick efficiency in any engine
• Utilizing superior Delta Lake options, like Liquid Clustering, with Delta Lake UniForm

Constructing the Open Lakehouse

Delta Lake provides a vibrant connector ecosystem with help from many common open supply frameworks and business engines. UniForm expands Delta Lake’s ecosystem by making the most of the inherent similarities among the many 3 open desk codecs. Delta Lake, Iceberg, and Hudi all retailer knowledge within the Apache Parquet file format however diverge in how they retailer further metadata. Delta Lake UniForm generates Iceberg metadata alongside Delta Lake whereas sustaining a single copy of the Parquet recordsdata. By writing as soon as to Delta Lake UniForm, you possibly can entry your knowledge utilizing any engine that helps any one of many open codecs:

 

Delta Lake UniForm permits you to decide on the very best software to your workload. With Delta Lake UniForm, you get the info flexibility to help any structure you select at the moment or sooner or later. 

Quick efficiency, all over the place

With extra platforms embracing open desk codecs, you possibly can write Delta Lake UniForm to entry a broader vary of instruments with out costly knowledge duplication. This supplies larger flexibility and decrease prices for knowledge beforehand saved in a proprietary format. With Delta Lake UniForm, you possibly can make the most of Databricks’ best-in-class ingestion and ETL price-performance and join with any knowledge warehousing or BI software in your stack. These price financial savings might be realized with out compromising on question efficiency downstream.

 

The benchmarks beneath examine efficiency ingesting Parquet recordsdata into Delta Lake UniForm utilizing Databricks and into Iceberg utilizing Snowflake. 

Databricks ingested Parquet 6x quicker than Snowflake. Databricks was additionally 90% inexpensive than Snowflake. As a result of Delta Lake UniForm writes each Delta and Iceberg metadata, the desk stays accessible to Snowflake. In Snowflake, Delta Lake UniForm might be learn utilizing an Iceberg catalog integration. A catalog integration lets you create an Iceberg desk in Snowflake referencing an exterior Iceberg catalog or object storage. Benchmarks present that out-of-box learn efficiency for Delta Lake UniForm is similar to Snowflake managed Iceberg:

 

 

The distinction in question efficiency is almost zero! With Delta Lake UniForm, you get the quickest efficiency and common connectivity all from a single copy of information in your personal storage bucket! 

 

With Delta Lake UniForm you get the very best of all codecs

When writing Delta Lake UniForm, you possibly can proceed to make the most of Delta Lake’s superior desk options. For instance, Delta Lake UniForm can now be enabled on Delta tables utilizing Liquid Clustering, a brand new function obtainable in Public Preview. Liquid Clustering is an clever knowledge administration method that dynamically clusters Delta tables, permitting knowledge structure to evolve alongside analytics wants. 

 

Collectively, Delta Lake UniForm and Liquid Clustering present quick question efficiency even when studying from Iceberg or Hudi engines. This works as a result of when Liquid Clustering optimizes the bodily knowledge structure, Delta Lake UniForm displays these enhancements in each Delta Lake and Iceberg metadata. As a result of Delta Lake UniForm is barely writing further metadata, there may be negligible overhead on writes. Liquid additionally routinely clusters new knowledge throughout ingestion, so question efficiency stays quick over time. 

How clients are utilizing Delta Lake UniForm

Throughout Public Preview, organizations proved Delta Lake UniForm’s compatibility with common Iceberg reader purchasers together with Snowflake, BigQuery, Redshift, and Athena for a spread of BI and analytics use circumstances.

 

 

Now in GA, Delta Lake UniForm is prepared to your manufacturing workloads. At Databricks, our clients have already began to see the advantages of writing UniForm:

At M Science, UniForm supplies us with the pliability to jot down a single copy of our knowledge that may be queried by any engine that helps Delta or Iceberg – that is key to decreasing prices and accelerating time-to-value

— Ben Tallman, Chief Expertise Officer at M Science

We’re excited to see clients and business distributors select the open Lakehouse structure for its simplicity, flexibility, and decrease prices. Put up GA, we’ll proceed to put money into making Delta Lake UniForm extra interoperable and seamless in order that customers can use any software of their ecosystem. 

New Delta Lake UniForm options can be found as a part of the Delta Lake 3.2 launch. Databricks clients can use these options by upgrading to Databricks Runtime model 14.3. 

 

You’ll be able to be taught extra about how one can learn Delta Lake UniForm out of your selection Iceberg reader within the hyperlinks beneath: