The Polaris Catalog is an open source catalog for Apache Iceberg that was recently announced by Snowflake at its user conference, Snowflake Summit, earlier this week.
While not technically fully available yet, the company says the catalog will be open sourced within the next 90 days.
Apache Iceberg is a format for big data analytics, and the Polaris Catalog will extend it by providing customers more flexibility and choice over their data architectures. It will have full interoperability with providers like AWS, Confluent, Dremio, Google Cloud, Microsoft Azure, Salesforce, and more, Snowflake said.
“AWS is committed to working with partners, such as Snowflake, on open source solutions that can accelerate choice for customers,” said Chris Grusz, Managing Director, Technology Partnerships, Amazon Web Services. “We’re pleased to work with Snowflake to continue to make Apache Iceberg stay interoperable across our engines.”
This catalog relies on Apache Iceberg’s REST protocol, which is an open standard for accessing and retrieving data from different sources so long as they support the protocol. Examples of such include Apache Flink, Apache Spark, Dremio, Python, and Trino.
Apache Iceberg was developed at Netflix, donated to the Apache Software Foundation in 2018, and became a top-level project in 2020. It now has over 5K stars on GitHub and nearly 500 developers contribute to the project.
“Organizations want open storage and interoperable query engines without lock-in. Now, with the support of industry leaders, we are further simplifying how any organization can easily access their data across diverse systems with increased flexibility and control,” said Christian Kleinerman, EVP of product at Snowflake. “Polaris Catalog extends Snowflake’s commitment to Apache Iceberg as the open standard of choice, and signals the intent from industry leaders in enabling customers and the wider Iceberg community to harness their data through an open and neutral approach, empowering cross-engine interoperability on that data.”
Read about other Open-Source Projects of the Week
