Data management overhaul driven by open formats and AI – SiliconANGLE

Data management is undergoing a major overhaul as companies face cost pressures and the need for consolidation.

The shift toward platformization is simplifying data operations, cutting expenses and boosting efficiency. This change is driven by the adoption of open data formats, artificial intelligence integration and enhanced governance solutions. As businesses focus on more sustainable and flexible data strategies, the future of data platforms promises greater innovation and operational effectiveness.

“One hundred percent of the data in Databricks is in open-source formats. And 92% is in Delta Lake,” said Ali Ghodsi (pictured), co-founder and chief executive officer of Databricks Inc. “I think this is just going to accelerate. I don’t see any way for it to go backwards.”

Ghodsi spoke with theCUBE Research’s John Furrier and Dave Vellante at the Supercloud 7: Get Ready for the Next Data Platform event, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed how companies are overhauling data management strategies by consolidating tools, adopting open data formats, integrating AI and governance solutions, and focusing on sustainable, flexible data solutions to reduce costs and improve efficiency.

Open formats, AI and governance: The pillars of modern data management

Open formats and interoperability between different data engines are crucial for future-proofing data platforms, according to Ghodsi. The move toward open-data formats, such as Delta Lake and Apache Iceberg, ensures that companies are not locked into proprietary systems, providing the freedom to switch between tools as needed.

“I think what has happened is that CIOs and IT departments, but also business leaders in line of business, they just feel like there’s cost pressures. So, they have to lower their cost basis,” he said. “That’s like trend number one that’s happening everywhere.”

One of the key developments at Databricks is the Unity Catalog, a governance solution that unifies data, AI models, dashboards and notebooks. This comprehensive approach to governance addresses not only security and access control but also cost management, sharing and collaboration. The integration of AI into data governance is particularly noteworthy, as it ensures that AI models are developed and deployed with strong oversight and accountability.

“Unity Catalog is a catalog for all of those data assets, not just for the tables. That’s what makes it unique in the market,” Ghodsi said. “We’ve seen further momentum now that people are adopting it because it’s open source. They can deploy it on-premise. They can deploy it in other environments. That’s what I would say is the second big important decision and factor that I see with enterprises. I mentioned AI, that’s the other big one.”

The path forward: Open source and strategic investments

Looking ahead, Ghodsi expressed optimism about the future of open source in the data platform industry. Databricks’ recent acquisition of Tabular, a company specializing in open-source data formats, underscores its commitment to open standards and interoperability. Databricks is working on Delta UniForm, which aims to unify Delta Lake and Apache Iceberg, providing seamless compatibility between the two formats, Ghodsi revealed.

“[Customers are] super excited to see interoperability between these formats,” he said. “They don’t want to see incompatibility between Delta Lake and Iceberg. The fact we’re unifying them in project UniForm, and now you can get both of those and you get full compatibility of your data, they loved it. I was a little bit surprised actually that there was so uniformly positive from the customer base and how positive it was.”

AI and gen AI also hold a strategic importance for Databricks, according to Ghodsi. The company’s focus on data intelligence — enabling organizations to extract actionable insights from proprietary data — positions it at the forefront of the AI revolution. The value of AI lies in its ability to provide accurate and secure answers based on a company’s unique data sets, he added.

“When gen AI happened, I would say two years ago, there’s huge demand for, ‘I want all my data, unstructured data, I care a lot about my AI models, I care a lot about governance of AI, I’m super worried about privacy, security of my AI and my data.’” Ghodsi said.  “AI has now been added to this equation, and Unity Catalog is a catalog for all of those data assets, not just for the tables, and that’s what makes it unique in the market.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE Research’s coverage of the Supercloud 7: Get Ready for the Next Data Platform event:

Photo: SiliconANGLE