Data transformation copilot boosts productivity for Databricks users – SiliconANGLE

Data transformation is entering a new era with the advent of generative AI, significantly altering how users access and manipulate data for enhanced productivity. Prophecy Inc.’s latest innovation, the industry’s first copilot for Databricks, exemplifies this trend by streamlining the preparation of raw data for analytics and AI applications.

“What’s happening with data transformation is people are like, ‘I’ve got to do more, I was doing this table stuff, I was doing reporting, I was doing dashboards and now I’ve got to do generative AI,’” said Raj Bains (pictured), founder and chief executive officer of Prophecy. “We are pushed beyond our limit. How do I do more? Productivity is the name of the game in data transformation.”

Bains spoke with theCUBE Research’s John Furrier and Savannah Peterson at the Data + AI Summit, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed how Prophecy is extending data transformation throughout an organization. (* Disclosure below.)

Prophecy’s Data Transformation Copilot democratizes data access with AI tools.

Prophecy’s copilot approach is designed to address limited accessibility to cloud data platforms for users other than data engineers. The Data Transformation Copilot integrates visual interfaces, compiler technology and generative AI to improve efficiency for data use within an organization.

“We are providing the tooling layer that makes data engineers and data analysts and line of business users, all of them productive with data,” Bains said. “A data analyst using a visual drag and drop and some…generative AI, just says: ‘Build me a pipeline that does this,’ and we build it for them. That runs on Databricks native code at scale.”

Prophecy’s customer base includes a wide range of users in the banking, insurance, healthcare and technology industries. The company’s solution has been adopted more recently by the World Series champion Texas Rangers major league baseball team which is leveraging Databricks’ Data Intelligence Platform and Prophecy to optimize hundreds of millions of records and terabytes of data.

“You can’t move fast with data if people are depending on other people and using Word docs,” Bains said. “Everybody who wants data has got to have access to it. You want to transform it for your business use case today.”

Broader access to data and the ability for users to build their own pipelines is reshaping the role of platform teams, According to Bains. This shift reduces dependency on platform teams, allowing users to be more productive and self-sufficient.

“We had one large customer, one of the largest pharma customers…they put the copilot straight into the data users’ hands, they’re building their pipelines themselves,” Bains said. “All the dependency, the back and forth with the data platform team is gone. They’re productive.”

Prophecy’s integration with Databricks supports preparing unstructured data for AI, along with the Unity Catalog and Delta Lake. Pipelines developed with Prophecy’s AI-powered visual interface can be turned into Apache Spark or SQL code with dbt core, making it available to all data users.

“We have a good argument, it’s simple,” Bains said. “You have users, we want to make them productive on the platform they already have. Most of the day-to-day pipelines go to all of the business users who understand their data, who understand what they need from their data. I think that’s the future and the copilots are going to make everybody a solid data engineer.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE Research’s coverage of the Data + AI Summit:

(* Disclosure: Prophecy Inc. sponsored this segment of theCUBE. Neither Prophecy nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE