Today, we are excited to announce Databricks AI/BI, a new type of business intelligence product built from the ground up to deeply understand your data’s semantics and enable anyone to analyze data for themselves. AI/BI is built on a compound AI system that draws insights about your data from its full lifecycle across the Databricks platform – including ETL pipelines, lineage, and other queries. It powers two complementary product experiences:
- AI/BI Dashboards: an AI-powered, low-code dashboarding solution that includes all the conventional BI capabilities you’d expect out-of-the-box, for answering a fixed set of business questions; and
- Genie: a conversational interface that can learn the underlying data and semantics continuously based on human feedback, and can answer a much broader set of business questions based on its reasoning capabilities, while still providing certified answers for query patterns specified by the data teams.
These features make AI/BI a significant step towards true self-service BI, significantly broadening the range of analytics that everyday users can perform. Additionally, AI/BI integration with Databricks’ Data Intelligence Platform ensures unified governance, lineage tracking, secure sharing, and top-tier performance at any data scale.
In the rest of the blog, we discuss the reasons why GenAI struggles to work so far in BI beyond demos. We then discuss why we believe AI/BI’s design can overcome these issues, and validate it through real-world evidence.
Why GenAI has fallen short in BI
For the last 30 years, business users have been given reports and dashboards to answer the data questions they have. However, as businesses evolve, these users rely on scarce and overworked data professionals to create new visualizations to answer new questions. Business users and data teams are trapped in this unfulfilling and never-ending cycle that generates countless dashboards but still leaves many questions unanswered.
With the excitement around LLMs, the BI industry started a new wave of incorporating AI assistants into BI tools to try and solve this problem. Unfortunately, while these offerings are promising in concept and make for impressive product demos, they tend to fail in the real world. When faced with the messy data, ambiguous language, and nuanced complexities of actual data analysis, these “bolt-on” AI experiences struggle to deliver useful and accurate answers.
The reality is that it’s not enough to just point an LLM at a database schema and do text-to-SQL, because the schema itself is missing a lot of knowledge, like definitions of business processes and metrics, or how to handle messy data. The other approach is to capture this understanding in formal semantic models, but they require significant up-front investment, can’t capture all the nuances, and are impractical to keep up-to-date as data and business processes evolve.
Compound AI System
The “real” semantic model lives in people’s heads, and it comes pouring out whenever they interact with Databricks systems to run queries, create dashboards, and perform analyses. Databricks AI/BI is a new BI product that captures this understanding from interactions across Databricks to augment the context already available in the Data Intelligence Platform, and leverages the resulting knowledge to deliver useful answers in the real world.
At the core of AI/BI is a compound AI system that utilizes an ensemble of AI agents to reason about business questions and generate useful answers in return. Each agent is responsible for a narrow but important task, such as planning, SQL generation, explanation, visualization and result certification. Due to their specificity, we can create rigorous evaluation frameworks and fine-tuned state-of-the-art LLMs for them. In addition, these agents are supported by other components, such as a response ranking subsystem and a vector index. Together, they provide reasoning capabilities far beyond any individual, monolith model.
The system is designed to continuously learn and improve its performance based on human feedback. For example, if told the definition of a churned customer, AI/BI will not only use that knowledge to address similar queries (e.g. churned customers in EMEA vs. America), but also use that knowledge to calculate churn rate, or infer the meaning of retained customers. AI/BI persists this knowledge beyond a single analysis or conversation to get better and better, much like a human analyst. In addition, AI/BI learns from other information about your data in the Databricks platform, such as ETL pipelines, lineage, popularity statistics, and other queries on the data.
This compound AI system is then used to power both Dashboards and Genie.
AI/BI Dashboards
Despite their aforementioned shortcomings, dashboards are still the most effective means of operationalizing pre-canned analytics for regular consumption. AI/BI Dashboards make this process as simple as possible, with an AI-powered low-code authoring experience that makes it easy to configure the data and charts that you want.
They come with standard BI capabilities you’d expect, including sleek visualizations, cross-filtering, and periodic PDF snapshots via email. But notably, they also don’t come with things you don’t want – no cumbersome semantic models, no data extracts, and no new services for you to manage. Furthermore, exploring insights unavailable in the dashboard is a click away into a complementary Genie space.
Genie
To answer the large and constantly changing set of questions that are unanswered by a dashboard, we expose the capabilities of AI/BI’s reasoning engine through a conversational interface, called Genie. No longer limited to a fixed set of charts, Genie can learn the underlying data, and flexibly answer user questions with queries and visualizations. It will ask for clarification when needed and propose different paths when appropriate.
But more importantly, Genie is not just an inscrutable black box. The type of questions business users ask can be high-stakes, and they should not blindly trust a blackbox AI system to provide the answer. As a result, the entire Genie workflow is designed to make the AI better over time through human feedback: it provides a suite of tools for analysts to verify assumptions and fill in the gaps as needed. Instructions, certified answers, confidence voting, and quality monitoring help data teams additionally tune, curate, and benchmark Genie’s performance, ensuring that what they deliver to the business users will be as trustworthy as possible.
Genie also uses the agentic concept of “tools” to provide a mechanism for ensuring trustworthiness. The concept of “certified answers” allows analysts to tell the system about a trusted piece of governed logic like Unity Catalog Functions and Metrics – that it can use as a “tool” to answer a question. This removes any possibility of incorrect logic inference on the system’s part. The Genie incorporates these “tools” into AI/BI’s reasoning framework and invokes them as appropriate to answer questions, sharing with the user the trusted status of the answer provided.
Platform Integration
AI/BI is built on top and tightly integrated with our Data Intelligence Platform. This means out-of-the-box, AI/BI features:
- Unified governance and lineage: AI/BI is deeply integrated into Databricks Unity Catalog. It follows the same governance framework, and any global policies set by administrators will apply in AI/BI. And thanks to Unity Catalog’s lineage capability, data producers or administrators can observe how their data assets are used in AI/BI, and end users can place higher trust on their analysis because they can trace back the origins of their datasets all the way back to data ingestion.
- Effortless sharing, without new user licenses: AI/BI is built right into the Databricks IAM platform which integrates directly with IDPs like Entra AD or Okta so that you can easily share your analysis with anyone in your organization. Because Databricks AI/BI does not have seat-based restrictions, you can add anyone from your organization without having to worry about procuring new licenses.
- Industry-leading price-performance: AI/BI is tightly integrated with Databricks SQL data warehouses and the Photon engine, which contain unique optimizations to deliver high-performance interactions. You will get the best performance regardless of the data volume, from megabytes to petabytes.
- No data extraction required: Consequently, you no longer need to extract the datasets of interest out to a separate BI engine, leading to better freshness of data as well as simpler data governance.
Real-world Validation
We have been testing AI/BI in private preview with a number of customers for the past few months. We want to emphasize that AI/BI is not an omniscient AI that has all the answers out of the box. However, the early feedback is extremely encouraging: users from all types of backgrounds, from business users to company executives, have reported that they can now reduce reliance on their data teams and answer more questions themselves.
Here are what some of our earlier adopters have to say about AI/BI:
Brian Fox, CTO, Sonatype: “Comparing using AI/BI Genie to past efforts is like comparing night and day. It’s been 20 years since I’ve seriously worked with SQL, so having the AI retrieve data is magical. Now, I can achieve this analysis without needing assistance from someone who uses SQL every day.”
Felix Baker, Head of data services, SEGA Europe: “At SEGA, we are using AI/BI to assist decision makers around the organization, by allowing them to ask ad-hoc questions in real-time about Sales and Player Behavior without having to depend on our data experts to construct dashboards and queries. Users have now been able to get detailed insights about game sales and gameplay data by simply asking in natural language. We are excited to utilize AI/BI to democratize data, increase productivity and to enhance the speed of data-driven decision-making throughout SEGA.”
Nick Crnkovich, Analytics Enablement Lead, Block.xyz: “AI/BI Dashboards allow us to quickly generate and distribute insights from the same platform where our data already resides – no additional connections or extractions to configure. Critical for us, our creators can leverage AI in the development process and our business users benefit from a focused view of their data without any additional complexity.”
Philipp Cüppers, Team Lead, Energy Markets and Asset Optimisation, Vattenfall Hydro Germany: “Databricks’ AI/BI offering has given us new tools to democratize data and insights. The enhanced dashboards are now our preferred way of providing unified views of critical data because they are quick to generate and easy to share; Genie enables our business users to ask and answer questions for themselves in real-time. We recently made Genie available to our key stakeholders so that they are able to ask and answer questions about electricity markets and our asset performance in live discussions without being reliant on a data analyst.”
What’s Next
We believe compound AI systems that can draw insights about your data from its full lifecycle will be transformative to the world of business intelligence. The initial release of AI/BI represents a first but significant step forward toward realizing this potential. The system will become smarter over time as usage ramps up and the system evolves. We are grateful for the MosaicAI stack, which enables us to iterate end-to-end rapidly.
AI/BI Dashboards are generally available on AWS and Azure and in public preview on GCP. Genie is available to all AWS and Azure customers in public preview, with availability on GCP coming soon. Genie requires Unity Catalog and Databricks SQL Serverless or Pro warehouses. Customer admins can enable Genie for workspace users through the Manage Previews page. There is no additional fee beyond warehouse costs for both products. For business users consuming Dashboards, we provide view-only access with no license required.
Beyond our efforts with AI/BI, we know many of our BI partners are innovating to make analyzing data in the Data Intelligence Platform easier. We are excited about the potential to open up our reasoning capabilities and semantic models as APIs for our BI partners to make it possible for all organizations to benefit from an AI-first approach to business intelligence, no matter which BI tool you’ve standardized on.
To learn more about Databricks AI/BI, visit our website and check out the keynote, sessions and in-depth content at Data and AI Summit. Enroll for Data Warehousing, Analytics and BI sessions at the Data + AI Summit, or watch the on-demand recordings online after the event. Also, follow us @Databricks for the latest news and updates.
Source link
lol
