Our customers say their biggest challenge in getting Generative AI from pilot to production is the “measurement problem.” It’s hard to measure and trust these systems. LLM providers share performance results in controlled tests, but companies change the models and add their own data. This makes real-world evaluation difficult.
In the current state of AI, most organizations have moved from simple single call LLM applications to AI systems. These systems use multiple tools, retrieval strategies, reasoning steps, and business rules, along with a LLM, to create a single output from a user prompt. There’s a lot going on under the hood.
At Databricks, we’re democratizing access to analytics and intelligent applications by marrying customers’ data with powerful AI models tuned to the unique characteristics of their business. We’re leading the way in the shift from general intelligence to what we call Data Intelligence. As our users can attest, even a small improvement in the quality and efficiency of data can make an outsized impact. And with the explosion of applications built on Databricks Mosaic AI this year, it is critical that Databricks is able to offer industry-leading and scalable evaluation for our customer’s compound systems.
We’re excited to share that Databricks Ventures has invested in the Series B funding round of Galileo, a startup focused on Evaluation Intelligence for AI teams everywhere. And with this deeper partnership, now all Databricks models are natively available to Galileo users, giving our customers both Data Intelligence and Evaluation Intelligence.
Why Galileo
Galileo offers a new type of Evaluation Intelligence with its Luna Evaluation Suite, a set of proprietary metrics and Evaluation Foundation Models. Galileo brings together Luna and its opinionated workflows for experimenting, monitoring, and real-time protection to empower teams with evaluations that:
- Span the entire AI development workflow
- Just work out-of-the-box without needing ground truth data
- Scale to millions of AI queries a month without impacting cost or latency
- Are equally helpful to engineers, developers, and business users
- Continuously improve by auto-adapting to the data unique to your use case
This allows teams to rapidly ship trustworthy applications while ensuring consistent outputs and positive brand experiences for internal and external users. Galileo has proven experience across the enterprise, including existing relationships with Fortune 50 Databricks customers and more than 800% business growth over the last year.
What’s Next for Galileo and Databricks
Galileo now offers Databricks’ latest generation of high-quality, pre-trained foundation models from its Unity Catalog, Databricks Marketplace, and Mosaic AI Model Service. All off-the-shelf and fine-tuned models available to users in Databricks can now be accessed to power evaluations in Galileo through our native integration, requiring only your OAuth Databricks credentials. Through this integration, users now get the best of both Data Intelligence and Evaluation Intelligence—all part of a single ecosystem.
This is just the first step in moving toward Data Intelligence with Databricks and Galileo. In the future, Galileo plans to close the full development loop by integrating with Databricks’ data layer, allowing for automatic algorithmic high-quality Test Set and Fine-tune Dataset curation for Evaluations and efficient RLHF — all natively within the joint ecosystem.
We’re excited to roll out these integrations — reach out to register interest here to get started with the joint solution today. Stay tuned for future updates and make sure to join the Databricks and Galileo team on October 29th at the GenAI Productionize 2.0 virtual summit to learn more about the future of AI evaluation.
Source link
lol
