From Generalists to Specialists: The Evolution of AI Systems toward Compound AI

The buzz around compound AI systems is real, and for good reason. Compound AI systems combine the best parts of multiple AI models, tools, and systems to solve complex problems that a single AI, no matter how powerful, might struggle to tackle efficiently.

A Look Back: From Monolithic to Microservices

Before diving into the magic of compound AI systems, let’s rewind a bit and explore how application development has evolved. Remember the days of monolithic applications? These were giant, all-in-one software systems that handled everything—front-end interactions, back-end processing, and database management—within a single codebase. They were powerful, but they had their drawbacks.

Monolithic Architecture Challenges:

• Slow Updates: A small tweak to one part of the application required redeploying the entire system.
• Scaling Issues: If one area of the system was under a heavy load, the entire system had to scale up.
• Single Point of Failure: If one component crashed, the whole system could go down with it.

This paved the way for Microservices Architecture, a game-changer that allowed businesses to split large, monolithic applications into smaller, self-contained services. Each microservice focused on a specific business function like user authentication or inventory management, offering flexibility and scalability that monolithic systems couldn’t match.

Microservices Advantages:

• Faster Updates: Update or deploy just one microservice without touching the rest.
• Scalability: Scale individual services based on demand.
• Fault Isolation: If one service crashes, the others keep running.

But, microservices weren’t without their challenges:

• Higher Overhead: Managing many services required more coordination and infrastructure.
• Latency: Inter-service communication could slow things down.
• Consistency Issues: Keeping data synchronized across services was tricky.

The AI World is Heading the Same Way

We’re seeing the same evolution in the AI world, where large language models (LLMs) like GPT-4 and Meta Llama have become powerful generalists. They excel at handling a wide range of tasks, but, much like monolithic apps, they aren’t perfect for every job.

Compound AI Systems are the GenAI version of microservices. These systems decompose AI tasks into specialized segments. Instead of relying on one giant model to do it all, several models, tools, and components are deployed, each optimized for specific tasks.

Why Compound AI Systems Work So Well:

• Generalists and Specialists: A large foundational model offers broad insights, while specialized models handle niche tasks like medical diagnostics or real-time cybersecurity threat detection.
• Modularity: Need a new model? Just swap it in without retraining the whole system.
• Optimization: Models and tools can be fine-tuned for specific parts of the task, making the entire system more efficient and accurate.

How Compound AI Systems Work

So, what does a compound AI system look like in practice? Picture a team of AI models, each excelling in a particular area, working together to solve complex tasks:

1. Multiple LLMs: Different language models can be used, each optimized for a particular task or domain.
2. External Tools: Search engines, APIs, or data retrieval systems can feed enriched information into the AI pipeline.
3. Orchestrators: A task orchestrator directs when and how to use each model or tool for the task at hand.

This modular approach allows compounded AI systems to break down complex challenges into smaller, manageable steps, much like how microservices revolutionized traditional application development.

Mosaic AI: The Power Behind Compound AI Systems

One platform leading the charge is Databricks Mosaic AI. It gives businesses the tools they need to build production-quality compound AI systems by integrating multiple AI models, data retrieval systems, and external APIs.

Why Databricks Mosaic AI Stands Out:

• Seamless Integration: It securely and easily connects to both internal data sources and external tools, providing rich, contextual data for models to work with.
• Scalability: Individual components can be scaled based on demand using Mosaic AI model serving.
• Customization: Each component can be fine-tuned on custom data to ensure more accurate results.

Building a Compound AI System for Maintenance Bots

To make this more concrete, let’s take a look at a Maintenance Bot powered by Databricks Mosaic AI. The bot is built to assist with troubleshooting machinery, accessing repair manuals, and providing contextual insights.

Step-by-Step Flow Breakdown:

1. Chunking and Storing Manuals:
   • Manuals are broken into smaller pieces and transformed into vector embeddings using Databricks’ embedding model. These embeddings are stored in a vector search index for quick retrieval.
2. Historical Data Collection and Storage:
   • The system collects maintenance logs, service requests, inventory data, and IoT sensor readings from factory equipment. This data is cleaned and aggregated stored in the medallion architecture and enriched data will be stored in a graph database, which stores relationships between machines, parts, defects, and error codes, etc.
3. Building the Compounded AI System:
   • Using the DsPy framework, the AI orchestrates multiple components:
     • The user’s question (e.g., “How to fix error DF-3466?”) is converted into a vector embedding and searched in the manual data inside the vector database.
     • Simultaneously, the question is converted into a Cipher query using a fine-tuned text-to-cypher Llama model. The cipher query is used to query the graph database to see if the error has been previously reported and how it was fixed, delivering contextual insights.
4. Response Summarization:
   • The DsPy framework combines both responses—from the manuals and the graph database—and summarizes the results for the user using the Llama foundation model.
5. Deploying with Mosaic AI:
   • The DsPy framework that orchestrates the compound AI systems is deployed on Databricks Model Serving, ensuring that the AI system is scalable and secure. The Mosaic AI Gateway manages endpoint access and security.
6. FAQ Generation with NLP:
   • Logs of user requests and responses are stored in Delta tables. Using NLP, frequently asked questions are identified, ranked, and served to users when similar issues arise in the future.

This Maintenance Bot is a perfect example of a compound AI system that combines multiple AI components, such as vector embeddings, graph databases, and LLMs, to resolve complex user queries efficiently and intelligently.

The Future is Compound

Just like microservices transformed how we build applications, compound AI systems are transforming how we solve complex problems with AI. With specialized models and tools working together, we can build AI systems that are more flexible, efficient, and powerful.

And with platforms like Databricks Mosaic AI, companies can deploy these systems at scale, ensuring their AI solutions are not only cutting-edge but also production-ready. So, why settle for one brain when you can have a team of genius AIs working together? The future of AI is compound, and it’s happening now. 

For more information on compound AI systems, you can read more in this blog post: The Shift from Models to Compound AI Systems.