Elastic’s new Search AI Lake enables highly scalable vector search for generative AI apps – SiliconANGLE

Enterprise search technology provider Elastic N.V. is hoping to make the data essential for generative artificial intelligence as well as security and observability workloads more accessible with the launch of its new Search AI Lake service.

Announced today, Elastic Search AI Lake is a radically different offering from existing Elasticsearch deployments. It decouples storage from compute, an approach that allows it to scale search across exponentially larger data volumes with rapid query performance for both traditional structured data and unstructured information represented as vectors. This latter capability makes Search AI Lake an especially useful data store for generative AI services, the company believes.

Elastic Search AI Lake is based on the company’s popular Elasticsearch technology, which in turn is built on the open-source Apache Lucene project. That platform is widely used by enterprises to store, search and analyze enormous volumes of data in real time. It sits at the heart of millions of applications globally that demand comprehensive search capabilities.

In addition to powering general search, Elasticsearch also caters to use cases such as threat detection and application observability, providing tools that can visualize networks and monitor their performance. More recently, Elastic launched the Elasticsearch Relevance Engine that merges vector search with its traditional search engine, enabling it to support unstructured data types such as audio, image and video files.

However, those older offerings are hindered by the fact that they couple storage with compute, which creates a barrier to scalability. By decoupling the two elements, that’s no longer a problem, said Elastic Chief Product Officer Ken Exner.

“To meet the requirements of more AI and real-time workloads, it’s clear a new architecture is needed that can handle compute and storage at enterprise speed and scale – not one or the other,” he said. “Search AI Lake pours cold water on traditional data lakes that have tried to fill this need but are simply incapable of handling real-time applications. This new architecture and the serverless projects it powers are precisely what’s needed for the search, observability and security workloads of tomorrow.”

In an interview with VentureBeat, Elastic Chief Executive Ash Kulkarni said Search AI Lake is built on a very different architecture than traditional data lake offerings such as Databricks Inc.’s Delta Lake and Snowflake Inc.’s cloud data warehouse. Unlike those platforms, it brings its search functionality into the data lake, enabling real-time exploration and queries of the information within, without the need for any predefined schemas.

Kulkarni added that Search AI Lake also offers dense vectors, hybrid search, faceted search and relevance ranking features that are suited to generative AI models and retrieval-augmented generation techniques.

Another difference is that the Elastic Search AI Lake doesn’t use traditional data table formats such as Apache Iceberg or Apache Hudi. That’s because those architectures can hinder data exploration, Kulkarni said.

When data is put into a data lake table, the user must also add metadata and make it possible to search for that metadata, or it becomes almost impossible to search for it. Search AI Lake, on the other hand, uses the Elastic Common Schema format and relies on the Elasticsearch Query Language, which enables data to be explored in a federated way across Elastic clusters.

With Search AI Lake, Elastic is trying to position itself as the data platform for generative AI models, which can be dramatically enhanced with highly scalable vector search. By supporting these capabilities, large language models can enhance their knowledge by searching for the most recent and relevant data as it’s made available in real time, improving their responses.

Search AI Lake is available as a standalone platform and also powers a new offering called Elastic Cloud Serverless. Both are available in technology preview starting today.

Image: SiliconANGLE/Microsoft Designer