Exclusive: Kong launches AI Gateway to help enterprises govern and scale generative AI

Kong Inc., the cloud connectivity company, has announced the general availability of Kong AI Gateway, providing enterprises with an AI-native API gateway to govern and secure generative AI workloads across any cloud environment. The product, first released in beta in February, has already seen significant adoption as organizations rush to operationalize generative AI technology.

The Kong AI Gateway offers a suite of infrastructure capabilities tailored for AI, including support for multiple large language models (LLMs), semantic caching, semantic routing, semantic firewalling, and model lifecycle management. Crucially, it integrates seamlessly with Kong’s existing API platform, allowing enterprises to manage their AI alongside traditional APIs.

“Organizations and developers are building new gen AI use cases to create better user experiences and customer experiences. But like every technology in the world, in the early days of starting something new with cutting edge technology, it is very hard to scale that in production without having proper infrastructure,” explained Marco Palladino, CTO and co-founder of Kong, in an exclusive interview with VentureBeat. 

According to Palladino, the Kong AI Gateway is “probably the most capable AI infrastructure technology in the world,” due to being built on top of Kong’s existing gateway features while providing deep AI-specific capabilities. He drew parallels to the early days of APIs, noting that APIs only truly exploded at scale once the right infrastructure emerged to support developers.

Introspecting AI traffic and providing a unified API

A key differentiator is Kong AI Gateway’s ability to introspect AI traffic and provide a unified API to consume one or more AI providers. “Every other API management platform, they take the LLM, the API that gives you access to the LLM, as being one API like any other API,” said Palladino. “That’s insufficient. Kong goes to the next level.”

The gateway provides prompt security, compliance, governance, templating, and a lifecycle around AI prompts. It also offers “L7 AI observability metrics” to give visibility into provider performance, token usage, and costs across AI traffic — capabilities Palladino says are unmatched, as competitors “stop at the API level, they don’t go deeper into the AI level.”

Monetizing fine-tuned AI models alongside APIs

Kong’s unified control plane, Kong Konnect, will enable organizations to monetize their fine-tuned AI models alongside traditional APIs. “We think that the next frontier to API monetization is monetization of the models themselves,” Palladino told VentureBeat. “Because when the models are being fine-tuned with specific corporate intelligence that only your organization has, there is value in harnessing and monetizing that intelligence.”

The launch comes amid skyrocketing interest in generative AI following the viral success of OpenAI’s ChatGPT. But many enterprises are still grappling with how to effectively deploy and govern the technology. Kong aims to simplify that with its purpose-built AI gateway.

“Every enterprise organization in the world was trying to build the same thing that we built,” said Palladino, noting the product was accelerated to general availability due to overwhelming demand. “Once we announced that there was a beta capability, they all wanted to learn more. We already have enterprise customers in production with this capability, some of them actually went in production with the beta version.”

Originally founded under the name Mashape in 2009 and spun out as Kong in 2017, the company has raised over $170 million in venture funding to date. The company powers trillions of API transactions for over 500 organizations globally. With its prescient bet on AI-native infrastructure, Kong is strongly positioned to enable the next wave of generative AI adoption in the enterprise.

“In order to unlock GenAI at scale, it is important that we deploy the right AI infrastructure,” argued Palladino. “Kong AI Gateway is exactly that.”