Redpanda acquires Benthos to enhance its data streaming platform – SiliconANGLE

Redpanda Data Inc. has acquired Benthos, the developer of an open-source platform that helps organizations move information between their applications.

The companies announced the deal this morning. Financial terms were not disclosed.

San Francisco-based Redpanda is backed by more than $160 million in funding from Alphabet Inc.’s GV fund and other investors. It provides a platform, also called Redpanda, that can move data between applications in near-real time. A manufacturer, for example, could use the software to stream malfunction alerts from a piece of factory equipment to a cloud-based troubleshooting application.

Redpanda says its platform can move data between systems with less latency than Apache Kafka, a popular open-source alternative. One reason the software provides better performance is that it’s written in the C++ programming language. C++ is significantly faster than Java, the language in which most of Kafka’s components are implemented.

Another contributor to Redpanda’s speed is its use of a processing approach known as the thread-per-core design. The platform implements this technique using an open-source C++ library called Seastar. According to the company, the technology allows its software to make more efficient use of the hardware on which it runs. 

The platform distributes the computations involved in processing data across a set of central processing unit threads. Instead of running multiple threads on each CPU core as is the usual practice, Redpanda runs only one per core using Seastar. That prevents the threads from interrupting one another’s calculations, which avoids the processing delays associated with such interruptions and thereby improves performance.

Redpanda also promises to provide other advantages over Kafka. The latter software performs several configuration tasks using an open-source tool called ZooKeeper that administrators have to install separately, which complicates the setup process. Redpanda, in contrast, doesn’t use ZooKeeper and can be deployed in as little as a few minutes.

Benthos is a London-based startup with an open-source data streaming engine of the same name. The platform includes more than 200 connectors for popular databases, cloud services and other products. Benthos’ connectors can be used to stream data from the products with which they integrate into Redpanda and vice versa.

Benthos also provides a number of other features. In addition to streaming data into Redpanda, the platform is capable of modifying that data to simplify processing. Developers can change the format of incoming records, enrich them with information from external sources and perform related tasks.

Redpanda used Benthos to develop a new commercial offering called Redpanda Connect that debuted today alongside the acquisition announcement. According to the company, the offering uses one-third the compute capacity required to run Kafka Connect, a similar product for Kafka environments.

“Our largest Redpanda Connect user is pushing 50Gbps via 2X 25Gbps NICs to consume market data,” Redpanda founder and Chief Executive Officer Alex Gallego detailed in a blog post today. “Our smallest Redpanda Connect user is using 100 millicores and 10 MiB of memory, processing hundreds of messages a second deployed on a heavily resource-constrained environment.”

Redpanda will keep Benthos’ core code base and most of its connectors available under an open-source license. Going forward, the company reportedly plans to extend the software with more features for connecting to artificial intelligence platforms.

Image: Redpanda