Confluent enhances support for Apache Flink, easing access to real-time data streams – SiliconANGLE

The data streaming company Confluent Inc. today introduced the latest version of flagship platform Confluent Cloud, with the headlines features aimed at simplifying and securing deployments of the Apache Flink data processing tool.

New features include support for Table API, which makes Apache Flink easier to use for developers who working with the Java and Python programming languages. It’s also adding an extension for Visual Studio Code to support real-time development use cases, plus a private networking capability and client-side field-level encryption to safeguard sensitive data.

Confluent Cloud is a managed data streaming platform that’s used by enterprises to help move data among applications more easily. It was originally based on the open-source Apache Kafka platform, but last year the company extended it with support for Apache Flink as well.

Apache Flink is an open-source big data processing tool that makes it easier to process large volumes of information in real time and move it across systems. By integrating Apache Flink’s capabilities, Confluent Cloud allows customers to stream data from one system to another in real time with Kafka, and modify and process that information using Flink.

It’s an important capability that has many use cases. For instance, companies can use Confluent Cloud to filter data such as purchase logs that might contain erroneous information, as it’s streamed from an on-premises server to the cloud. It can also help to merge multiple data streams into a single stream.

Doing that makes it possible for real-time data to be enriched with additional information from external sources. Other use cases include machine learning, predictive maintenance, personalized recommendations and fraud detection.

Apache Kafka and Apache Flink are a powerful combination, but neither system is that easy to use. In fact, deploying and maintaining those systems created major headaches for information technology teams, which is why Confluent Cloud is so popular. With its platform, Confluent automates much of the work involved, so more companies can explore the benefits of real-time data processing.

Simplifying and securing Apache Flink

With the addition of support for the Table application programming interface in open preview, Confluent’s fully managed Apache Flink offering becomes more accessible for developers familiar with the Java and Python programming languages.

Previously, support was limited to the SQL API, which requires knowledge of the Structured Query Language. Though SQL is good for quickly writing and executing queries, many teams favor Java and Python for their advantages in business logic and custom processing. By supporting both Flink SQL and the Table API, Confluent Cloud for Apache Flink lets developers choose the best language for their use cases.

Meanwhile, private networking for Flink gives users access to a secure environment for their data streams, paving the way for businesses with strict regulatory requirements, such as healthcare companies, to start streaming data. The company explained that users will be able to safeguard in-transit data and merge multiple streams in a protected environment, taking advantage of its simplified network configuration process.

Private networking support is generally available now for Confluent Cloud on Amazon Web Services and in dedicated clusters, with support for additional cloud platforms on the way. Additional security benefits are provided with the debut of client-side field-level encryption, which helps companies to encrypt sensitive information such as personally identifiable data before moving it to another system or application.

Currently in limited availability, this capability allows customers to encrypt individual fields within their data streams, which means that even if a hacker somehow gains access to that information, they won’t be able to make sense of it. It will become generally available to all Confluent Cloud customers soon, the company said.

Finally, Confluent announced the Confluent Extension for Microsoft Corp.’s popular integrated development environment Visual Studio Code, making it simpler for developers to incorporate real-time data streams into any application written in VS Code. The extension is currently in early access, with general availability slated for later this year.

Shaun Clowes, chief product officer at Confluent, said these enhancements will make Confluent Cloud more accessible, so developers of all skill levels can start using its technology.

International Data Corp. research analyst Stewart Bond said the biggest advantage of Apache Flink is that it enables applications that can analyze and respond to real-time data instantly. It’s a powerful advantage that can often justify the expense of investing in a fully managed version of Flink, he added.

“Managed Apache Flink solutions can eliminate the complexities of infrastructure management while saving time and resources,” he said. “But businesses must look for a Flink solution that seamlessly integrates with the tools, programming languages and data formats they’re already using for easier implementation.”

Image: SiliconANGLE/Microsoft Designer