MinIO expands its enterprise object storage offering to handle AI workloads – SiliconANGLE

Saying generative artificial intelligence model training has changed the game in object storage, cloud-native storage startup MinIO Inc. today launched what it says is a major expansion of its product line that addresses the data creation and management needs of exabyte-scale infrastructure.

MinIO sells a high-performance, Kubernetes-native object store compatible with Amazon Web Services Inc. S3 and permits rapid access to cloud-hosted data. Object storage is a highly scalable method for storing a wide variety of structured and unstructured data types distributed across multiple hardware devices. MinIO says that more than half of Fortune 500 companies use its software-defined object storage.

The company said its new MinIO Enterprise Object Store is expressly designed for the performance and scale needs of massive AI workloads with billions of objects, hundreds of thousands of cryptographic operations per node per second, and queries against an exabyte-scale namespace.

The new service provides a catalog for indexing, organizing, and searching objects using GraphQL, an open-source query language that describes how a client should request information through an application program interface.

“The problem with Postgres (a widely used open-source relational database management system) is that developers have had to write APIs to access data because of schema changes,” said Anand Periasamy (pictured), co-founder and chief executive of MinIO. “GraphQL is a data query API that can work with any structured data store. You can use it to query object data for which you have no S3 API, such as showing all namespaces created by a certain user.”

Data-specific firewall

A built-in firewall is designed for data and built to protect billions of objects. Standard Layer 7 firewalls “understand application traffic but have no idea about data traffic,” Periasamy said. “This firewall, because it understands the S3 API, understands if policies have been modified to grant access to data.  You can configure it to throttle the number of requests at the bucket or object level. It’s also a load balancer.”

A MinIO-specific key management server addresses the specific performance, availability, fault tolerance, and security issues of managing billions of cryptographic keys. It supports multi-tenancy, with each tenant able to be assigned to its own isolated enclave independently. Object storage in data lakes is often encrypted because of the number of users who have access to the lake.

“The challenge of dealing with billions and billions of objects that have to be encrypted on a per-object basis with hundreds of thousands of encryption activities occurring simultaneously at subsecond speeds weren’t the ones conventional key management servers were designed for,” said Jonathan Symonds, MinIO’s chief marketing officer. “We built a solution that’s very, very lightweight, but very, very secure, that allows you to attack those types of problems to  run encrypted all the time.”

Transparent caching

An included caching service uses server DRAM memory to create a distributed shared cache for high performance. “It is completely transparent to any application requesting an object from our service,” Symonds said.

An observability suite provides data infrastructure-centric metrics, audit logs, error logs and traces across all system components in every instance, cluster and erasure code set. “There are some great observability solutions out there, but the challenge is you have to be elite at those solutions and incredibly knowledgeable of your data infrastructure,” he said. “We’ve taken our knowledge of very large-scale data infrastructure and built a very bespoke solution around metrics, logs and traces to allow you to see down to the drive level and erasure code set problems like failing drives or slow networks that might be problematic.”

Tying it all together is a console that displays an organization’s instances of MinIO across public and private clouds, edge locations and colocation instances. “In the past, you could only see one instance for one console,” Symonds said. “Now you can see multiple instances and manage features across all of them.”

MinIO said it plans no change in pricing, but the minimum amount of data a customer needs to qualify for the Enterprise Object Store will double to 200 terabytes. “That does effectively increase the starting price for the enterprise light tier, but the per-unit pricing remains the same,” he said.

Photo: SiliconANGLE