The data observability market is booming, as companies look to get a handle on their data flows in preparation for generative AI and other data-intensive initiatives. That’s good news for vendors like Acceldata, which has built a full data observability stack for the modern enterprise.
The observability market is currently in the midst of a big transformation from the previous generation of product, which was based on an application-centric view of enterprise IT operations, to a new data-centric view, according to Rohit Choudhary, the CEO and founder of Acceldata.
“This is a new and emerging field,” Choudhary said. “Companies have become ever so dependent upon data applications, data products, and data pipelines. And so we are essentially helping them monitor them, update them, and make sure that there are very high [quality] and reliable outcomes of their data systems and of their AI systems.”
The growing importance of data is forcing companies to rethink not only how they build their ETL/ETL data pipelines, but how they monitor them for problems. Data observability systems like Acceldata’s look at the flow of data within these pipelines and gauge the reliability of it as it flows from the point of origin to the point of consumption. They also analyze the performance of the underlying infrastructure, as well as the cost of the whole system.
“It’s a new architectural paradigm,” Choudhary said. “Data processing is very different than application processing.”
Rohit Choudhary is the CEO and founder of Acceldata
The previous generation of IT observability tools are focused on monitoring applications, Web services, and APIs, which leaves a gap when it comes to keeping up with data flows. That is where Acceldata and other data observability tools step in.
“This is now a multi-generational data stack. Folks still have AS/400s. [They have] Spark and Databricks and Snowflake and Redpanda,” Choudhary said. “The volume of data is just going up by 2.5 to 4 times, and there’s no number of engineers who will be able to solve the operability problem.”
A large part of data observability is verifying that data flowed where it needed to. Companies are under increasing regulatory pressure to ensure that their numbers are correct, and tools like Acceldata’s can provide the checks that give companies the confidence that their data is timely and accurate.
“There’s a revenue problem, there’s a retention problem, and then there is a compliance problem,” Choudhary said.
A recent Gartner report on data observability concluded that, “by 2026, 50% of enterprises implementing distributed data architectures will have adopted data observability tools to improve visibility over the state of the data landscape, up from less than 20% in 2024.”
There will likely be seven to eight big vendors serving the bulk of the data observability market by 2028 before the market starts to consolidate around 2030, Choudhary said. “So this is a completely new territory,” he added.
Acceldata developed its data observability architecture using a mixture of open-source systems and proprietary development. It has a mix of various databases, including the Clickhouse real-time analytics database under the cover to surface fast-moving data.
“You have data coming to you on the observability platform at very many different velocities, and so what you need is different databases to capture that level of intensity of data, both in format and flexibility,” he said. “So the [Acceldata platform] is a combination of NoSQL databases, some search indexes, document data stores, and relational databases and all of that is then encapsulated in the platform, which is essentially what gives us the power of the platform.”
Traditional observability tools look at logs, traces, and metrics, but the big three for data observability is cost, data quality, and performance–and sometimes infrastructure, Choudhary said. The data observability space mostly lacks industry standards for these data types at this point, so Acceldata is using other standards used in the wider observability world, Choudhary said.
“We use the OpenTelemetry standard for our SDKs,” he said. “We actually created the model because we know that a lot of application developers will convert into data engineering developers and they’ll expect similar interfaces, and the same ways by which they can actually log potentially important information for them which they would like to look at in production time frames. So we’re following exactly those standards.”
Eventually, the field of data observability will likely require its own standards, a la OpenTelemetry, Choudhary concedes. But at the moment, the field is too young and moving too fast for the industry to plant a flag with one universal approach.
Related Items:
How Acceldata Helped T-Mobile’s Data Modernization Strategy
Data Observability in the Age of AI: A Guide for Data Engineers
There Are Four Types of Data Observability. Which One is Right for You?
Source link
lol
