Combining your real-time data into a single source of truth – in this case, one table that you can manipulate – makes for easier, efficient analysis.
In the previous two parts of our Prometheus series, we discussed how to ingest data from both the Prometheus REST API and from Prometheus alert webhooks. Now we have two steady streams of data: one that tracks our metrics, and one that tells us when alerts have been fired and resolved.
In this post, we’ll combine these streams of data into a single table, allowing us to track our metrics with the alerts that are fired and resolved.
As of now, our Prometheus app is producing two sources of real-time data. To further our analysis, we want to combine these tables.
Deephaven supports several join operations; in general, tables are joined via a number of columns used as keys to match rows from the two tables. The Prometheus REST API and the Prometheus alerts webhooks have three fields in common: the job, the instance, and the timestamp. By using the same column names to track these tables, the two tables can be seamlessly joined.
The Prometheus job and instance data are easy to join. They’re simply strings that come from what jobs Prometheus is monitoring in what locations, so we know these will always have overlap in the two tables.
However, the timestamps aren’t as simple. The alerts and metrics do not resolve on identical schedules, so it’s unlikely that the timestamps in the two tables match.
A typical relational-DB approach can be supported in Deephaven by binning the data with lowerBin and then doing a join or natural_join.
But there is a better solution.
Deephaven excels with time series joins. You can use an as-of join (aj). The last join-key is the timestamp, so a binary search to find the closest match will yield the join. The query below shows how to do this with our Prometheus tables.
prometheus_alerts_metrics = prometheus_alerts.aj(table = prometheus_metrics, on =["Job, Instance, PrometheusQuery, PrometheusDateTime"], joins =["Value, MetricTimeStamp = PrometheusDateTime"])
We now have a single table that contains data from both of our ticking tables! From here, you can manipulate and analyze that data in the Deephaven UI – for example, dynamically plot your data or filter to values of interest.
The Prometheus alerts metrics sample app demonstrates how to combine two tables containing ticking time data and store it in Deephaven.
This project is available to be run by anyone, so feel free to run this locally and modify the alert rules, data tables, or any other configuration to see different things you can accomplish using Deephaven!
Source link
lol
