(MZStock/Shutterstock)
Making sure data flows where it needs to go can be a daunting task, especially in large enterprises with thousands of data silos and complex applications. One company that’s seeking to make it easier is Kestra, which today announced the completion of an $8 million seed round to further develop its open source orchestration solution.
Kestra develops an event-driven orchestration and scheduling platform that lets users automate complex data and business workflows involving many data types, platforms, languages, and executables. Users can declaratively build their workflows from a command line interface by using YAML to stitch together scripts written in any language. Alternatively, users can build workflows in a drag-and-drop manner from a graphical interface.
The software can automate a wide range of data-oriented tasks. For instance, a customer could use it to automate the process of transforming a JSON file into a CSV using Python-based script, automatically emailing the file to certain recipients and sending an alert via Slack. Or could it be used to automatically grab files from an SFTP site in response to a trigger and move them to Google Cloud Storage.
At runtime, Ketra supports task runners, which allows users to scale up their workflows automatically. For instance, a task runner could be programmed to run a given Python script on AWS, leveraging Terraform to provision the necessary compute and storage.
Kestra lets users schedule tasks to execute on a regular basis or to run in response to a triggering event via APIs or webhooks. It also supports human-into-the loop capabilities for workflow approvals. Once the workflow is activated, Kestra monitors the jobs in real time. Users can watch the jobs unfold from a dashboard; alternatively, Kibana or Grafana can be used.
The open source software boasts a collection of more than 500 plug-ins, enabling automated data movement and task execution across Snowflake, Databricks, dbt Labs, DuckDB, and more. It tracks all the work it does via Git version control.
Kestra provides a dashboard to monitor data orchestration workflows
Adoption of the open source software has been skyrocketing the past year. Since the company announced its pre-seed round of $3 million in 2023, adoption of Kestra has increased dramatically, and the software today is used to manage several hundred million workflows, which is 10x more than a year ago, the company says.
“Large organizations–including banks, retail companies, software providers, and government entities–are executing millions of workflows with Kestra,” Ludovic Dehon, CTO and co-founder of Kestra, said in a press release today. “The platform excels even in highly specialized cases like network monitoring and real-time fraud detection. Most importantly, configuring and scheduling these workflows remains effortless, thanks to a language-agnostic, declarative syntax combined with webhooks and event-driven triggers.”
The seed round was led by Alven with participation from ISAI, Axeleo Capital, and a host of prominent executives in big data, including dbt Labs CEO Tristan Handy; Airbyte CEO Michel Tricot, Datadog CEO Olivier Pomel; Hugging Face CEO Clement Delangue; Talend (owned by Qlik) Co-Founder Bertrand Diard; Algolia CEO Nicolas Dessaigne; and Frédéric Plais of Platform.sh.
“The orchestration market is fragmented with siloed solutions addressing specific job scenarios. Business leaders face challenges in unifying best practices and tools across data, infrastructure, business process management, microservices, and more,” Emmanuel Darras, CEO and Co-Founder of Kestra, said in a press release. “They seek a unified platform to streamline operations, and this is where Kestra excels. Our growing enterprise customers show there’s demand for an API-first, user-friendly product that unites teams, workflows, and pipelines of all types.”
Related Items:
Data Pipelines of a Higher Order
3 Reasons Enterprises Can’t Afford to Ignore Data Integration Any Longer
Source link
lol
