Cloud Giants Gain with DSML Platforms, While GenAI Fans the Flame: Gartner

The advent of generative AI is driving enterprises to adopt consolidated data science and machine learning (DSML) platforms that can handle traditional ML as well as new GenAI use cases, according to Gartner’s latest Magic Quadrant report, which found the cloud giants are quickly gaining share.

Gartner’s definition of a DSML platform is an integrated set of libraries designed to enable data scientists to complete all aspects of the data science lifecycle, either through low-code or code-based approaches. In addition to helping to clean and prepare the data, the platforms–which run either on the Web or are installed on PCs–allow data scientists to analyze the data to understand it, and then build and deploy ML and AI models into production.

While traditional ML is focused on structured data, such as tables of numbers in a database, newer AI approaches, such as GenAI, are based on unstructured data, such as text and images. Today’s DSML platforms can work with both types of data.

“The supported machine learning techniques range from classic regression or decision trees to more complex deep learning and reinforcement learning and GenAI,” the Gartner analysts–Afraz Jaffri, Aura Popa, Peter Krensky, Jim Hare, Raghvender Bhati, Maryam Hassanlou, and Tong Zhang–write. “The models built using these techniques can be used for tasks within business processes such as credit scoring, churn prediction, predictive maintenance, recommendation, and image classification.”

GenAI is driving a lot of growth in DSML platform adoption these days. Gartner says that 53% of respondents in a recent survey cited GenAI demand “as driving a major increase in DSML platform spend in 2024 and beyond.” However, building GenAI products is notoriously difficult, and GenAI projects vastly outnumber actual GenAI deployments.

“The surge in demand for AI solutions, including GenAI, is at its peak,” the Gartner analysts write, “yet the raw materials of data, models, code, and infrastructure have never been more complex to assemble into trusted, scalable products.”

(greenbutterfly/Shutterstock)

The good news for GenAI afficionados (i.e. all of us) is that DSML platforms are ready to step up and help bring GenAI into the AI and ML fold. DSML platforms have developed established processes for building all sorts of ML and AI products, and new GenAI workloads can benefit from that progress.

However, there’s a bit of a gap between what organizations want with GenAI and how DSML tools will get them there, in terms of the personas who are using these tools. That’s because GenAI is bringing more folks from the business side of the house into data science, Gartner says. They typically have less advanced skills than full-blown data scientists.

The rapid pace of AI development is changing the roles of the humans who put it all together. Gartner says that, by 2027, 50% of data analysts will be retrained as data scientists. Today’s data scientists, meanwhile, will become tomorrow’s AI engineers.

But there’s good news here too, according to Gartner, as features like AutoML–where software makes decisions related to the features, weights, and ML models to use–have become commonplace in DSML platforms.

Plus, these AutoML capabilities are being complemented with GenAI-based capabilities like coding assistants and natural language querying, which will further lower the barrier to entry and encourage more democratization of data science.

As GenAI drives demand for more AI and pushes more people into the AI business, DSML platforms will play a critical role, Gartner says.

“The issue for data science and AI leaders,” the Gartner analysts write, “is how to manage and provide governance over the activities of distributed DSML teams and maximize efficiencies through collaboration with centralized resources.”

Giants of the Cloud

(Source: Gartner)

The cloud giants have made sizable gains in the market for data science and machine learning platforms, Gartner said in its latest Magic Quadrant report. But thanks to tailwinds from GenAI and the need for inter-team collaboration, smaller software companies are expected to continue to innovate and thrive.

Amazon Web Services, Google Cloud, and Microsoft Azure were all in the leaders quadrant of Gartner’s latest Magic Quadrant for Data Science and Machine Learning Platforms, where they were joined by Databricks, Dataiku, DataRobot, SAS, and Altair.

The authors of the report say that hyperscaler offerings are gaining more traction in the market for DSML platforms “due to the availability of compute, data and infrastructure needed for DSML development.”

“Yet, there is still room for others to thrive, especially in relation to enabling collaboration between teams, a key pillar for DSML and GenAI development,” the authors continue. “Bringing DSML techniques to more enterprises, and every area of the enterprise, is an opportunity that can be grasped by vendors and end users alike. The foundational use case of data science for insight-driven decision making must not be lost in the GenAI noise, and DSML platforms offer the perfect place to unite advanced analytics and AI development.”

It’s remarkable how quickly the DSML leaderboard has changed in just a few years. Among the eight vendors currently in Gartner’s leaders quadrant, only SAS and RapidMiner (now owned by Altair) were there in 2019. Even in 2021, none of the cloud giants were in the leaders quadrant, although Databricks, Dataiku, and SAS made the cut.

Vendors that were considered leaders in the DSML Magic Quadrant in 2021 have regressed in terms of their ability to execute and completenesss of vision, including KNIME, TIBCO, Mathworks, and IBM. Alteryx, which was in the challengers quadrant, is now in the niche players quadrant, along with MathWorks and KNIME.

Cloudera, meanwhile, has moved from the niche players quadrant in 2021 into the visionaries quadrant for 2024, where it sits with H20.ai and Domino Data Lab. Alibaba Cloud, meanwhile, has moved up from the niche players category into the challenger’s quadrant, where IBM also currently sits. Anaconda is still in the niche players quadrant, where it has been since 2019.

Related Items:

How Broad Is Your Database’s Data Ecosystem? Gartner Takes a Look

A ‘Glut’ of Innovation Spotted in Data Science and ML Platforms

The ‘Big Bang’ of Data Science and ML Tools