We are excited to announce that Azure Private Link is now Generally Available (GA) for Databricks serverless and Mosaic AI Model Serving workloads! Now you can enable private connectivity from Databricks SQL, Jobs, Notebooks, Delta Live Tables, and Mosaic AI Model Serving CPU/GPU endpoints to your Azure Data Lake Storage (ADLS) and managed Azure resources. Today we are also introducing new support for 60+ Azure 1st party resources, such as Azure OpenAI and Azure SQL, which expands on our announcement earlier this year of private link support for DBSQL warehouses to Azure Storage.
Azure Private Link provides a direct, secure connection for outbound access to customer resources like cloud storage, secrets, SQL databases, and AI models without exposing these connections to the public internet. If you’re eager to try this out, log in to your Databricks account console and configure these connections in a few clicks using the Network Connectivity Configuration (NCC) tool. If you are unfamiliar with the NCC or how serverless networking works, read on! In this blog, we will give an overview of serverless network connectivity and some of the top use cases we are seeing from customers.
An important reminder – private link data processing charges are currently waived for outbound connections from serverless!
Simple onboarding at scale with the Network Connectivity Configuration (NCC)
Serverless compute provides many benefits, such as fast start-up times, automatic scaling, and performance improvements. We do this while safeguarding customer data, by providing multiple layers of security. When accessing their data, many of our customers want dedicated and private connectivity.
“Azure Private Link enables us to achieve robust environment separation with serverless compute, enhancing our security and operational efficiency. Specifically, we are using Azure Private Link for Model Serving, which offers dedicated and private connectivity, significantly improving our network security.”
— Vukola Milenkovic, Databricks Solution Manager, Erste Group
With the Network Connectivity Configuration (NCC) tool, configuring Azure Private Link is simple. This centralized management tool allows you to map and apply connectivity settings, minimizing the number of private endpoints needed. As our serverless products and backend connections support expand, NCC remains your primary hub for managing connectivity—keeping network administration efficient and streamlined across your Databricks deployment.
The NCC, a regional account-level object, is also a powerful way to scale to your enterprise’s needs. You can create up to 10 NCCs in an account, and map 50 workspaces to each NCC. Each NCC represents a logical network, allowing you to re-use a private endpoint you’ve configured in an NCC across many workspaces. Further, all our serverless workloads re-use the same endpoint when accessing a resource. The net result is simplified management, reduced private endpoint per hour costs you pay, and scale enablement to many private endpoints without worrying about limits and quotas.
“Azure Databricks advanced networking features provide secure and straightforward management for large-scale serverless data transformations and analytics. Private Link access to our own resources, like ADLS and SQL Server, is essential for our security posture and operations.”
— Filippo Pannarella, Cyber Security Architecture and Engineering Lead, Plenitude
Unlocking New Possibilities: Top Use Cases with Azure Private Link for Serverless Compute
The General Availability of Azure Private Link across Databricks serverless compute opens up exciting, secure ways to connect with key Azure services. Here are some of the top scenarios we’ve seen our customers configure:
Register and Serve Models From Unity Catalog over Private Link
You can register and serve Unity Catalog (UC) models from a storage account over Private Link. All the writes and reads will be private and work seamlessly. In addition, Inference Table, a feature to capture incoming requests and outgoing responses for your model endpoints, now also supports logging to a UC table over Private Link.
Connect Privately to Azure OpenAI as an External Model
You can also create and access external models in Azure OpenAI from Databricks serverless compute workloads —all as if the model exists in your Databricks network, with a private, secure connection. Contact your account team if you are interested in this scenario.
Private Lakehouse Federation for Azure SQL
Many of our healthcare customers store their clinical data in Azure SQL. Private Link works with Lakehouse Federation to allow our customers to securely access this data.
Get Started
Azure Private Link is available on the Premium Tier version of Azure Databricks. The list of Azure-managed services that support Azure Private Link can be found here. Refer to our documentation for step-by-step instructions on configuring the NCC for private connectivity on your Databricks workspaces. If you need NCC limits extended beyond published limits, contact your account team.
Starting December 4, 2024, Databricks will begin charging for networking costs on serverless workloads that connect to external resources. However, all serverless outbound Azure Private Link connections currently come with no additional data processing charges. This TCO benefit will continue as we expand Private Link support to other first-party and third party Azure resources.
Visit the Security and Trust Center for more information about Databricks security best practices and features available to customers.
Source link
lol
