We are excited to announce the Public Preview of Databricks serverless budget policies. Administrators can use budget policies to ensure that the correct tags appear automatically on serverless resources without depending on users to attach tags manually, allowing for customized cost reporting and chargebacks.
Since we made serverless compute Generally Available in July, customers have leveraged notebooks, jobs, and Delta Live Tables (DLT) pipelines with serverless compute to take advantage of rapid startup times, streamlined operations, and exceptional reliability.
However, it’s important for organizations to effectively manage serverless workload costs by attributing them to specific projects, cost centers, or departments. Detailed cost attribution creates transparency, promotes accountability, and ultimately helps optimize the usage of Databricks.
Tagging your serverless workloads with budget policies for cost attribution
Effective spend management is crucial to optimizing cost efficiency for serverless compute. By using tags, administrators can group billing records based on cost centers, projects, or other relevant categories. This strategy ensures comprehensive visibility into costs associated with each tag, simplifying budget management across departments and projects.
Tag enforcement as a best practice
With the introduction of serverless budget policies, you can now ensure that tags are applied to any serverless resource that is being created. These policies, which contain one or more tags, can be applied to specific users, groups and/or service principles.
When creating a serverless resource (such as a workflow task or DLT pipeline configured to use serverless compute) users will be able to choose between budget policies assigned to them. In most cases, a user has a single budget policy assigned to them, which will be chosen by default. The budget policy’s tags will then be applied automatically to the serverless resource.
When serverless compute is enabled in your Databricks account, it is available as a compute option for all users in your workspace. Therefore, it is best practice to establish budget policies and assign at least one budget policy to every user in your workspace. This will guarantee that all serverless resources are properly accounted for and associated with a tag.
Getting started
If you still don’t have serverless compute enabled in your account you can do so by following these instructions in AWS or Azure.
Take advantage of our introductory discounts: get 50% off serverless compute for Jobs and Pipelines and 30% off for Notebooks, available until January 31, 2025. This limited-time offer is the perfect opportunity to explore serverless compute at a reduced cost.
You must be a workspace admin to configure budget policies for serverless compute in your workspace. Our documentation for AWS or Azure provides detailed instructions on how to configure budget policies.
Source link
lol
