Avoid syncing cluster policy values into shared bundle config by ilyakuz-db · Pull Request #7170 · databricks/cli

ilyakuz-db · 2026-10-23T14:51:58Z

Changes

config-remote-sync no longer copies cluster policy values into bundle YAML. After changes are extracted and selected, job and cluster specs whose remote policy_id is set are checked against that policy (read lazily, once per sync):

Fields under fixed rules, including * wildcards, are never synced. The backend rejects any other value for them, so the remote value always comes from the policy.
Fields under other rules are dropped only when they equal the policy defaultValue and apply_policy_default_values is set; any other value is a user edit and syncs.
Values authored in YAML are kept, and policy values never remove them.
If the policy no longer exists, policy-dependent fields are listed in skippedChanges with the new partial status, and other edits still sync.

Clusters without a policy are not filtered. The blanket custom_tags/cluster_log_conf skip is removed, so tags added in the UI now sync.

Why

Syncing a UI edit could copy one target's policy values into shared YAML, and deploying that YAML to a target with a different policy then failed validation.

Tests

Unit tests for rule matching, plus acceptance tests for cross-target deploys, missing policies and deploy-and-plan after sync.

This PR was written with assistance from Codex.

This pull request and its description were written by Isaac.

github-actions · 2026-10-23T14:58:27Z

Commit: a832169e2 | Workflow run

Regression test report

Selected 6 changed tests (limit=20, 0 not selected)

12 tests: regression=10 unreleased=0 coverage=2 failing=0

test	branch	main (`814bdfe`)	latest (v1.22.1)
TestAccept/bundle/config-remote-sync/cluster_policy_cross_target/DMS=	✅	❌	❌
TestAccept/bundle/config-remote-sync/cluster_policy_cross_target/DMS=true	✅	❌	❌
TestAccept/bundle/config-remote-sync/cluster_policy_remote_addition/DMS=	✅	❌	❌
TestAccept/bundle/config-remote-sync/cluster_policy_remote_addition/DMS=true	✅	❌	❌
TestAccept/bundle/config-remote-sync/cluster_policy_replaces_array/DMS=	✅	❌	❌
TestAccept/bundle/config-remote-sync/cluster_policy_replaces_array/DMS=true	✅	❌	❌
TestAccept/bundle/config-remote-sync/external_cluster_policy_cross_target/DMS=	✅	❌	❌
TestAccept/bundle/config-remote-sync/external_cluster_policy_cross_target/DMS=true	✅	❌	❌
TestAccept/bundle/config-remote-sync/missing_cluster_policy_structural_changes/DMS=	✅	❌	❌
TestAccept/bundle/config-remote-sync/missing_cluster_policy_structural_changes/DMS=true	✅	❌	❌
TestAccept/bundle/config-remote-sync/policy_injected_cluster_fields/DMS=	✅	✅	✅
TestAccept/bundle/config-remote-sync/policy_injected_cluster_fields/DMS=true	✅	✅	✅

TestAccept/bundle/config-remote-sync/cluster_policy_cross_target/DMS=: branch ✅ | main (814bdfe) ❌ | latest (v1.22.1) ❌

main (814bdfe):

acceptance_test.go:1341: Diff:
        --- bundle/config-remote-sync/cluster_policy_cross_target/output.txt
        +++ /tmp/TestAcceptbundleconfig-remote-synccluster_policy_cross_target2702789762/002/output.txt
        @@ -13,6 +13,9 @@
         
         Resource: resources.jobs.policy_sync
           description: replace
        +  job_clusters[job_cluster_key='main'].new_cluster.custom_tags['workspace']: add
        +  job_clusters[job_cluster_key='main'].new_cluster.spark_env_vars: add
        +  job_clusters[job_cluster_key='main'].new_cluster.workload_type: add
         
         
         
        @@ -26,56 +29,39 @@
         +      description: Edited in the UI
                job_clusters:
                  - job_cluster_key: main
        -
        -=== Deploy shared YAML with the prod policy
        ->>> [CLI] bundle deploy -t prod
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/prod/files...
        -Created cluster_policies.policy
        -Created jobs.policy_sync
        -Files: 4 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -
        -=== Add a user tag beside the policy tag in the UI
        ->>> [CLI] bundle config-remote-sync -t dev --save
        -Detected changes in 1 resource(s):
        -
        -Resource: resources.jobs.policy_sync
        -  job_clusters[job_cluster_key='main'].new_cluster.custom_tags['UserTag']: add
        -
        -
        -
        ->>> diff.py databricks.yml.backup databricks.yml
        ---- databricks.yml.backup
        -+++ databricks.yml
        -@@ -39,4 +39,5 @@
        +@@ -39,4 +39,11 @@
                      custom_tags:
                        owner: bundle-author
        -+              UserTag: edited-in-ui
        ++              workspace: dev
        ++            spark_env_vars:
        ++              ENVIRONMENT: dev
        ++            workload_type:
        ++              clients:
        ++                jobs: true
        ++                notebooks: false
                tasks:
                  - task_key: main
         
        -=== Deploy the user tag with the prod policy
        +=== Deploy shared YAML with the prod policy
         >>> [CLI] bundle deploy -t prod
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/prod/files...
        -Updated jobs.policy_sync
        -Files: 2 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 1 unchanged
        +Created cluster_policies.policy
        +Error: cannot create resources.jobs.policy_sync: Cluster validation error: Validation failed for custom_tags, workspace must be prod (is "dev") (400 INVALID_PARAMETER_VALUE)
         
        ->>> [CLI] jobs get [POLICY_SYNC_ID_2] -o json
        -{
        -  "UserTag": "edited-in-ui",
        -  "owner": "bundle-author",
        -  "workspace": "prod"
        -}
        +Endpoint: POST [DATABRICKS_URL]/api/2.2/jobs/create
        +HTTP Status: 400 Bad Request
        +API error_code: INVALID_PARAMETER_VALUE
        +API message: Cluster validation error: Validation failed for custom_tags, workspace must be prod (is "dev")
         
        +Files: 4 uploaded, 0 deleted
        +
         >>> [CLI] bundle destroy -t prod --auto-approve
         The following resources will be deleted:
           delete resources.cluster_policies.policy
        -  delete resources.jobs.policy_sync
         
         All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/prod
         
        -Destroy: 2 deleted
        +Destroy: 1 deleted
         
         >>> [CLI] bundle destroy -t dev --auto-approve
         The following resources will be deleted:
        @@ -85,3 +56,5 @@
         All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/dev
         
         Destroy: 2 deleted
        +
        +Exit code: 1