bundle: delete libs/dyn; typed configuration with generated walkers

.agents/rules/auto-generated-files.md

@@ -14,7 +14,10 @@ globs:
   - "internal/genkit/tagging.py"
   - "internal/mocks/**/*.go"
   - "bundle/direct/dresources/configs/*.generated.yml"
-  - "bundle/internal/validation/generated/*.go"
+  - "bundle/config/validate/*_walk.go"
+  - "bundle/config/walk/*_walk.go"
+  - "bundle/config/mutator/resourcemutator/defaults_walk.go"
+  - "bundle/config/resources_iterate.go"
   - "bundle/schema/jsonschema.json"
   - "python/databricks/bundles/version.py"
   - "python/databricks/bundles/*/__init__.py"
@@ -33,7 +36,10 @@ paths:
   - "internal/genkit/tagging.py"
   - "internal/mocks/**/*.go"
   - "bundle/direct/dresources/configs/*.generated.yml"
-  - "bundle/internal/validation/generated/*.go"
+  - "bundle/config/validate/*_walk.go"
+  - "bundle/config/walk/*_walk.go"
+  - "bundle/config/mutator/resourcemutator/defaults_walk.go"
+  - "bundle/config/resources_iterate.go"
   - "bundle/schema/jsonschema.json"
   - "python/databricks/bundles/version.py"
   - "python/databricks/bundles/*/__init__.py"

.agents/rules/style-guide-go.md

@@ -113,7 +113,7 @@ return fieldPaths
 
 ### Encoding
 
-**RULE: When mutating an API response by round-tripping JSON, decode with `json.Decoder` + `UseNumber()`.** The naive `json.Marshal` → `map[string]any` → `Marshal` path corrupts any int64 larger than 2^53 (it degrades to a float64 mantissa — e.g. a real `spark_context_id`) and alphabetizes object keys. `libs/dyn/jsonloader` preserves key order but also lacks `UseNumber`, so it shares the int64 hazard.
+**RULE: When mutating an API response by round-tripping JSON, decode with `json.Decoder` + `UseNumber()`.** The naive `json.Marshal` → `map[string]any` → `Marshal` path corrupts any int64 larger than 2^53 (it degrades to a float64 mantissa — e.g. a real `spark_context_id`) and alphabetizes object keys.
 
 **RULE: Be careful with `encoding/csv` `Writer.UseCRLF = true`.** It rewrites both record terminators AND embedded newlines inside quoted fields to `\r\n`, so tests for quoted multiline fields must expect `\r\n`, not just the line endings between rows.
 
@@ -191,7 +191,7 @@ func NewService(deps ServiceDeps) *Service { ... }
 
 ### Configuration patterns
 
-- Bundle config uses `dyn.Value` for dynamic typing
+- Bundle config is typed structs; `libs/structs/structvar` carries `${...}` references and source locations
 - Config loading supports includes, variable interpolation, and target overrides
 - Schema generation is automated from Go struct tags
 

.agents/skills/bump-sdk/SKILL.md

@@ -9,7 +9,7 @@ allowed-tools: Read, Edit, Write, Bash, Glob, Grep, WebFetch, AskUserQuestion
 
 The SDK version lives in `go.mod` (`github.com/databricks/databricks-sdk-go`) and the pinned spec SHA lives in `.codegen/_openapi_sha`.
 These two move as a pair; everything else in this skill is regenerated from them or is fallout you fix by hand.
-Do not hand-edit generated files (`.codegen/cli.json`, `cmd/workspace/*`, `cmd/account/*`, `bundle/schema/jsonschema.json`, `bundle/internal/validation/generated/*`, `bundle/direct/dresources/configs/*.generated.yml`, `python/databricks/bundles/**`); regenerate them.
+Do not hand-edit generated files (`.codegen/cli.json`, `cmd/workspace/*`, `cmd/account/*`, `bundle/schema/jsonschema.json`, `bundle/config/validate/*_walk.go`, `bundle/config/walk/*_walk.go`, `bundle/direct/dresources/configs/*.generated.yml`, `python/databricks/bundles/**`); regenerate them.
 
 The Python tasks (`pydabs-*`, and the `pydabs-codegen` step inside `generate-check`) all run through `uv`. If one fails because `uv` is missing or because the host's `python3` is too old (e.g. 3.9), install `uv` (`curl -LsSf https://astral.sh/uv/install.sh | sh`) rather than touching the system Python: `uv run` provisions the interpreter each package pins (`>=3.10`, and `==3.13.*` under `python/codegen/`) and downloads it if needed. Do not chase the system Python version.
 

.github/workflows/python_push.yml

@@ -39,7 +39,7 @@ jobs:
             acceptance/bundle/python/ \
             go.mod \
             go.sum \
-            libs/dyn/dynvar/testdata/reference_vectors.json \
+            libs/structs/structvar/testdata/reference_vectors.json \
             python/ \
             tools/task/ \
           )

.nextchanges/bundles/7098-presence.md

@@ -0,0 +1 @@
+* `bundle validate` no longer treats a written zero value in a field that cannot record it (no `ForceSendFields`, e.g. `job_id: 0` in `run_job_task`) or an empty mapping (e.g. `presets: {}`) as set, and does not show the pipeline `development` deprecation warning for `development: true` under `mode: development`. ([#7098](https://github.com/databricks/cli/pull/7098))

.nextchanges/bundles/7098.md

@@ -0,0 +1 @@
+* Speed up bundle commands on large bundles: configuration is kept as typed structs instead of being converted to and from a dynamic tree (`bundle validate` and `bundle plan` with 1000 jobs are about 10x faster and use about 3.5x less memory). ([#7098](https://github.com/databricks/cli/pull/7098))

AGENTS.md

@@ -14,7 +14,7 @@ This is the Databricks CLI, a command-line interface for interacting with Databr
 
 **RULE: Keep each PR focused on one change.** If you notice an unrelated cleanup, bug fix, or refactor while making your primary change, leave it alone or put it in a separate PR. Reviewers consistently ask to split mixed PRs, especially when a dependency bump or schema diff rides along with a feature change.
 
-**RULE: Before adding a new helper, search the codebase for an existing one.** Common homes: `libs/` (shared utilities), `libs/databrickscfg/` (config), `libs/git/`, `libs/filer/`, `libs/cmdio/` (CLI I/O, spinners, prompts), `libs/env/` (env vars), `libs/testserver/`, `libs/structpath/` and `libs/dyn/` (path / dynamic values), `acceptance/bin/` (acceptance test helpers), `internal/mocks/` (generated mocks). A function that duplicates an existing name and signature in the same package is a compile error waiting to happen; grep before you name.
+**RULE: Before adding a new helper, search the codebase for an existing one.** Common homes: `libs/` (shared utilities), `libs/databrickscfg/` (config), `libs/git/`, `libs/filer/`, `libs/cmdio/` (CLI I/O, spinners, prompts), `libs/env/` (env vars), `libs/testserver/`, `libs/structs/structpath/` and `libs/structs/structvar/` (paths / typed config values), `acceptance/bin/` (acceptance test helpers), `internal/mocks/` (generated mocks). A function that duplicates an existing name and signature in the same package is a compile error waiting to happen; grep before you name.
 
 # Development Commands
 
@@ -73,7 +73,7 @@ GIT_EDITOR=true GIT_SEQUENCE_EDITOR=true VISUAL=true GIT_PAGER=cat git rebase or
 - `bundle/phases/` - High-level deployment phases
 
 **libs/** - Shared libraries and utilities
-- `libs/dyn/` - Dynamic configuration value manipulation
+- `libs/structs/structvar/` - Typed configuration values with references and locations
 - `libs/filer/` - File system abstraction (local, DBFS, workspace)
 - `libs/auth/` - Databricks authentication handling
 - `libs/sync/` - File synchronization between local and remote

Taskfile.yml

@@ -973,17 +973,20 @@ tasks:
       - "go run ./bundle/internal/schema ./bundle/internal/schema ./bundle/schema/jsonschema.json .codegen/cli.json"
 
   generate-validation:
-    desc: Generate enum and required field validation code
+    desc: Generate the walkers of the enum and required field validators
     sources:
       - "**/*.go"
       - exclude: "**/*_test.go"
       - go.mod
       - go.sum
       - "{{.EMBED_SOURCES}}"
     generates:
-      - bundle/internal/validation/generated/**/*.go
+      - bundle/config/validate/*_walk.go
+      - bundle/config/walk/*_walk.go
+      - bundle/config/mutator/resourcemutator/defaults_walk.go
+      - bundle/config/resources_iterate.go
     cmds:
-      - "sh -c 'go run ./bundle/internal/validation/. && gofmt -w -s ./bundle/internal/validation/generated'"
+      - "go run ./bundle/internal/validation/."
 
   generate-direct:
     desc: Generate direct engine config (apitypes + resources)

acceptance/bundle/override/cluster_id_clears_new_cluster_reference/databricks.yml

@@ -0,0 +1,36 @@
+bundle:
+  name: test-bundle
+  cluster_id: existing-cluster
+
+variables:
+  cluster:
+    type: complex
+    default:
+      spark_version: 15.4.x-scala2.12
+      node_type_id: i3.xlarge
+      num_workers: 1
+
+resources:
+  clusters:
+    source:
+      cluster_name: source
+      spark_version: 15.4.x-scala2.12
+      node_type_id: i3.xlarge
+      num_workers: 2
+  jobs:
+    job:
+      name: job
+      tasks:
+        - task_key: task
+          notebook_task:
+            notebook_path: /Workspace/notebook
+          # Replaced by bundle.cluster_id: the reference below new_cluster must go too.
+          new_cluster:
+            spark_version: 15.4.x-scala2.12
+            node_type_id: i3.xlarge
+            num_workers: ${resources.clusters.source.num_workers}
+        - task_key: task_var
+          notebook_task:
+            notebook_path: /Workspace/notebook
+          # A reference at new_cluster must not bring it back either.
+          new_cluster: ${var.cluster}

acceptance/bundle/override/cluster_id_clears_new_cluster_reference/out.test.toml

@@ -0,0 +1,2 @@
+Cloud = false
+EnvMatrix.DMS = ["", "true"]

acceptance/bundle/override/cluster_id_clears_new_cluster_reference/output.txt

@@ -0,0 +1,21 @@
+
+>>> [CLI] bundle validate -o json
+[
+  {
+    "task_key": "task",
+    "existing_cluster_id": "existing-cluster",
+    "new_cluster": null
+  },
+  {
+    "task_key": "task_var",
+    "existing_cluster_id": "existing-cluster",
+    "new_cluster": null
+  }
+]
+
+>>> [CLI] bundle deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Created clusters.source
+Created jobs.job
+Files: 1 uploaded, 0 deleted
+Resources: 2 created, 0 changed, 0 deleted, 0 unchanged

acceptance/bundle/override/cluster_id_clears_new_cluster_reference/script

@@ -0,0 +1,2 @@
+trace $CLI bundle validate -o json | jq '[.resources.jobs.job.tasks[] | {task_key, existing_cluster_id, new_cluster}]'
+trace $CLI bundle deploy

acceptance/bundle/override/preset_clears_forced_schedule_reference/databricks.yml

@@ -0,0 +1,39 @@
+bundle:
+  name: test-bundle
+
+resources:
+  quality_monitors:
+    src:
+      table_name: main.default.src
+      assets_dir: /Workspace/monitors/src
+      output_schema_name: main.default
+      inference_log:
+        granularities: [1 day]
+        model_id_col: model_id
+        prediction_col: prediction
+        problem_type: PROBLEM_TYPE_CLASSIFICATION
+        timestamp_col: timestamp
+      schedule:
+        quartz_cron_expression: "0 0 12 * * ?"
+        timezone_id: UTC
+    dst:
+      table_name: main.default.dst
+      assets_dir: /Workspace/monitors/dst
+      output_schema_name: main.default
+      inference_log:
+        granularities: [1 day]
+        model_id_col: model_id
+        prediction_col: prediction
+        problem_type: PROBLEM_TYPE_CLASSIFICATION
+        timestamp_col: timestamp
+      # Forced null: the target override replaces it with a reference.
+      schedule: null
+
+targets:
+  dev:
+    presets:
+      trigger_pause_status: PAUSED
+    resources:
+      quality_monitors:
+        dst:
+          schedule: ${resources.quality_monitors.src.schedule}

acceptance/bundle/override/preset_clears_forced_schedule_reference/out.test.toml

@@ -0,0 +1,2 @@
+Cloud = false
+EnvMatrix.DMS = ["", "true"]

acceptance/bundle/override/preset_clears_forced_schedule_reference/output.txt

@@ -0,0 +1,22 @@
+
+>>> [CLI] bundle deploy
+Warning: required field "quartz_cron_expression" is not set
+  at resources.quality_monitors.dst.schedule
+  in databricks.yml:39:21
+     databricks.yml:30:17
+
+Warning: required field "timezone_id" is not set
+  at resources.quality_monitors.dst.schedule
+  in databricks.yml:39:21
+     databricks.yml:30:17
+
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/dev/files...
+Created quality_monitors.dst
+Created quality_monitors.src
+Files: 1 uploaded, 0 deleted
+Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
+
+>>> print_requests.py //tables/main.default.dst/monitor
+{
+  "schedule": null
+}

acceptance/bundle/override/preset_clears_forced_schedule_reference/script

@@ -0,0 +1,2 @@
+trace $CLI bundle deploy
+trace print_requests.py //tables/main.default.dst/monitor | jq '{schedule: .body.schedule}'

acceptance/bundle/override/preset_clears_forced_schedule_reference/test.toml

@@ -0,0 +1 @@
+RecordRequests = true

acceptance/bundle/python/pipeline-development-deprecated/output.txt

@@ -1,22 +1,14 @@
 
 >>> uv run [UV_ARGS] -q [CLI] bundle validate
-Warning: The "development" property of pipelines is deprecated. Use "mode: development" in your databricks.yml instead. See https://docs.databricks.com/dev-tools/bundles/deployment-modes.
-  at resources.pipelines.with_development.development
-  in resources.py:7:1
-
 Name: my_project
 Target: dev
 Workspace:
   User: [USERNAME]
   Path: /Workspace/Users/[USERNAME]/.bundle/my_project/dev
 
-Found 1 warning
+Validation OK!
 
 >>> uv run [UV_ARGS] -q [CLI] bundle validate --output json
-Warning: The "development" property of pipelines is deprecated. Use "mode: development" in your databricks.yml instead. See https://docs.databricks.com/dev-tools/bundles/deployment-modes.
-  at resources.pipelines.with_development.development
-  in resources.py:7:1
-
 {
   "with_development": true,
   "without_development": true

acceptance/bundle/resource_deps/whole_map_reference_hides_entries/databricks.yml

@@ -0,0 +1,18 @@
+bundle:
+  name: test-bundle
+
+presets:
+  tags:
+    owner: ${resources.jobs.j.id}
+
+resources:
+  schemas:
+    s:
+      name: s
+      catalog_name: main
+      properties: {owner: fixed}
+  jobs:
+    j:
+      name: j
+      # A whole-map reference: the preset's tags added to the typed map stay hidden.
+      tags: ${resources.schemas.s.properties}

acceptance/bundle/resource_deps/whole_map_reference_hides_entries/out.test.toml

@@ -0,0 +1,2 @@
+Cloud = false
+EnvMatrix.DMS = ["", "true"]

acceptance/bundle/resource_deps/whole_map_reference_hides_entries/output.txt

@@ -0,0 +1,6 @@
+
+>>> [CLI] bundle plan
+create jobs.j
+create schemas.s
+
+Plan: 2 to add, 0 to change, 0 to delete, 0 unchanged

acceptance/bundle/resource_deps/whole_map_reference_hides_entries/script

@@ -0,0 +1 @@
+trace $CLI bundle plan

acceptance/bundle/summary/modified_status/output.txt

@@ -200,4 +200,4 @@ All files and directories at the following location will be deleted: /Workspace/
 Destroy: 4 deleted
 
 >>> [CLI] bundle summary -o json
-{}
+null

acceptance/bundle/validate/app_resource_duplicate_name_location/output.txt

@@ -2,7 +2,7 @@
 >>> [CLI] bundle validate
 Warning: invalid value "INVALID" for enum field. Valid values are [CAN_MANAGE CAN_MANAGE_RUN CAN_VIEW IS_OWNER]
   at resources.apps.app.resources[0].job.permission
-  in databricks.yml:13:25
+  in databricks.yml:17:25
 
 Name: test-bundle
 Target: default
@@ -15,7 +15,7 @@ Found 1 warning
 >>> [CLI] bundle validate -o json
 Warning: invalid value "INVALID" for enum field. Valid values are [CAN_MANAGE CAN_MANAGE_RUN CAN_VIEW IS_OWNER]
   at resources.apps.app.resources[0].job.permission
-  in databricks.yml:13:25
+  in databricks.yml:17:25
 
 [
   {

acceptance/bundle/validate/app_resource_duplicate_name_location/script

@@ -1,4 +1,4 @@
-# A diagnostic on a field of a duplicated app resource entry points at the first entry when the duplicate's value is the same.
+# A diagnostic on a field of a duplicated app resource entry points at the entry that is kept (the later one).
 errcode trace $CLI bundle validate
 
 # The entries are merged by name, the later one wins.

acceptance/bundle/validate/includes/clears_unforced_preset_values/output.txt

@@ -1,9 +1,13 @@
 
 >>> [CLI] bundle validate -o json
 {
-  "presets": {},
+  "presets": {
+    "artifacts_dynamic_version": true,
+    "jobs_max_concurrent_runs": 7,
+    "name_prefix": "base-"
+  },
   "job": {
-    "name": "job",
-    "max_concurrent_runs": 1
+    "name": "base-job",
+    "max_concurrent_runs": 7
   }
 }

acceptance/bundle/validate/includes/clears_unforced_preset_values/test.toml

@@ -0,0 +1 @@
+Badness = "An include cannot clear a preset with \"\", 0 or false: the base value stays (a written zero is not recorded without ForceSendFields)."

acceptance/bundle/validate/includes/merge_failure_short_circuits_diagnostics/output.txt

@@ -1,9 +1,13 @@
 
 >>> [CLI] bundle validate
+Warning: unknown field: unexpected_after_failure
+  at bundle
+  in b.yml:2:3
+
 Error: cannot merge map with string
 
 Name: test-bundle
 
-Found 1 error
+Found 1 error and 1 warning
 
 Exit code: 1

acceptance/bundle/validate/includes/merge_failure_short_circuits_diagnostics/script

@@ -1,2 +1,2 @@
-# Includes after one that fails to merge do not report their diagnostics.
+# Includes after one that fails to merge still report their diagnostics.
 errcode trace $CLI bundle validate

acceptance/bundle/validate/includes/merge_failure_short_circuits_diagnostics/test.toml

@@ -1 +1 @@
-Badness = "The merge error has no location or path, and diagnostics of includes loaded after the failed merge (the unknown field in b.yml) are not reported, although they do not depend on the merge."
+Badness = "The merge error has no location or path."

acceptance/bundle/validate/nulls/resources_container/output.txt

@@ -1,9 +1,9 @@
 
 >>> [CLI] bundle validate
-Error: expected a map at "resources", found nil
-
 Name: test-bundle
+Target: default
+Workspace:
+  User: [USERNAME]
+  Path: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default
 
-Found 1 error
-
-Exit code: 1
+Validation OK!

acceptance/bundle/validate/nulls/resources_container/script

@@ -1,2 +1,2 @@
-# resources: null is rejected.
+# resources: null is accepted.
 errcode trace $CLI bundle validate

acceptance/bundle/validate/references/resolved_invalid_map_value/output.txt

@@ -5,5 +5,6 @@ Warning: expected string, found map
   in databricks.yml:15:23
 
 {
+  "bad.key[0]": "",
   "good": "ok"
 }

acceptance/bundle/validate/references/resolved_invalid_map_value/script

@@ -1,2 +1,2 @@
-# A map value whose reference resolves to the wrong type is dropped, with a warning at its path.
+# A map value whose reference resolves to the wrong type becomes an empty string, with a warning at its path.
 trace $CLI bundle validate -o json | jq .resources.jobs.j.tags