Skip to content

Support best effort deployment history mode #7169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
shreyas-goenka wants to merge 2 commits into
base: main
Choose a base branch
from
+1,072 −343
Open

Support best effort deployment history mode #7169

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
83de547
Default deployment history to enforced file-backed recording
shreyas-goenka Oct 26, 2026
db9b734
Preserve file serial when history markers are unchanged
shreyas-goenka Oct 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 10 additions & 7 deletions acceptance/bin/print_state.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,23 +174,26 @@ def get_last_version_id(target):


def print_recorded_state(filename, target):
"""Print the state file with its resources filled in from the deployment metadata service.
"""Print DMS resources using the state file's header.

While recording, the file itself carries only the header - the service holds the resources - so
printing it raw would show an empty state and differ from the same test's non-recording run.
This checks history uploads even though resources.json remains authoritative.
Use --no-dms to inspect the file state instead.
"""
data = json.loads(open(filename).read())
# Recording stamps each resource payload with the deployment and version; drop it here so the
# printed state matches a non-recording run without every caller piping through nostamp.
data["state"] = scrub(get_recorded_state(target))

# The file's serial is a copy of the DMS version to ensure outdated resource state is not read.
# Rebuild in header order with the authoritative serial from the deployment.
# Pair DMS resources with the DMS version. The file serial can advance independently.
last_version_id = get_last_version_id(target)
serial = int(last_version_id) if last_version_id else 0

# Recording itself is not what these tests assert, so drop the feature that marks it.
features = {k: v for k, v in (data.get("features") or {}).items() if k != "deployment_history"}
features = {
k: v
for k, v in (data.get("features") or {}).items()
if k not in {"deployment_history", "deployment_history_best_effort"}
}

rebuilt = {}
for key in ("state_version", "cli_version", "lineage"):
Expand All @@ -213,7 +216,7 @@ def main():
"--no-dms",
action="store_true",
help="Print resources.json as-is without fetching state from DMS. "
"Use in bundle/dms tests to assert the tombstone content of the file.",
"Use in bundle/dms tests to assert the file state and its feature markers.",
)
args = parser.parse_args()

Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/deploy/readplan/lineage-mismatch/plan.dms.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,6 @@
"serial": 0,
"plan": {},
"features": {
"deployment_history": {}
"deployment_history_best_effort": {}

@shreyas-goenka shreyas-goenka Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll rename this in a followup.

}
}
3 changes: 0 additions & 3 deletions acceptance/bundle/dms/bind-unbind-unsupported/databricks.yml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,5 @@
bundle:
name: dms-bind-unbind-$UNIQUE_NAME
experimental:
deployment_history: true

@denik denik Oct 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how is deployment_history set in this test? It is set somehow since bind outputs "Error: bind is not supported for a bundle target that records deployment history" but I don't see where. Env var somewhere?

@shreyas-goenka shreyas-goenka Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test only runs for EnvMatrix.DMS = ["true"] so this bit was redundant.


resources:
jobs:
foo:
Expand Down
10 changes: 7 additions & 3 deletions acceptance/bundle/dms/bind-unbind-unsupported/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,17 @@
Error: bind is not supported for a bundle target that records deployment history


=== Deploy so the bundle records state, then unbind is refused too
=== Unbind is allowed for file-backed history state
>>> [CLI] bundle deploy
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-bind-unbind-[UNIQUE_NAME]/default/files...
Created jobs.foo
Files: 2 uploaded, 0 deleted
Resources: 1 created, 0 changed, 0 deleted, 0 unchanged

>>> musterr [CLI] bundle deployment unbind foo
Error: unbind is not supported for a bundle target that records deployment history
>>> [CLI] bundle deployment unbind foo

>>> jq {features, state} .databricks/bundle/default/resources.json
{
"features": null,
"state": {}
}
7 changes: 4 additions & 3 deletions acceptance/bundle/dms/bind-unbind-unsupported/script
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ job_id=$($CLI jobs create --json '{"name": "bind-target-'$UNIQUE_NAME'"}' | jq -
add_repl "$job_id" JOB_ID
trace musterr $CLI bundle deployment bind foo "$job_id"

# unbind refuses when it opens an existing recording state, so it needs a deployment first.
title "Deploy so the bundle records state, then unbind is refused too"
# File state remains writable even when it carries the history marker.
title "Unbind is allowed for file-backed history state"
trace $CLI bundle deploy
trace musterr $CLI bundle deployment unbind foo
trace $CLI bundle deployment unbind foo
trace jq '{features, state}' .databricks/bundle/default/resources.json
30 changes: 11 additions & 19 deletions acceptance/bundle/dms/declined-deploy/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
"lineage": "[UUID]",
"serial": 1,
"features": {
"deployment_history": {}
"deployment_history_best_effort": {}

@denik denik Oct 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need the feature anymore with direct being source of truth, right? Sure older clients will not write to DMS but that's defensible for beta. Advantage is that you can drop DMS without any back migration.

@shreyas-goenka shreyas-goenka Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed offline, we'll keep the state, but rename it to deployment_history_shadow in a followup PR.

}
}

Expand All @@ -79,18 +79,10 @@ Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
"method": "GET",
"path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]"
}
{
"method": "GET",
"path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/resources"
}
{
"method": "GET",
"path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]"
}
{
"method": "GET",
"path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/resources"
}

>>> print_state.py --no-dms
{
Expand All @@ -99,9 +91,17 @@ Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
"lineage": "[UUID]",
"serial": 1,
"features": {
"deployment_history": {}
"deployment_history_best_effort": {}
},
"state": {}
"state": {
"resources.schemas.foo": {
"__id__": "main.dms_declined_deploy_schema_[UNIQUE_NAME]",
"state": {
"catalog_name": "main",
"name": "dms_declined_deploy_schema_[UNIQUE_NAME]"
}
}
}
}

=== A destructive change without --auto-approve is declined: this console cannot prompt
Expand All @@ -124,10 +124,6 @@ Files: 2 uploaded, 0 deleted
"method": "GET",
"path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]"
}
{
"method": "GET",
"path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/resources"
}

=== The recorded resources are unchanged by the decline: the schema is still at version 1
>>> MSYS_NO_PATHCONV=1 [CLI] bundle-deployments list-resources deployments/[DEPLOYMENT_ID] -o json
Expand Down Expand Up @@ -163,10 +159,6 @@ false
"method": "GET",
"path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]"
}
{
"method": "GET",
"path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/resources"
}
{
"method": "POST",
"path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions",
Expand Down
52 changes: 48 additions & 4 deletions acceptance/bundle/dms/depends-on/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"plan_version": [PLAN_VERSION],
"cli_version": "[CLI_VERSION]",
"features": {
"deployment_history": {}
"deployment_history_best_effort": {}
}
}

Expand Down Expand Up @@ -61,7 +61,7 @@ Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
"lineage": "[UUID]",
"serial": 1,
"features": {
"deployment_history": {}
"deployment_history_best_effort": {}
}
}

Expand Down Expand Up @@ -91,9 +91,53 @@ Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged
"lineage": "[UUID]",
"serial": 1,
"features": {
"deployment_history": {}
"deployment_history_best_effort": {}
},
"state": {}
"state": {
"resources.jobs.child": {
"__id__": "[CHILD_ID]",
"state": {
"deployment": {
"deployment_id": "[DEPLOYMENT_ID]",
"kind": "BUNDLE",
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/dms-depends-on-[UNIQUE_NAME]/default/state/metadata.json",
"version_id": "1"
},
"description": "depends on [PARENT_ID]",
"edit_mode": "UI_LOCKED",
"format": "MULTI_TASK",
"max_concurrent_runs": 1,
"name": "child",
"queue": {
"enabled": true
}
},
"depends_on": [
{
"node": "resources.jobs.parent",
"label": "${resources.jobs.parent.id}"
}
]
},
"resources.jobs.parent": {
"__id__": "[PARENT_ID]",
"state": {
"deployment": {
"deployment_id": "[DEPLOYMENT_ID]",
"kind": "BUNDLE",
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/dms-depends-on-[UNIQUE_NAME]/default/state/metadata.json",
"version_id": "1"
},
"edit_mode": "UI_LOCKED",
"format": "MULTI_TASK",
"max_concurrent_runs": 1,
"name": "parent",
"queue": {
"enabled": true
}
}
}
}
}

>>> [CLI] bundle destroy --auto-approve
Expand Down
4 changes: 2 additions & 2 deletions acceptance/bundle/dms/deployment-metadata-change/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
"lineage": "[UUID]",
"serial": 1,
"features": {
"deployment_history": {}
"deployment_history_best_effort": {}
}
}

Expand Down Expand Up @@ -128,7 +128,7 @@ A common practice is to use a username or principal name in this path, i.e. use
"lineage": "[UUID]",
"serial": 1,
"features": {
"deployment_history": {}
"deployment_history_best_effort": {}
}
}

Expand Down
14 changes: 11 additions & 3 deletions acceptance/bundle/dms/empty-grants/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
"lineage": "[UUID]",
"serial": 2,
"features": {
"deployment_history": {}
"deployment_history_best_effort": {}
}
}

Expand All @@ -163,9 +163,17 @@ Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
"lineage": "[UUID]",
"serial": 2,
"features": {
"deployment_history": {}
"deployment_history_best_effort": {}
},
"state": {}
"state": {
"resources.schemas.foo": {
"__id__": "main.dms_empty_grants_[UNIQUE_NAME]",
"state": {
"catalog_name": "main",
"name": "dms_empty_grants_[UNIQUE_NAME]"
}
}
}
}

>>> [CLI] bundle destroy --auto-approve
Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/dms/existing-state/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,6 @@ Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
{
"state_version": 3,
"features": {
"deployment_history": {}
"deployment_history_best_effort": {}
}
}
2 changes: 1 addition & 1 deletion acceptance/bundle/dms/existing-state/script
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ musterr trace DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true $CLI bundle deploy

title "A leftover local WAL with the wrong lineage is still refused"
mkdir -p .databricks/bundle/default
printf '{"lineage":"stale","serial":99,"state_version":2,"features":{"deployment_history":{}}}\n' > .databricks/bundle/default/resources.json.wal
printf '{"lineage":"stale","serial":99,"state_version":2,"features":{"deployment_history_best_effort":{}}}\n' > .databricks/bundle/default/resources.json.wal
musterr trace DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true $CLI bundle deploy

title "Destroy clears the deployment. With nothing recorded remotely the bundle is treated as new, so recording can be enabled once the stale WAL is gone"
Expand Down
56 changes: 47 additions & 9 deletions acceptance/bundle/dms/failed-delete/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
"lineage": "[UUID]",
"serial": 1,
"features": {
"deployment_history": {}
"deployment_history_best_effort": {}
}
}

Expand All @@ -92,9 +92,28 @@ Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
"lineage": "[UUID]",
"serial": 1,
"features": {
"deployment_history": {}
"deployment_history_best_effort": {}
},
"state": {}
"state": {
"resources.jobs.stuck": {
"__id__": "[STUCK_ID]",
"state": {
"deployment": {
"deployment_id": "[DEPLOYMENT_ID]",
"kind": "BUNDLE",
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/dms-failed-delete-[UNIQUE_NAME]/default/state/metadata.json",
"version_id": "1"
},
"edit_mode": "UI_LOCKED",
"format": "MULTI_TASK",
"max_concurrent_runs": 1,
"name": "stuck",
"queue": {
"enabled": true
}
}
}
}
}

=== A resource that fails to delete is recorded as a failed operation carrying the error, so the history says why rather than showing the resource gone
Expand Down Expand Up @@ -176,22 +195,41 @@ API message: Fault injected by test.
}
]

=== The local state file records failed version 2
=== The local state file keeps the resource and its serial after the failed delete
>>> print_state.py --no-dms
{
"state_version": 3,
"cli_version": "[CLI_VERSION]",
"lineage": "[UUID]",
"serial": 2,
"serial": 1,
"features": {
"deployment_history": {}
"deployment_history_best_effort": {}
},
"state": {}
"state": {
"resources.jobs.stuck": {
"__id__": "[STUCK_ID]",
"state": {
"deployment": {
"deployment_id": "[DEPLOYMENT_ID]",
"kind": "BUNDLE",
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/dms-failed-delete-[UNIQUE_NAME]/default/state/metadata.json",
"version_id": "1"
},
"edit_mode": "UI_LOCKED",
"format": "MULTI_TASK",
"max_concurrent_runs": 1,
"name": "stuck",
"queue": {
"enabled": true
}
}
}
}
}

=== The remote checkpoint stays at serial 1 while DMS advances to version 2
>>> MSYS_NO_PATHCONV=1 [CLI] workspace export /Workspace/Users/[USERNAME]/.bundle/dms-failed-delete-[UNIQUE_NAME]/default/state/resources.json --format RAW
{"serial":1}

>>> print_state.py
{"serial":2}
>>> MSYS_NO_PATHCONV=1 [CLI] bundle-deployments get-deployment deployments/[DEPLOYMENT_ID] -o json
{"last_version_id":"2"}
Loading
Loading