Skip to content

feat(e2e): add end-to-end test job on Databricks serverless - #131

Open
tombonfert wants to merge 17 commits into
mainfrom
feature/e2e_tests_on_databricks
Open

feat(e2e): add end-to-end test job on Databricks serverless#131
tombonfert wants to merge 17 commits into
mainfrom
feature/e2e_tests_on_databricks

Conversation

@tombonfert

@tombonfert tombonfert commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds an end-to-end test suite that runs Impulse reports on Databricks serverless against real silver data and compares the gold tables with expected results. It covers Unity Catalog and Delta behaviour that the local tests don't reach, and gives us a baseline for testing larger features such as #129.

What's new

  • Bundle and job: databricks.yml and resources/e2e_job.yml define one serverless job: setup, then six tests in parallel, then teardown. Each test writes its own table prefix in a temporary schema e2e_<run_id>.
  • Tests (tests/e2e/e2e_*.py):
    • rpm_histogram: baseline report; a compute-only run writes nothing, and persisted reruns update the gold tables in place.
    • incremental: two silver loads in batches of 15; earlier rows stay untouched, with one MERGE per batch.
    • incremental_updates: a no-op rerun, a re-ingested container and changed bins each recompute only what changed.
    • container_filter: tag and metric filters select the same containers as plain SQL.
    • pipeline: container event, statistics, heatmap and a calculated channel that agrees with StatsAggregator.
    • events: point, sequence and time-window events; windows tile each container.
  • Checks: gold tables are compared with CSVs in tests/e2e/expected/, plus invariants that don't depend on them (the histogram adds up to the event durations, windows have no gaps or overlaps, rows of unchanged containers are untouched).
  • Running: make test-e2e locally; .github/workflows/e2e.yml runs it nightly and on demand.
  • Cleanup: teardown drops the schema only when all tests pass. Otherwise the run fails and the schema is kept for debugging, and setup drops this job's schemas older than one day.
  • Removes the databricks-labs-pytester dependency.

See tests/e2e/README.md for the job layout, input data and how to refresh expected files.

tombonfert added 16 commits August 1, 2026 08:55
@tombonfert
Add on-Databricks end-to-end test suite with Databricks Asset Bundle
9e2c6b8 
Runs the impulse report pipeline against a real workspace on serverless
compute, complementing the local-Spark unit suite. Deploys a serverless
job via a DAB that installs the impulse wheel + databricks-labs-pytester
and runs tests/e2e (pytest) in-job; a new e2e.yml workflow (nightly +
manual) drives bundle deploy/run with OAuth M2M auth. The workspace host
and credentials come from the environment (DATABRICKS_HOST + secrets),
so nothing sensitive is committed.

Co-authored-by: Isaac
@tombonfert
ci: temporarily allow serverless E2E job to fail while iterating
93e290e
@tombonfert
ci: rename default UC catalog for e2e tests to impulse_tests_catalog
e0c9a1d
@tombonfert
Merge remote-tracking branch 'origin/main' into feature/e2e_tests_on_…
998c04b 
…databricks
@tombonfert
chore: ignore local scripts directory
3938a72
@tombonfert
feat(e2e): switch to read-only source schema and add source-schema pa…
1e8c02e 
…rameter

- Replace the local CSV-loading `reporting_demo_dir` fixture with a `source_schema` fixture that points to a pre-populated, read-only silver schema passed via `--source-schema`.
- Add `source_schema` bundle variable and wire it through `resources/e2e_job.yml`, `tests/e2e/run_in_job.py`, and `tests/e2e/conftest.py`.
- Update the e2e test to use `report.run()` and assert idempotency across two runs.
- Trim verbose comments in `databricks.yml`, `Makefile`, `pyproject.toml`, `.github/workflows/e2e.yml`, and e2e resources/tests.
- Refresh `uv.lock` to include the `databricks-labs-pytester` e2e dependency tree.
@tombonfert
feat(e2e): replace pytester-based suite with explicit Databricks job …
1fc70cb 
…steps and golden-file validation

- Remove `databricks-labs-pytester` dependency and the `e2e` pytest marker.
- Replace `tests/e2e/run_in_job.py` and `tests/e2e/conftest.py` with explicit job tasks:
  `setup_schema.py`, `run_report.py`, `validate.py`, `teardown.py`, `fail_on_error.py`.
- Split `resources/e2e_job.yml` into setup → run_report → run_report_rerun → validate → teardown,
  asserting idempotency across two report runs and comparing gold tables to expected CSVs.
- Add `tests/e2e/expected/histogram_fact.csv` and `event_instance_fact.csv` as golden files.
- Change `source_schema` bundle default to `default`.
@tombonfert
chore(deps): remove databricks-labs-pytester and transitive e2e depen…
abbbb24 
…dencies from uv.lock
@tombonfert
feat(e2e): convert e2e job tasks from spark_python_task to notebook_task
faed048 
- Switch `resources/e2e_job.yml` tasks to `notebook_task` with `base_parameters` instead of `spark_python_task` CLI arguments.
- Convert `tests/e2e/*.py` scripts to Databricks notebook format with `dbutils.widgets` parameter access.
- Reorder `environment_key` and `disable_auto_optimization` fields for consistency in `e2e_job.yml`.
@tombonfert
feat(e2e): split e2e job into parallel test notebooks with shared uti…
135e166 
…lities and per-test expected files

- Replace the single `run_report`/`validate` flow with four parallel test notebooks:
  `e2e_rpm_histogram.py`, `e2e_incremental.py`, `e2e_pipeline.py`, `e2e_container_filter.py`.
- Add `tests/e2e/e2e_utils.py` with shared helpers for config, report setup, expected/actual comparison, and CSV upload.
- Update `resources/e2e_job.yml` to run tests in parallel after setup, with teardown only on success and `fail_on_error` on any failure.
- Move expected files under per-test directories (`expected/rpm_histogram/`, etc.) and add `expected/containers.csv` for source-data validation.
- Update `setup_schema.py` to validate source containers against `expected/containers.csv` and clean up stale schemas.
- Remove `run_report.py` and `validate.py`.
@tombonfert
docs(e2e): update README to reflect parallel test structure and add p…
aca13b0 
…ipeline expected files

- Refresh `tests/e2e/README.md` to describe the new parallel task layout, per-test table prefixes, shared `e2e_utils.py`, and instructions for adding tests.
- Add expected gold files for the `pipeline` test: `histogram2d_fact.csv` and `stats_aggregator_fact.csv`.
@tombonfert
feat(e2e): add incremental update, event, and calculated-channel tests
5b7bb2e 
- Add `e2e_incremental_updates.py` to verify incremental runs skip unchanged
  containers, recompute only affected containers on source changes, and
  recompute only changed aggregations when bins change.
- Add `e2e_events.py` covering points-in-time, sequence, and time-window events
  with point-value and window-statistics aggregations.
- Extend `e2e_pipeline.py` with a calculated channel and cross-check its
  persisted samples, scoped stats, and unscoped stats.
- Strengthen `e2e_rpm_histogram.py` with a compute-only run that writes no
  tables, and expand `e2e_container_filter.py` to use combined tag and metric
  filters.
- Add shared helpers in `e2e_utils.py` for silver copies, table versions, and
  event dimension lookups.
- Wire new `incremental_updates` and `events` tasks into `resources/e2e_job.yml`
  and update `tests/e2e/README.md`.
- Add expected gold files for the new `events` test and the updated `pipeline`
  test.
@tombonfert
feat(e2e): harden incremental update assertions and workflow cleanup
0bc290a 
- Strengthen `e2e_incremental_updates.py` to verify that re-ingesting one
  container leaves all other containers' rows unchanged (including `_created_at`).
- Switch `e2e_utils.py` to absolute tolerance only for DataFrame comparisons,
  avoiding false negatives on epoch timestamps, and catch all comparison errors.
- Remove temporary branch trigger and `continue-on-error` from `e2e.yml`,
  drop the fork-gate job, and configure concurrency so overlapping runs queue
  instead of cancelling the active Databricks job; cancel the job run when the
  workflow itself is cancelled.
- Update `tests/e2e/README.md` to describe the new incremental-update assertion.
@tombonfert
feat(e2e): strengthen assertions across incremental, event, and histo…
8b42b4d 
…gram tests

- Harden `e2e_events.py` to verify 5-minute windows tile each container without gaps or overlaps, and that `slow_down` point values have speed at most 100 and one value per channel.
- Strengthen `e2e_incremental_updates.py` to compare against full reference runs after each change and assert histogram bins still cover event durations.
- Tighten `e2e_incremental.py` to verify the second load leaves prior rows untouched and commits exactly one MERGE per batch of at most 15 containers.
- Update `e2e_rpm_histogram.py` to assert persisted runs rewrite the same Delta tables in place and that the histogram covers event durations.
- Add `assert_unchanged` and `assert_histogram_covers_events` helpers to `e2e_utils.py`.
- Fix `e2e_pipeline.py` calculated-channel mean to ignore NaN samples in the duration denominator.
- Refresh `tests/e2e/README.md` to describe the new assertions.
@tombonfert
fix(e2e): align calculated-channel mean with StatsAggregator NaN hand…
d8b08c6 
…ling

Update `e2e_pipeline.py` so NaN samples contribute zero to the weighted sum
while their duration still counts in the denominator, matching the
`StatsAggregator` behavior (`nansum(v * d) / nansum(d)`).
@tombonfert
fix(e2e): remove redundant fail_on_error task and simplify incrementa…
b235483 
…l update assertions

- Drop the `fail_on_error` task from `resources/e2e_job.yml`; teardown already ends as "upstream failed" when a test fails, which fails the job run and preserves the schema for debugging.
- Remove `tests/e2e/fail_on_error.py`.
- Simplify `e2e_incremental_updates.py` by removing the intermediate full-run reference after container re-ingestion; instead assert the recomputed histogram covers the container's new events, and keep only the final full-run comparison.
- Minor cleanups: extract `batches` variable in `e2e_incremental.py`, clarify comments in `e2e_utils.py` and `e2e_job.yml`, and update `tests/e2e/README.md` accordingly.
@tombonfert
tombonfert requested a review from a team as a code owner October 21, 2026 12:46
@codecov

codecov Bot commented Oct 21, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.30%. Comparing base (8fa2124) to head (18d8e83).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files 
@@           Coverage Diff           @@
##             main     #131   +/-   ##
=======================================
  Coverage   90.30%   90.30%           
=======================================
  Files          68       68           
  Lines        6068     6068           
  Branches      758      758           
=======================================
  Hits         5480     5480           
  Misses        466      466           
  Partials      122      122
Flag Coverage Δ
query_engine 86.80% <ø> (ø)
reporting 94.92% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tombonfert
feat(e2e): deploy isolated bundle per PR and clean up on close
18d8e83 
- Add `suffix` bundle variable to `databricks.yml` and use it in the workspace root path.
- Append `suffix` to the e2e job name in `resources/e2e_job.yml`.
- Trigger `e2e.yml` on PRs into main, nightly, and on demand; use per-PR bundle/job copies via `BUNDLE_VAR_suffix` so PRs run in parallel.
- Add a `destroy` job to remove the PR's bundle when the PR is closed.
- Skip fork PRs in CI; document mirroring them with `make fork-sync PR=<n>`.
- Update `tests/e2e/README.md` to describe the new CI behavior.

This branch was successfully deployed

1 active deployment
e2e — 18d8e834 Deployed Oct 21, 2026 by tombonfert via E2E on Databricks (serverless) #15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@tombonfert