Repository navigation
Wireless Coverage example + distributed h3_viewshed_towers (0.5.2) - #115
Merged
Merged
Conversation
The first step of the interactive 2-step wheel install uses --no-deps, so the [light_env*] extra on that line resolves nothing — it only expands and constraint-checks the extra's dep set, which can fail the force-reinstall. Make step 1 a bare "geobrix @ file://..." and keep the extra on the plain step 2 (which installs the extra's deps). Covers the installation.mdx Serverless tip, the h3-rasterize example notebook, and the notebooks rule that documents the incantation. Co-authored-by: Isaac <no-reply@databricks.com>
The H3-native sibling of rst_viewshed_towers: a per-tower line-of-sight viewshed fanned one-tower-per-task over the shipped h3_los_visible core, the mechanism that lets the Wireless Coverage siting scale across Spark. Non-columnar DataFrame function (towers + surface/ground maps in, visible cells out), so no SQL binding — matching rst_viewshed_towers. The fan-out is mapInPandas over a repartition by tower (no .rdd/sparkContext), so the single path runs on Serverless Connect as well as local tests. H3 cells flow as BIGINT, converted to/from the h3 string form internally. Co-authored-by: Isaac <no-reply@databricks.com>
Adds the Databricks Asset Bundle scaffold and shared configuration module for the Wireless Coverage Lakeflow Spark Declarative Pipeline, so that Tasks 3-7 have stable import interfaces (cfg/paths/register_gbx) and a validated bundle to deploy against without re-doing structural plumbing. - databricks.yml: serverless pipeline + 4-task job (land → pipeline → siting → validate), variable defaults, wc_env environment spec; mirrors vapor-eyes/lakeflow conventions (include glob, block-form YAML, pipeline environment comment explaining single-pass extras rationale). - databricks.override.yml.example: workspace profile + laz_dir/water_mask_dir override hints for sharing the notebook series' staged LiDAR. - transformations/_config.py: cfg()/paths()/register_gbx(); all keys and defaults from the task-2 spec, register_gbx imports deferred to call-time. - tests/conftest.py + tests/test_config.py: 4 pure-host pytest cases (FakeSpark, defaults, derived paths, shared-volume laz_dir override, demo AOI). - land/land.py, siting/siting.py, validate/validate_wc_lf.py: minimal placeholder stubs required because `databricks bundle validate` rejects spark_python_task and notebook_task paths that do not exist locally; later tasks replace the stubs with real implementations. YAML reconciliation vs vapor-eyes (recorded): - Added mode: development/production to targets (brief requirement). - notebook_path uses ./validate/validate_wc_lf.py (with .py extension) rather than extension-less; DAB CLI requires an extension on local notebook paths and strips it on workspace deploy (Databricks notebook convention). - No warehouse_id variable (wireless coverage has no AI/BI dashboard task). - No stac extra in wheel dependency (not needed for this pipeline). - environment_version "2" per Databricks job environment spec schema version. Co-authored-by: Isaac <no-reply@databricks.com>
mode: development/production caused unwanted [dev <user>] prefixing on all deployed resources in a demo/showcase bundle; omitting mode (vapor-eyes pattern) keeps resource names clean. environment_version "2" was an unverified guess; "5" is the known-good value on this workspace (matches vapor-eyes which uses the same light_env6 extra and works). Co-authored-by: Isaac <no-reply@databricks.com>
First DAB job task: downloads USGS 3DEP LiDAR (.laz) and Overture base/water GeoParquet into the pipeline input Volume, skipping both if the outputs are already present (idempotent re-run). CLI parses --catalog/--schema/--volume/--full-aoi/--laz-dir/--water-mask-dir; path derivation mirrors transformations/_config.paths() exactly so the land task and the rest of the pipeline agree on input locations. Exposes _download_lidar as a monkeypatch seam for offline tests. Co-authored-by: Isaac <no-reply@databricks.com>
Add bronze_lidar.py — a modern Lakeflow (@dp.table) streaming table that uses cloudFiles/binaryFile Auto Loader to inventory staged USGS 3DEP .laz EPT nodes. Task 5 (silver) reads laz_inventory to wire the dependency edge without a hard import. Uses pathGlobFilter=*.laz and a per-pipeline schemaLocation under paths()["schema_loc"]. Co-authored-by: Isaac <no-reply@databricks.com>
The SILVER layer turns staged LiDAR into the surface rasters the gold H3 grids and tower-viewshed siting depend on. Reproducing the shipped Wireless Coverage notebooks (01 binning, 02a TIN DTM) as four materialized views lets the pipeline recompute surfaces declaratively, incrementally, and with lineage, instead of re-running notebooks by hand. register_gbx(spark) is called inside each MV body (never at import) and every upstream read goes through spark.read.table(...) so the pipeline sees the dependency edges (laz_inventory -> pts -> dsm/dtm -> chm). The notebooks' hard-coded constants (X0..Y1, TILE_M, TPX, SRID, TIN_MAX_PTS) and the DEMO window are replaced by cfg()/paths() values, so full-vs-demo is the full_aoi config toggle rather than a code or table-name change. Co-authored-by: Isaac <no-reply@databricks.com>
Ports notebook 03a_h3_gridding's H3-gridding transforms into the declarative pipeline as three materialized views, so Part-4 tower analysis consumes H3-indexed elevation/canopy tables with one stable join key instead of raw raster surfaces. DEM/DSM reproduce _surface_to_h3_cells (rst_transform 3857->4326, rst_clip to the Overture-derived land mask, rst_isoband, h3_try_coverash3). CHM uses the gbx_rst_h3_rastertogridmax UDTF for max canopy per cell, clamps artifacts at 350 m, gap-fills with h3_cellfill (k=1 IDW), and re-applies the land mask. All three add parent_cellid / parent_cellid_res (h3_toparent at join_res) and an expect_or_fail guard. register_gbx runs inside each body; params and the water mask come from _config cfg/paths. config_nb helpers are reproduced inline because the pipeline cannot %run-import them. Co-authored-by: Isaac <no-reply@databricks.com>
Port Part-4 (04_tower_viewsheds) into the DAB job's post-pipeline task. The expensive exact per-tower viewshed moves off the driver onto the distributed pygx.h3_viewshed_towers primitive (mapInPandas, one tower per task), so siting scales to the full San Francisco AOI; the cheap coarse quick-pass stays a driver screen by design. Writes six regular tables the workshop consumes (surf/grnd r13 bins, quick-pass scores, ranked candidate sites, and res-12 coverage depth + top-10). Vendored policy helpers (quick_survivors / required_mast / rank_candidates) are pure and unit-tested without a cluster; spark/gbx imports are deferred so the module loads offline. Co-authored-by: Isaac <no-reply@databricks.com>
Replace the Task-2 stub .py with the real validate_wc_lf.ipynb: checks all twelve terminal tables for existence, non-empty count, and required columns (H3 tables: parent_cellid / parent_cellid_res join key); validates coverage depth >= 1; renders static H3 maps via vizx; raises loudly on any failure so the DAB job validate task surfaces a clear error. Update databricks.yml to point to the .ipynb (bundle validate OK). Co-authored-by: Isaac <no-reply@databricks.com>
Add a plot_raster call on a wc_surface_chm tile to the preview cell so an actual raster surface is visualized (not only the H3-aggregated form). Fix execution_count from 0 to null on all code cells (unexecuted-notebook nbformat convention). Co-authored-by: Isaac <no-reply@databricks.com>
Rename the three series resolution identifiers in nb03b and nb04 to match the Lakeflow pipeline (LDP) config keys so both codebases share one vocabulary: RES_FINE → SURFACE_BIN_RES (H3 bin resolution, finest level) FINE_RES → VIEWSHED_RES (viewshed detail resolution) QUICK_RES → QUICKPASS_RES (coarse quick-pass resolution) TOWER_RES unchanged (already matches LDP) Renames applied whole-word only to cell['source']; outputs untouched. Also adds a markdown note to config_nb on aligning LAZ_DIR/OUT_DIR with the LDP Volume path to avoid a duplicate LiDAR download. Co-authored-by: Isaac <no-reply@databricks.com>
The Wireless Coverage Lakeflow tab was a stub ("planned for a future
release"). Replace it with real user-facing docs sourced from the
actual bundle code: how-it-works six-stage narrative, full table
catalog (bronze/silver/gold/siting layers), deploy/run commands,
configuration table including the shared-Volume laz_dir override,
prerequisites, and the validation notebook reference.
Add notebooks/examples/wireless-coverage/lakeflow/README.md for
GitHub readers, following the vapor-eyes lakeflow README structure
with the same content plus the full files table and YAML override
example.
Co-authored-by: Isaac <no-reply@databricks.com>
The job tasks (land/siting/validate) pin geobrix[light_env6], which the pyproject defines as the protobuf-6 regime for Serverless environment 6 (idna 3.11, typing_extensions 4.13+) — the same environment the notebook series runs on. The DAB skeleton inherited environment_version "5" from the vapor-eyes bundle (a latent mismatch there), which runs those env-6 deps on an env-5 base and risks the "core package versions changed" startup failure. Align the base to the extra: env 6. Co-authored-by: Isaac <no-reply@databricks.com>
The full-SF LiDAR is already staged at the series path, so point laz_dir there in the override template — the land task then finds it present and skips the ~830 M-point re-download. Keeps the LDP and the notebook series sharing one staged copy by default. Co-authored-by: Isaac <no-reply@databricks.com>
SDP flow analysis runs each materialized view against empty upstreams; the CHM MV's cell gap-fill helper did an eager .first()["p"] on an empty aggregate, returning None and crashing flow analysis (TypeError: NoneType not subscriptable). The lazy DEM/DSM MVs analyzed fine. Drop the gap-fill from wc_h3_chm_res10 (max canopy per cell, clamped, land- masked remains) and delete the now-unused helper. land_cells now seeds from spark.range(1) instead of createDataFrame so the MV stays lazy. The land-mask helper (_land_wkb) is unchanged: its collect reads a staged external file, which is SDP-safe (DEM/DSM call it and analyze fine). Eager driver actions on upstream data do not belong in a declarative materialized-view body. Co-authored-by: Isaac <no-reply@databricks.com>
wc_surface_chm joined DSM x DTM then mapped rst_chm; the join is small so Spark broadcast it (no (tx,ty) shuffle), piling every tile into one task that held all tiles' raster bytes and ran every warp -> light Serverless worker OOM (seen at demo scale: 24 tiles). repartition(512, tx, ty) forces one tile per task (bounded ~tens of MB each) and scales to full SF without a larger worker. Mirrors the gold step's per-tile fan-out. Co-authored-by: Isaac <no-reply@databricks.com>
The Lakeflow SDP compute is the DLT runtime channel = DBR 18.3.x (protobuf 5), not Serverless env 6 / DBR 19. Match the extra to the actual runtime: pipeline -> [light_dbr18]; job tasks -> environment_version "5" (DBR 18 <-> Env 5) + [light_env5] (= light_dbr18, protobuf-5). Supersedes the earlier env "6"/[light_env6] (protobuf-6) choice, which pinned env-6 deps on the protobuf-5 base. Co-authored-by: Isaac <no-reply@databricks.com>
The DBR-18/Env-5 extra choice is point-in-time: Lakeflow pins compute by a runtime channel (DBR 18.3 as of this example). Add a caveat in the DAB comments, the lakeflow README, and the flavor-tab docs telling users to check the current Lakeflow runtime channel at deploy time and switch the GeoBrix extra to match (e.g. DBR 19 -> light_dbr19/light_env6). Also fixes the stale [light_env6] references in the lakeflow docs to [light_dbr18]. Co-authored-by: Isaac <no-reply@databricks.com>
The bundle defaults to Serverless (pipeline + job tasks) and that runs the whole medallion. Document -- in the DAB pipeline comment, the README, and the flavor-tab docs -- that classic compute (serverless:false + a sized cluster spec) is a supported alternative for heavier runs (notably full San Francisco) to get dedicated memory and throughput. Co-authored-by: Isaac <no-reply@databricks.com>
A Databricks spark_python_task runs the file via exec(compile(...)) with no __file__, so the sibling-import trick (os.path.abspath(__file__) -> add lakeflow/ to sys.path -> import transformations._config) raised NameError at runtime and failed the Serverless siting task. Inline register_gbx's body and the ten _config.cfg() algorithm defaults (a job task never carries the pipeline's spark.conf block, so cfg already returned exactly these) to drop the path dependency entirely. Pure policy helpers and their tests are unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
wc_h3_dsm_res10 spills ~1 TB to produce ~1864 cells; this TEMPORARY diagnostic MV (_diag_dsm_amp) measures the gold-chain amplification to find the cause. It reproduces the exact DSM reproject -> clip -> isoband stages, then counts H3 cells per isoband polygon (array size, no explode) plus vertex count and polygon area per band -- so a huge max area or cells-per-polygon flags a degenerate reprojected extent or an over-dense band. TEMPORARY -- to be removed after reading the numbers. Not wired to any downstream consumer and carries no parent-key expectation. Co-authored-by: Isaac <no-reply@databricks.com>
The _diag_dsm_amp probe showed the DSM gold chain spilled ~1 TB on geometry VOLUME, not an h3-cover explosion: rst_isoband on the noisy DSM yields ~400K polygons with up to ~471K vertices each, for only ~412K cells. The dense 1 m pixel-staircase vertex rings, not the cell count, drove the spill. Add st_simplify in _surface_to_h3_cells between rst_isoband and h3_try_coverash3 at _SIMPLIFY_TOL_DEG = 3e-5 deg (~3 m) -- far below the ~65 m res-10 H3 cell edge, so the covered-cell set is unchanged while the staircase vertices collapse. Guard with st_makevalid before simplify (simplify can self-intersect these complex polygons) and drop any still-invalid geom via st_isvalid so h3_try_coverash3 never chokes. Applies to dem/dsm (both go through _surface_to_h3_cells; near no-op for the lighter dem). Also removes the temporary _diag_dsm_amp diagnostic MV, which has served its purpose. Co-authored-by: Isaac <no-reply@databricks.com>
The prior round wrapped the isoband simplify in product st_makevalid, but st_makevalid is UNRESOLVED_ROUTINE on the DBR 18.3 serverless runtime. Product st_simplify itself DOES resolve there (only st_makevalid was missing) and st_makevalid is not actually needed: st_simplify can occasionally self-intersect, but h3_try_coverash3 is the try_ variant (NULL on a bad geom) and the existing cellid-not-null filter drops it, so a rare bad polygon just loses its tiny cell contribution. Switch _surface_to_h3_cells to st_asbinary(st_simplify(st_geomfromwkb(geom_wkb), 3e-5)) -- product functions only, no gbx_st_* and no makevalid/isvalid. TOL unchanged (~3 m << the ~65 m res-10 cell edge, so covered cells are unchanged). Co-authored-by: Isaac <no-reply@databricks.com>
Product st_simplify is not topology-preserving: on the noisy isoband polygons it self-intersected ~8% of them, so h3_try_coverash3 returned NULL and the cellid-not-null filter dropped those cells (1864 -> 1715, ~8% loss). Switch _surface_to_h3_cells to geobrix's gbx_st_simplifypreservetopology (shapely, preserve_topology=True) -- the sanctioned product-not-able fallback -- which keeps the geometry valid and recovers the lost cells. TOL unchanged (3e-5 deg). Register pyvx (vx.register) in _config.register_gbx so the UDF is available, and in siting.py's inlined register block to keep the two copies in sync. No product st_* and no makevalid/isvalid remain in gold_h3_grids.py. Co-authored-by: Isaac <no-reply@databricks.com>
The user wants to see how much spatial info the isoband simplify drops, but the exact (no-simplify) 1864-cell DSM grid was overwritten by the simplified build and MVs are not time-travelable. Add a simplify_geom param to _surface_to_h3_cells (default True, so the 3 real gold MVs are unchanged) and a TEMPORARY _cmp_dsm_exact MV that re-materializes the DSM H3 grid with simplify_geom=False -- an exact baseline whose schema matches wc_h3_dsm_res10. Only _cmp_dsm_exact will be refreshed (the simplify path in wc_h3_dsm_res10 that OOMs is not exercised). TEMPORARY -- to be removed after the comparison. Co-authored-by: Isaac <no-reply@databricks.com>
The full-window exact (no-simplify) re-materialization OOM'd the driver on the ~400K isoband polygons (up to ~471K vertices each), so the simplify-vs-exact comparison has to run on a small tile window where the no-simplify branch completes. Add _CMP_WIN_TX/_CMP_WIN_TY and two TEMP MVs that filter wc_surface_dsm to that window: _cmp_dsm_exact (simplify_geom=False) and _cmp_dsm_simp (simplify_geom=True) -- apples-to-apples on the same cells. Also revert the simplify in _surface_to_h3_cells back to product st_simplify: the geobrix topology-preserving fallback (a shapely UDF) OOMs on the 471K-vertex polygons and is dead. The 3 real gold MVs are unchanged (still default simplify_geom=True -> product st_simplify). These _cmp_* MVs are TEMPORARY. Co-authored-by: Isaac <no-reply@databricks.com>
Currently the isoband simplification is hardcoded to the product st_simplify path (default behavior, essential for full-scale SF performance). Expose a user-configurable simplify=true/false option so users can opt into the exact (unsimplified) baseline for comparison or other analysis. When True (default): apply st_simplify for ~3x density reduction with no coverage loss. When False: exact isobands with potential band overlap but heavy/slow at full scale. Wire the config through _config.py (default True), dem/dsm MVs in gold_h3_grids, databricks.yml variables/configuration, and user-facing docs (README + mdx). CHM unaffected (no isoband step). Temp comparison MVs unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
Add a user-configurable SIMPLIFY parameter (default True) to the Wireless Coverage notebook series for controlling isoband simplification. When True, apply product st_simplify before H3 cover for fast, seam-sharpened gridding without coverage loss (needed for full-SF performance). When False, users can opt for exact isobands but accept heavier/slower performance at scale. - config_nb: add SIMPLIFY parameter and simplify=True to _surface_to_h3_cells - 03a_h3_gridding: pass simplify=SIMPLIFY to all isoband calls; update narrative to document the step and its performance implications Co-authored-by: Isaac <no-reply@databricks.com>
The windowed _cmp_dsm_exact/_cmp_dsm_simp MVs (and _CMP_WIN_* window constants) were throwaway scaffolding for measuring how much spatial information the isoband simplify drops. The comparison is done: product st_simplify removes zero cells (only trims redundant band-overlap memberships at seams), so the real gold MVs keep simplify on by default. Drop the scaffolding now that it has served its purpose. Co-authored-by: Isaac <no-reply@databricks.com>
At full-SF scale (336 tiles) the CHM gold MV's repartition(64, "tx") grouped a whole tx-strip (~16 tiles) into a single task, so the rst_h3_rastertogridmax Python UDTF loaded ~16 rasters per worker and OOM'd (UDF_PYSPARK_ERROR.OOM). The crashing worker took down shared executors, which surfaced as collateral executor-loss on the concurrent DEM flow. Fan out by (tx, ty) so each task processes one tile, matching the DEM/DSM path. Demo scale was unaffected (few tiles per strip). Co-authored-by: Isaac <no-reply@databricks.com>
rst_h3_rastertogridmax loads a tile's full band into a float64 numpy array per call (gridagg.py), so a dense full-SF 1024px CHM tile exhausts the ~1 GB Serverless Python worker and OOMs deterministically -- even one tile per task. Sparse GGP demo tiles slipped under the limit, hiding it at demo scale. Window each tile into fixed 256px sub-tiles with rst_retile and chain the LATERAL so the UDTF sees one small, constant sub-tile at a time; peak per-task memory is now bounded regardless of AOI size or pixel density. Max-per-cell is associative, so the result is identical to whole-tile processing. Co-authored-by: Isaac <no-reply@databricks.com>
A star cannot be a table-valued-function argument, so struct(t.*) inline in gbx_rst_h3_rastertogridmax(...) failed analysis (INVALID_USAGE_OF_STAR_OR_REGEX). Re-nest the flat rst_retile output (V2_TILE_SCHEMA columns) into a single tile struct in a separate step, then pass that named column to the H3 UDTF. Same bounded-memory sub-tile path; fixes the SQL analysis. Co-authored-by: Isaac <no-reply@databricks.com>
Root cause of the full-SF CHM OOM was not per-tile memory but that the repartition(512, tx, ty) fan-out was silently undone: the silver surfaces are only a few hundred rows (each an inlined multi-MB raster), so AQE judged the shuffle tiny by row count and coalesced it back to a few partitions, piling dozens of full-tile bands onto one worker. That is why demo scale (few tiles) never OOM'd and why the earlier rst_retile attempt made it worse (more rows per coalesced task). Set spark.sql.adaptive.coalescePartitions.enabled=false on the pipeline so the per-tile fan-out holds, and revert the retile workaround to the clean single-UDTF CHM gridding. Co-authored-by: Isaac <no-reply@databricks.com>
The full-SF CHM OOM root cause: the optimizer's InferFiltersFromGenerate rule pushed the expensive LATERAL raster->H3 UDTF generate down into the scan task, BEFORE the repartition(512, tx, ty) -- so the per-tile fan-out never isolated the UDTF and every tile's full band piled into the scan stage, OOMing the Python worker at full-SF density. (This is why demo scale never showed it and why plain repartition did not help.) Exclude that optimizer rule so the UDTF runs AFTER the repartition, one tile per task; keep AQE coalesce off so the fanned-out shuffle is not merged back. Ref: Databricks ES-2236400. Co-authored-by: Isaac <no-reply@databricks.com>
CHM reprojected (rst_transform) inside the pre-repartition select, so the optimizer ran it in the scan stage -- many tiles reprojected per task -> OOM -- unlike the DEM/DSM path which repartitions the raw tiles first. Repartition the raw CHM tiles before rst_transform so the reprojection runs post-shuffle, one tile per task, matching DEM/DSM. Combined with the InferFiltersFromGenerate exclusion (UDTF generate) this keeps every heavy Python op after the fan-out. Also cap pipelines.numUpdateRetryAttempts to 1 so a failed update fails fast instead of burning retry passes. Co-authored-by: Isaac <no-reply@databricks.com>
numUpdateRetryAttempts caps only whole-update retries; a failing flow was still retried per pipelines.maxFlowRetryAttempts (default) and then fell back to legacy execution mode, so a failed flow ran several times. Set maxFlowRetryAttempts=0 so a flow fails on first attempt -- faster, clearer iteration without burning repeated passes. Co-authored-by: Isaac <no-reply@databricks.com>
The Delaunay barycentric path wrote interpolated zvals into in-hull cells with no finiteness check, unlike the sibling IDW paths. Ill-conditioned sliver triangles yield a non-finite barycentric transform -> NaN zvals, which land as genuine NaN pixels (distinct from the no_data sentinel) and crash downstream GDAL warp/isoband with a native SIGABRT (observed as the full-SF DEM H3 gridding failing while the binned DSM path succeeded). Keep non-finite cells as the configured no_data (no hard-coded sentinel), mirroring the IDW finiteness guard. Add a regression test asserting the DTM never emits NaN/Inf. Co-authored-by: Isaac <no-reply@databricks.com>
Isolated the full-SF DEM SIGABRT (code 134) by probing each op: every chain that included rst_clip crashed natively; transform+isoband without clip ran clean. rst_clip rasterizes the land cutline onto the raster grid and aborts on the Delaunay-TIN DTM's grid at full-SF scale (the binned DSM clips fine). Add a clip toggle to _surface_to_h3_cells and have the DEM MV skip the raster clip, masking land in H3 space via an inner-join instead -- the same approach the CHM MV already uses, and equivalent output since the TIN leaves water as NoData out-of-hull. DSM keeps its working rst_clip. Co-authored-by: Isaac <no-reply@databricks.com>
Isolation showed transform+isoband runs clean on the TIN DTM, but adding any third fused native op (the GDAL land-clip OR the H3 cover) SIGABRTs on the TIN's messy isoband geometry at full-SF scale; the binned DSM is clean and unaffected. Split the DEM into three materialized stages -- wc_dem_clipped (transform+clip), wc_dem_isobands (isoband+simplify), wc_h3_dem_res10 (cover) -- so a Delta write/read boundary sits between the native ops and each runs in its own task. Keeps clip (masks water, cuts polygons) and simplify (de-densifies pixel-staircase rings, cell-set unchanged). Interims are restartable and inspectable. DSM stays a single MV (does not hit the fused-native crash). Co-authored-by: Isaac <no-reply@databricks.com>
Staging isolated the DEM cover failure to h3_try_coverash3 raising 'SafeInt overflowed / div-by-zero' -- the materialized isobands table showed 6 of 597007 polygons invalid. st_simplify is non-topology- preserving and self-intersected them; H3's polyfill overflows on a self-intersecting ring and its 'try' does not catch that internal math error. Repair with gbx_st_makevalid right after simplify (cheap now that the geoms are de-densified) so the stored isobands are all valid and the cover stage is safe. Only in the simplify branch. Co-authored-by: Isaac <no-reply@databricks.com>
st_simplify can self-intersect isobands; h3_try_coverash3 SafeInt-overflows on self-intersecting rings (uncaught). Apply gbx_st_makevalid after simplify to repair topology. Also fixes syntax error: closes outer assignment before if. Co-authored-by: Isaac <no-reply@databricks.com>
LidarDownloader.download(spark=...) returns a LAZY mapInPandas DataFrame whose per-node .laz writes are a side effect of execution. land.py's _download_lidar discarded the return without an action, so the mapInPandas never ran and the land task silently SUCCEEDED while staging zero LAZ -- a fresh deploy (the real CARTO path) got an empty input dir. Trigger an action (df.count()) so the LAZ actually stages. Caught by running the download task against a throwaway fresh dir (the full-job land skips the download because the LAZ is pre-staged). Co-authored-by: Isaac <no-reply@databricks.com>
The land task's stage_water (and the pipeline's _land_wkb read of real Overture water) use OvertureClient, which needs the stac + overture geobrix extras -- but both LDP envs declared only [light_*,vizx]. So stage_water's discover/download under-functioned and silently staged no water parquet, leaving the land mask effectively the full AOI box. Add stac,overture to the pipeline and job environments (single wheel[extras] pip resolution). Caught while verifying the download on a fresh dir. Co-authored-by: Isaac <no-reply@databricks.com>
The v0.5.2 release notes referenced the Orthomosaic series but had no Wireless Coverage callout, and the examples overview under-described the example (surfaces only, omitting tower siting / viewshed / the Lakeflow flavor). Add a dedicated release-notes subsection beside the orthomosaic mention, expand the overview row to the full LiDAR -> H3 -> tower-siting scope, and cross-link the series README to the Lakeflow SDP flavor README. Co-authored-by: Isaac <no-reply@databricks.com>
stage_water() inlined the OvertureClient discover/filter/download, so it had no offline test seam while stage_lidar() did -- the QC test-completeness check flagged the asymmetry. Extract a _download_water() seam mirroring _download_lidar(), and add stage_water skip/download unit tests that monkeypatch the seam (no network, spark=None). Co-authored-by: Isaac <no-reply@databricks.com>
sfm.py and the vizx pose/flythrough renderers created scratch via raw tempfile.mkdtemp/NamedTemporaryFile, which the centralized-primitives QC check flags at fail level -- all src/ scratch must route through the canonical pyrx.core.local_temp helpers so the temp root is relocatable via GBX_LOCAL_TEMP_DIR. Swap to new_local_temp_dir / new_local_temp_file; the default root stays tempfile.gettempdir() (/tmp), preserving the FUSE-seekable local-disk MP4 behavior. Affected light-tier suites green (sfm, pointcloud_flythrough/poses, show_flythrough). Co-authored-by: Isaac <no-reply@databricks.com>
The notebooks tab lacked a table-of-tables and had no visual proof of a full-San-Francisco run. Add a per-notebook table catalog with a Lakeflow (LDP) flavor column (Shared / Notebook-only / Lakeflow-only), and embed three executed full-SF renders (DEM-vs-DSM, CHM pixel-vs-H3, top-10 tower viewsheds) in the docs page and the WC example README so the payoff of a full-scale run is visible without opening a notebook. Co-authored-by: Isaac <no-reply@databricks.com>
config_nb now registers pyvx (so gbx_st_makevalid resolves) and provides a DEMO-aware show_cell_map helper: a single Golden Gate Park map at DEMO=True, an SF overview plus GGP detail at DEMO=False, so full-scale runs stay legible instead of rendering an unreadable all-SF res-10 grid. nb4's four maps are split into two side-by-side cells (A|B candidates/survivors, C|D coverage/top-10) -- one figure per cell, which also lets the executed outputs be captured -- with a full-SF else-path overlaying sited towers in red and zooming each detail window. Every notebook links to the notebooks-flavor docs tab; nb4 ends with SHOW TABLES, a land-cover "Coming Soon" pointer, and a land-mask caveat. 02b gains a per-tile repartition fan-out so the morphological-opening DTM no longer OOMs at full SF. DEMO=True outputs refreshed via reuse runs; American spelling throughout. Co-authored-by: Isaac <no-reply@databricks.com>
Align the Lakeflow siting comments and the validate notebook with the American spelling used across the rest of the series (neighborhood, centers) so the flavor pair reads consistently. Co-authored-by: Isaac <no-reply@databricks.com>
At DEMO=True show_cell_map built its own axes and passed it to plot_static, which skips _fix_colorbar_position (that pin only runs when plot_static owns the figure). The colorbar then spanned the full square axes while the equal-aspect map realized a shorter box, so the legend overran the plot on the DEM and CHM maps. Let plot_static create the figure (no ax=) for the single-map case so the colorbar is pinned to the map box, matching the other, already-aligned plots. Outputs refreshed from a DEMO=True reuse run. Co-authored-by: Isaac <no-reply@databricks.com>
Isaac Review on the WC example surfaced seven issues; this resolves them:
- CHM (03a): the gap-filled map rendered the raw wc_h3_chm table (artifacts
to ~1250 m) under a "gap-filled" caption. Now 03a persists the clean
chm_filled_land (clamped <=350 m, cellfill gap-filled, land-masked) AS
wc_h3_chm, and the map renders that table -- so the output table matches
the land-masked DEM/DSM tables and its own caption (verified: demo table
1181 cells, max_chm_z 2.6-337.1 m). The pre-clamp raw grid stays an
in-notebook intermediate.
- Docs catalog (mdx): the duplicate "## Table catalog" headings collided on
slug, so nb4's #table-catalog deep link hit the lakeflow heading. The
notebooks heading now carries an explicit {#table-catalog-notebooks} id
and nb4 links to it; the lakeflow heading keeps #table-catalog. The CHM
catalog row no longer calls the table "land-masked" ambiguously.
- Labels (04): the full-SF coverage map omitted res=, mislabeling res-12
data as "res 10"; now passes res=VIEWSHED_RES.
- Colorbar (config_nb): the earlier pin only covered the DEMO=True single
map; added a guarded per-panel pin for the DEMO=False multi-panel figure.
- Showcase (mdx + README): the top-10 render is a Golden Gate Park demo
view, so the full-SF section now frames it as the GGP detail.
Co-authored-by: Isaac <no-reply@databricks.com>
mjohns-databricks
temporarily deployed
to
runtime
October 7, 2026 17:27 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
mjohns-databricks
temporarily deployed
to
runtime
October 7, 2026 17:27 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
mjohns-databricks
temporarily deployed
to
runtime
October 7, 2026 17:59 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
mjohns-databricks
temporarily deployed
to
runtime
October 7, 2026 18:11 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wireless Coverage — 0.5.2
Adds the Wireless Coverage worked example in two flavors plus the primitive it needs, over a San Francisco USGS 3DEP LiDAR AOI on the lightweight tier.
What's new
notebooks/examples/wireless-coverage/,config_nb+ 01–04): classify LiDAR → DSM / bare-earth DTM / CHM surfaces → H3-grid all three → raster-pixel-vs-H3 contrast → naive H3 tower siting (quick-pass rule-out → exacth3_los_visibleviewshed → two-factor scoring). Ships the Golden Gate Park demo (DEMO=True); runs full San Francisco atDEMO=False..../lakeflow/): the production counterpart — a Databricks Asset Bundle (land → pipeline → siting → validate), medallion bronze → silver → gold on Serverless, same AOI and sources.pygxh3_viewshed_towers.Notable engineering
show_cell_map: GGP demo map, full-SF overview + windowed detail), nb4 side-by-side maps, colorbar pinned to the map box.Reviewed with Isaac Review; findings resolved. American spelling throughout.
This pull request and its description were written by Isaac.