Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
128 changes: 99 additions & 29 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,54 +2,124 @@ name: CI

on:
push:
branches: ["main", "develop", "workspace-esparso"]
branches: [main, develop]
pull_request:
branches: ["main", "develop"]
branches: [main, develop]
workflow_dispatch: # allows running the workflow manually

jobs:
lint:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: "3.12"

- name: Install Ruff
# keep in sync with the `ruff` pin in pyproject.toml's `dev` extra
run: pip install "ruff>=0.16,<0.17"

- name: Run Ruff
run: ruff check src/ tests/ examples/ scripts/

test:
# dissmodel + every optional geospatial extra that does not depend on
# repositories still under construction (brmangue-dissmodel is left out
# on purpose). Starts at 3.11 because the `zarr` extra needs
# zarr-python 3, which requires Python >= 3.11 (same floor as disscube).
name: test (py${{ matrix.python-version }})
runs-on: ubuntu-latest
timeout-minutes: 20

strategy:
fail-fast: false
matrix:
include:
# Núcleo puro: sem dissmodel nem nenhum extra geoespacial.
# Garante que engine.py, disk/workspace.py, disk/convergence.py
# continuam com zero dependências em runtime.
- name: "core (sem extras)"
extras: "dev"
# dissmodel + todos os extras geoespaciais opcionais que não
# dependem de repositórios externos ainda em construção
# (brmangue-dissmodel fica de fora de propósito).
- name: "full (dissmodel + geo extras)"
extras: "dev,dissmodel,geotiff,zarr,zarr-test,geomosaic,dissmodel-ca"
python-version: ["3.11", "3.12"]

steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: pip
cache-dependency-path: pyproject.toml

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e ".[dev,dissmodel,geotiff,zarr,zarr-test,geomosaic,dissmodel-ca]"

# Coverage XML is produced but not uploaded anywhere yet.
- name: Run tests with coverage
run: pytest tests/ --cov=haloexec --cov-report=term-missing --cov-report=xml

test-minimal:
# Core install without optional extras: guarantees engine.py,
# disk/workspace.py and disk/convergence.py keep zero runtime
# dependencies beyond numpy, and that tests needing optional packages
# are skipped, not failed. Runs on 3.10, the lowest version the core
# supports (requires-python), which the full job cannot cover.
name: test (core install, no extras, py3.10)
runs-on: ubuntu-latest
timeout-minutes: 20

steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: "3.10"
cache: pip
cache-dependency-path: pyproject.toml

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e ".[dev]"

- name: Run tests
run: pytest tests/ -rs

docs:
# 1) builds the mkdocs site in strict mode (broken links/anchors fail);
# 2) extracts every ```python block from docs/ and runs it, to catch
# broken quickstarts (import errors, wrong mixin/inheritance order,
# renamed APIs).
name: docs (strict build + code blocks)
runs-on: ubuntu-latest
timeout-minutes: 20

steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
cache-dependency-path: pyproject.toml

- name: Install package (${{ matrix.name }})
run: pip install -e ".[${{ matrix.extras }}]"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e ".[dev,docs,dissmodel,geotiff,zarr,zarr-test,geomosaic,dissmodel-ca]"

- name: Run test suite
run: pytest -v
- name: Build site (strict)
run: mkdocs build --strict

- name: Extract and smoke-test Python blocks from docs/
if: matrix.name == 'full (dissmodel + geo extras)'
- name: Run Python blocks from docs/
run: |
python - <<'EOF'
import re, subprocess, sys, pathlib

docs = pathlib.Path("docs").glob("*.md")
failures = []
# Palavras que indicam um bloco ilustrativo (referencia dado
# externo que não existe neste repo, ou escala grande demais
# pra rodar em CI) -- pulamos, não são candidatos a bug de
# import/herança, que é o que este smoke-test mira.
# Markers of illustrative blocks (external data not in this repo,
# or a scale too large for CI) -- skipped, since they are not the
# import/inheritance bugs this smoke test targets.
SKIP_MARKERS = ("50_000", "study_area.tif", "tile_001.zarr", "/tmp/my_large_workspace")

for doc in docs:
Expand All @@ -60,7 +130,7 @@ jobs:
continue
if "import" not in block:
continue
print(f"--- {doc.name} bloco {i} ---")
print(f"--- {doc.name} block {i} ---")
result = subprocess.run(
[sys.executable, "-c", block],
capture_output=True, text=True, timeout=120,
Expand All @@ -70,9 +140,9 @@ jobs:
print(result.stderr)

if failures:
print(f"\n{len(failures)} bloco(s) de código quebrado(s) em docs/:")
print(f"\n{len(failures)} broken code block(s) in docs/:")
for name, i, err in failures:
print(f" - {name} bloco {i}")
print(f" - {name} block {i}")
sys.exit(1)
print("Todos os blocos de código testáveis em docs/ rodaram sem erro.")
print("All testable code blocks in docs/ ran without error.")
EOF
24 changes: 24 additions & 0 deletions .github/workflows/docs_deploy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
name: Deploy documentation to GitHub Pages

on:
push:
branches: [main]
workflow_dispatch:

jobs:
deploy:
runs-on: ubuntu-latest
permissions:
contents: write # needed to push to gh-pages (ghp-import)
steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: "3.12"

- name: Install dependencies
run: pip install -e ".[docs]"

- name: Deploy to GitHub Pages
run: mkdocs gh-deploy --force --strict
8 changes: 7 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,10 @@ __pycache__/
.pytest_cache/
*.egg-info/
.venv
raster_map_frames/
raster_map_frames/
.coverage
coverage.xml
htmlcov/
build/
dist/
site/
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# haloexec

[![CI](https://github.com/DisSModel/haloexec/actions/workflows/ci.yml/badge.svg)](https://github.com/DisSModel/haloexec/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

Cellular Automaton execution engine using **Domain Decomposition** with
**Halo Zones** (Ghost Cell Pattern), integrated with real
[dissmodel](https://pypi.org/project/dissmodel/) via a pip dependency —
Expand Down
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ env.run()

### 2. Out-of-Core Disk-Backed Simulation

When the domain exceeds physical memory, initialize a [`MemmapRasterWorkspace`](../src/haloexec/disk/workspace.py#L81) and run out-of-core:
When the domain exceeds physical memory, initialize a [`MemmapRasterWorkspace`](https://github.com/DisSModel/haloexec/blob/main/src/haloexec/disk/workspace.py#L81) and run out-of-core:

```python
from pathlib import Path
Expand Down
4 changes: 2 additions & 2 deletions docs/api_reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,11 @@ This document provides a comprehensive, exhaustive reference for all public clas
- [DiskChunkedSyncRasterModel](#diskchunkedsyncrastermodel)
- [haloexec.disk.cellular_automaton](#haloexecdiskcellular_automaton)
- [DiskChunkedRasterCellularAutomaton](#diskchunkedrastercellularautomaton)
- [haloexec.disk.io.geotiff](#haloexecdiskio_geotiff)
- [haloexec.disk.io.geotiff](#haloexecdiskiogeotiff)
- [load_geotiff_into_workspace](#load_geotiff_into_workspace)
- [load_geotiffs_into_workspace](#load_geotiffs_into_workspace)
- [save_workspace_to_geotiff](#save_workspace_to_geotiff)
- [haloexec.disk.io.zarr](#haloexecdiskio_zarr)
- [haloexec.disk.io.zarr](#haloexecdiskiozarr)
- [load_zarr_into_workspace](#load_zarr_into_workspace)
- [load_zarr_tiles_into_workspace](#load_zarr_tiles_into_workspace)
- [haloexec.ram.cellular_automaton](#haloexecramcellular_automaton)
Expand Down
14 changes: 7 additions & 7 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,12 +69,12 @@ graph TD
## 3. Component Deep Dive by Layer

### Layer 0: Pure Core Primitives (`haloexec.engine`)
- **[Block](../src/haloexec/engine.py#L21)**: An immutable, hashable dataclass representing a rectangular domain slice $[r_0:r_1, c_0:c_1)$. Provides properties `.shape` and `.core` (ready-to-use tuple of slice objects).
- **[make_blocks](../src/haloexec/engine.py#L41)**: Pure generator/function that subdivides arbitrary 2D dimensions $(H, W)$ into regular blocks of target size $(b_h, b_w)$, naturally accommodating edge remainders.
- **[resolve_boundary_value](../src/haloexec/engine.py#L55)**: Sentinel resolution utility mapping array names to boundary nodata values, with automatic fallback for `<name>_past` temporal layers.
- **[Block](https://github.com/DisSModel/haloexec/blob/main/src/haloexec/engine.py#L21)**: An immutable, hashable dataclass representing a rectangular domain slice $[r_0:r_1, c_0:c_1)$. Provides properties `.shape` and `.core` (ready-to-use tuple of slice objects).
- **[make_blocks](https://github.com/DisSModel/haloexec/blob/main/src/haloexec/engine.py#L41)**: Pure generator/function that subdivides arbitrary 2D dimensions $(H, W)$ into regular blocks of target size $(b_h, b_w)$, naturally accommodating edge remainders.
- **[resolve_boundary_value](https://github.com/DisSModel/haloexec/blob/main/src/haloexec/engine.py#L55)**: Sentinel resolution utility mapping array names to boundary nodata values, with automatic fallback for `<name>_past` temporal layers.

### Layer 1: Storage & Workspace Management (`haloexec.disk.workspace`)
- **[MemmapRasterWorkspace](../src/haloexec/disk/workspace.py#L81)**: Manages binary disk arrays on the filesystem.
- **[MemmapRasterWorkspace](https://github.com/DisSModel/haloexec/blob/main/src/haloexec/disk/workspace.py#L81)**: Manages binary disk arrays on the filesystem.
- **Directory Hierarchy**:
```text
workspace_root/
Expand All @@ -97,11 +97,11 @@ graph TD
- `checkpoint(step)`: Atomically persists execution state.

### Layer 2: Spatial Ingestion & Egress (`haloexec.disk.io`)
- **[geotiff.py](../src/haloexec/disk/io/geotiff.py)**:
- **[geotiff.py](https://github.com/DisSModel/haloexec/blob/main/src/haloexec/disk/io/geotiff.py)**:
- Streams GeoTIFF and VRT rasters into a `MemmapRasterWorkspace` block-by-block using `rasterio.windows.Window`.
- Validates coordinate reference systems (CRS) and dimension consistency across multi-file inputs.
- Exports workspace slots to Cloud-Optimized GeoTIFFs (COG) with block-aligned tiling.
- **[zarr.py](../src/haloexec/disk/io/zarr.py)**:
- **[zarr.py](https://github.com/DisSModel/haloexec/blob/main/src/haloexec/disk/io/zarr.py)**:
- Streams directly from Zarr groups or arrays (compatible with `disscube` data cubes).
- Normalizes coordinate axis orders using Zarr v3 dimension metadata (`arr.metadata.dimension_names`) to prevent silent transpose bugs.
- Assembles multi-tile footprints (`load_zarr_tiles_into_workspace`) into continuous seamless rasters.
Expand Down Expand Up @@ -166,7 +166,7 @@ By ensuring that `self.backend` and `self.shape` point to `block_backend` during
### 4.3 Decimation Adapter Pattern (`WorkspaceRasterBackend`)
Visualizing multi-million-cell rasters with `matplotlib` or interactive dashboards causes severe performance degradation and memory spikes.

Rather than copying and downsampling the entire raster in RAM, [WorkspaceRasterBackend](../src/haloexec/disk/backend.py#L17) implements an on-the-fly strided view:
Rather than copying and downsampling the entire raster in RAM, [WorkspaceRasterBackend](https://github.com/DisSModel/haloexec/blob/main/src/haloexec/disk/backend.py#L16) implements an on-the-fly strided view:

```python
# haloexec/disk/backend.py:L57
Expand Down
20 changes: 20 additions & 0 deletions docs/javascripts/mathjax.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
// MathJax configuration for pymdownx.arithmatex (generic mode).
window.MathJax = {
tex: {
inlineMath: [["\\(", "\\)"]],
displayMath: [["\\[", "\\]"]],
processEscapes: true,
processEnvironments: true,
},
options: {
ignoreHtmlClass: ".*|",
processHtmlClass: "arithmatex",
},
};

document$.subscribe(() => {
MathJax.startup.output.clearCache();
MathJax.typesetClear();
MathJax.texReset();
MathJax.typesetPromise();
});
16 changes: 8 additions & 8 deletions docs/theory_and_concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ A ubiquitous practice in numerical computing is zero-padding (`np.pad(..., const
If unmanaged, zero-padding external boundaries injects artificial river channels or sea-level cells along the edge of the study area, generating phantom colonization and divergent model dynamics.

### 4.2 Per-Array Sentinel Mapping
`haloexec` prevents this failure via [resolve_boundary_value](../src/haloexec/engine.py#L55):
`haloexec` prevents this failure via [resolve_boundary_value](https://github.com/DisSModel/haloexec/blob/main/src/haloexec/engine.py#L55):
- `boundary_value` accepts a dictionary mapping array names to their true domain `nodata` sentinel:
```python
boundary_value = {"uso": 0, "alt": -9999.0, "solo": -1}
Expand All @@ -149,7 +149,7 @@ If unmanaged, zero-padding external boundaries injects artificial river channels
When scaling to regional or national extents (e.g., a 40,000 $\times$ 40,000 grid encompassing 1.6 billion cells, or 6.4 GB per float32 layer), allocating global in-memory NumPy arrays or performing global `np.pad` results in immediate Out-Of-Memory (OOM) termination.

### 5.1 Memory-Mapped I/O (`numpy.memmap`)
[`MemmapRasterWorkspace`](../src/haloexec/disk/workspace.py#L81) stores spatial arrays directly on the filesystem as uncompressed binary disk files (`.dat`) accessed through POSIX `mmap()` system calls.
[`MemmapRasterWorkspace`](https://github.com/DisSModel/haloexec/blob/main/src/haloexec/disk/workspace.py#L81) stores spatial arrays directly on the filesystem as uncompressed binary disk files (`.dat`) accessed through POSIX `mmap()` system calls.

Under `mmap`:
1. The kernel maps the file on disk into the virtual address space of the process.
Expand All @@ -174,7 +174,7 @@ In a 40,000 $\times$ 40,000 Conway's Game of Life simulation (1.6 billion cells
This proves that `haloexec`'s memory footprint is $O(b_h \cdot b_w)$—proportional strictly to the block size, and **strictly independent of the global domain size** $H \times W$.

### 5.3 Filesystem Sparsity Economics
When creating new arrays, [`MemmapRasterWorkspace.create()`](../src/haloexec/disk/workspace.py#L145) uses `mode="w+"` to truncate and set the file length without writing zeros across the disk blocks.
When creating new arrays, [`MemmapRasterWorkspace.create()`](https://github.com/DisSModel/haloexec/blob/main/src/haloexec/disk/workspace.py#L144) uses `mode="w+"` to truncate and set the file length without writing zeros across the disk blocks.

On modern filesystems (ext4, XFS, APFS, NTFS), this creates **POSIX sparse files**:
- Unwritten disk blocks occupy zero physical storage on disk.
Expand Down Expand Up @@ -230,7 +230,7 @@ backend.arrays["alt_past"] = backend.arrays["alt"].copy()
```
If applied naively to a disk-backed memory-mapped workspace, this call would materialize the entire multi-gigabyte array in RAM, crashing the system.

`haloexec` resolves this via [write_block_to_read_slot](../src/haloexec/disk/workspace.py#L269):
`haloexec` resolves this via [write_block_to_read_slot](https://github.com/DisSModel/haloexec/blob/main/src/haloexec/disk/workspace.py#L269):
1. Historical arrays `<name>_past` are allocated directly within both disk slots.
2. In `pre_execute()` and `post_execute()`, synchronization copies data **block-by-block** directly within the **active read slot**:
$$\text{Slot}_{\text{read}}[name\_past][block] \longleftarrow \text{Slot}_{\text{read}}[name][block]$$
Expand All @@ -248,10 +248,10 @@ Certain spatial processes cannot be bounded by a finite, localized halo radius:
Resolving these dynamics with a static halo would require setting $h = \max(H, W)$, degenerating into a monolithic execution that exhausts memory.

### 7.2 Gauss-Seidel Iterative Convergence
For these unbounded processes, [`sweep_until_convergence`](../src/haloexec/disk/convergence.py#L46) replaces the fixed halo ping-pong mechanism with an **iterative relaxation sweep**:
For these unbounded processes, [`sweep_until_convergence`](https://github.com/DisSModel/haloexec/blob/main/src/haloexec/disk/convergence.py#L46) replaces the fixed halo ping-pong mechanism with an **iterative relaxation sweep**:
- Uses a minimal halo ($h = 1$).
- Traverses all blocks in the domain repeatedly.
- Updates are written **immediately in-place** to the active read buffer via [write_block_core_in_place](../src/haloexec/disk/workspace.py#L279).
- Updates are written **immediately in-place** to the active read buffer via [write_block_core_in_place](https://github.com/DisSModel/haloexec/blob/main/src/haloexec/disk/workspace.py#L279).

```
Sweep k:
Expand Down Expand Up @@ -279,7 +279,7 @@ and the state space $\mathcal{S}$ is finite (e.g., binary connectivity $\{0, 1\}
## 8. Spatial Ingestion & Coordinate Anomalies

### 8.1 GeoTIFF Windowed Streaming
Rather than loading large GeoTIFF or VRT rasters monolithically into memory, [`load_geotiffs_into_workspace`](../src/haloexec/disk/io/geotiff.py#L74) maps each block's bounding box to a `rasterio.windows.Window`:
Rather than loading large GeoTIFF or VRT rasters monolithically into memory, [`load_geotiffs_into_workspace`](https://github.com/DisSModel/haloexec/blob/main/src/haloexec/disk/io/geotiff.py#L75) maps each block's bounding box to a `rasterio.windows.Window`:

$$\text{Window}(\text{col\_off} = c_0, \; \text{row\_off} = r_0, \; \text{width} = c_1 - c_0, \; \text{height} = r_1 - r_0)$$

Expand All @@ -292,7 +292,7 @@ When ingesting data cubes from modern cloud stores (such as `disscube` or xarray
$$\text{shape}(arr) = (N, N) \equiv (N, N)$$
fails to detect inverted axes. This causes silent, catastrophic transpose corruption (e.g., swapping latitude and longitude).

To safeguard data integrity, [`load_zarr_into_workspace`](../src/haloexec/disk/io/zarr.py#L86) interrogates native Zarr v3 dimension metadata (`arr.metadata.dimension_names`):
To safeguard data integrity, [`load_zarr_into_workspace`](https://github.com/DisSModel/haloexec/blob/main/src/haloexec/disk/io/zarr.py#L93) interrogates native Zarr v3 dimension metadata (`arr.metadata.dimension_names`):
1. Resolves actual dimension positions: e.g., identifying whether `"y"` is at axis 0 or axis 1.
2. Generates canonical index projections during chunk extraction.
3. Automatically transposes extracted sub-arrays to standard `(y, x)` canonical orientation before committing blocks to the workspace.
3 changes: 1 addition & 2 deletions examples/gol/gol_patterns_haloexec.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,13 +22,12 @@
from __future__ import annotations

import numpy as np

from dissmodel.core import Environment
from dissmodel.geo import raster_grid
from dissmodel.visualization.raster_map import RasterMap
from dissmodel_ca.models.game_of_life import PATTERNS

from haloexec import HaloChunkedRasterCellularAutomaton
from dissmodel_ca.models.game_of_life import PATTERNS


# ---------------------------------------------------------------------------
Expand Down
Loading
Loading