Skip to content

ci(nextflow): nf-test infrastructure and GitHub Actions workflow#110

Open
FIrgolitsch wants to merge 1 commit into
pr-n4-bias-fieldfrom
pr-ci-nextflow
Open

ci(nextflow): nf-test infrastructure and GitHub Actions workflow#110
FIrgolitsch wants to merge 1 commit into
pr-n4-bias-fieldfrom
pr-ci-nextflow

Conversation

@FIrgolitsch

@FIrgolitsch FIrgolitsch commented Apr 28, 2026

Copy link
Copy Markdown
Contributor

Stacked PR 10/22 — review order: #115#97#98#99#100#101#108#106#107#87#116#110#111#40#112#113#117#118#120#121#122#123#124#125

Base: pr-n4-bias-field. Retargets to main as upstream PRs merge.


PR #110 — Nextflow CI: nf-test infrastructure + workflow streamline

Adds end-to-end testing for Nextflow pipelines using nf-test (all processes invoked in stub mode so tests run without real imaging data) and streamlines the workflow files to share configuration between pipelines.

Streamline refactor (one commit)

refactor(workflows): streamline configs, extract cpu_limits.config and Helpers.groovy

  • New workflows/shared/cpu_limits.config — single source of truth for enable_cpu_limits / max_cpus / reserved_cpus / processes and the beforeScript thread-cap closure, included by both pipelines
  • New workflows/reconst_3d/lib/Helpers.groovy — moves all the helper functions out of soct_3d_reconst.nf (303 lines extracted), auto-loaded by Nextflow via the lib/ convention
  • Consolidates publishDir calls and standardises on ${params.X} interpolation
  • Reorders sections in nextflow.config so the file reads top-down: params → process config → profiles
  • Net change: 6 files, +120/−386

CI (one squashed commit)

ci(nextflow): make CI green — pin nf-test 0.9.5, fix .gitignore for lib/ and zarr stubs, filter lint false-positives, cap test processes

  • .github/workflows/nextflow-ci.yml: triggers on workflows/** and .github/workflows/nextflow-ci.yml changes (main, dev); pinned to nf-test 0.9.5 (latest broke compilation by not auto-loading lib/); ANSI-strip + awk filter on lint output to drop the 29 known Helpers.X(...) false-positives that nextflow lint reports because it doesn't load Groovy classes from lib/
  • .gitignore: re-include rules for workflows/**/lib/** (was caught by the Python lib/ ignore — Helpers.groovy was never pushed) and for *.ome.zarr directories under tests/data/ (was caught by the *.zarr ignore — stub .gitkeep placeholders were never pushed)
  • nf-test.config: root nf-test configuration; tests/nextflow.config: shared test config
  • Stub blocks added to all processes in preproc_rawtiles.nf and soct_3d_reconst.nf

Tests

  • workflows/preproc/tests/preproc_rawtiles.nf.test — stub-run preproc pipeline (minimal tile inputs in workflows/preproc/tests/data/session_01/...)
  • workflows/reconst_3d/tests/soct_3d_reconst.nf.test — stub-run reconst pipeline (minimal shifts_xy.csv and mosaic_grid_z*.ome.zarr placeholders); processes = 2 cap on the optional-branch test so it fits 4-CPU GitHub-hosted runners

@FIrgolitsch FIrgolitsch force-pushed the pr-ci-nextflow branch 2 times, most recently from 954feb5 to b4c029d Compare April 30, 2026 02:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant