Skip to content

Add workflow debugging feature#203

Open
sminot wants to merge 6 commits intomainfrom
debug-workflow
Open

Add workflow debugging feature#203
sminot wants to merge 6 commits intomainfrom
debug-workflow

Conversation

@sminot
Copy link
Copy Markdown
Contributor

@sminot sminot commented Apr 17, 2026

Adds the ability to inspect and debug failed Nextflow workflow executions directly from the Cirro SDK and CLI.


What's new for users

cirro debug — a new CLI command to inspect a failed dataset. Prints the last 50 lines of the execution log, identifies the primary failed task automatically, and shows its script, log, input files, and output files. Pass -i/--interactive to enter a menu-driven exploration mode where you can browse inputs and outputs, drill into source tasks, and read file contents directly in the terminal (as text, JSON, or CSV).


CLI

Command Description
cirro debug --project <name> --dataset <name> Non-interactive: print task debug summary, recurse through input chain
cirro debug -i Interactive: menu-driven task and file exploration

New SDK classes

DataPortalTask (cirro/sdk/task.py)

Represents a single task from a Nextflow workflow execution. Metadata is read from the WORKFLOW_TRACE artifact; logs and files are fetched on demand.

Attribute Description
name, status, exit_code, hash, work_dir, task_id Trace-derived metadata
logs Task stdout/stderr (via execution API, with .command.log fallback)
script The shell script Nextflow ran (.command.sh, with log-artifact fallback)
inputs WorkDirFile list parsed from .command.run, each linked to its source_task
outputs Non-hidden files in the task's S3 work directory

WorkDirFile (cirro/sdk/task.py)

Represents a file in a Nextflow S3 work directory or dataset staging area.

Attribute / Method Description
name, size, source_task File metadata
read(), readlines() Read as text (supports gzip)
read_json() Parse as JSON
read_csv() Parse as a Pandas DataFrame (auto-infers .gz/.bz2/.xz/.zst compression)

Additions to existing SDK classes

Addition Description
DataPortalDataset.executor Executor type (NEXTFLOW, CROMWELL) for the dataset's process
DataPortalDataset.logs Top-level execution log via Cirro API (CloudWatch)
DataPortalDataset.tasks Full list of DataPortalTask objects from the trace artifact
DataPortalDataset.primary_failed_task Auto-identifies the root-cause failed task by cross-referencing exit codes with the execution log; returns None gracefully for non-Nextflow executors, empty traces, or successful runs

Internal changes

  • FileAccessContext.scratch_download() — new classmethod for accessing Nextflow scratch bucket files
  • FileService._get_scratch_read_credentials() — cached credential fetch for scratch bucket reads
  • Null-guard added in ExecutionService for resp.events when log responses are empty

@sonarqubecloud
Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant