Running Plugins in ChELL

1. Overview

ChELL transforms plugin execution into a natural, filesystem-based workflow. Instead of complex API calls or verbose command-line invocations, you simply navigate to your data and type the plugin name—just like running a local program.

1.1. The Vision

ChRIS analyses are powerful but traditionally require understanding feeds, plugin instances, and graph topologies. ChELL abstracts this complexity behind a familiar metaphor: your filesystem location is your execution context.

cd ~/uploads/brain-scans
pl-dcm2niix-v2.1.1 --outputdir results

Behind this simple command, ChELL: 1. Creates a new feed 2. Copies your data into the feed 3. Runs the plugin on that data 4. Returns a job ID for monitoring

All of this happens transparently, letting you focus on your analysis instead of infrastructure.

2. The Two Execution Contexts

ChELL supports two distinct workflows based on where you are in the filesystem.

2.1. Context 1: Starting a New Analysis

When you’re in a regular directory (not inside a feed), typing a plugin name initiates a new analysis:

cd ~/uploads/SAG-anon
pl-dcm2niix-v2.1.1 --outputdir converted --compress true

What happens:

Feed Creation: ChELL creates a new feed using the special pl-dircopy plugin to copy your current directory into the ChRIS feed system
Data Preparation: Your data (SAG-anon) is now the root of a new feed
Plugin Execution: Your specified plugin (pl-dcm2niix) runs with the copied data as input
Result: You get back feed and job IDs

Output:

Feed created: 123
Job scheduled: pl-dcm2niix-v2.1.1 (ID: 789)
Output will be in: ~/feeds/feed_123/pl-dcm2niix_789/data/

The feed structure looks like:

~/feeds/feed_123/
  ├── pl-dircopy_456/
  │   └── data/              # Your copied SAG-anon data
  └── pl-dcm2niix_789/
      └── data/              # Converted NIfTI output

2.2. Context 2: Continuing an Existing Analysis

When you’re inside a feed directory, typing a plugin name continues the analysis:

cd ~/feeds/feed_123/pl-dircopy_456/data/
pl-segmentation-v1.0.0 --threshold 0.5

What happens:

Context Detection: ChELL recognizes you’re in a feed directory
Parent Resolution: Extracts the plugin instance ID (456) from the directory name
Plugin Execution: Runs segmentation with the dircopy output as input
Result: New plugin instance added to the feed graph

Output:

Job scheduled: pl-segmentation-v1.0.0 (ID: 891)
Output will be in: ~/feeds/feed_123/pl-segmentation_891/data/

The feed now has a branching structure:

~/feeds/feed_123/
  ├── pl-dircopy_456/
  │   └── data/
  ├── pl-dcm2niix_789/
  │   └── data/              # From pl-dircopy_456
  └── pl-segmentation_891/
      └── data/              # Also from pl-dircopy_456

3. The Filesystem Topology = Analysis Graph

The ChRIS feed filesystem directly represents your analysis graph:

~/feeds/feed_123/
  ├── pl-dircopy_456/          # Root node
  │   └── data/
  ├── pl-dcm2niix_789/         # Child of 456
  │   └── data/
  ├── pl-segmentation_891/     # Another child of 456
  │   └── data/
  └── pl-registration_912/     # Child of 789
      └── data/

The naming convention is critical:

<plugin-name>_<instance-id>/

This encoding allows ChELL to: - Determine which plugin created which data - Automatically wire up parent-child relationships - Navigate the graph by navigating directories

When you cd into any /data/ directory, the parent directory name tells ChELL: "This is the output of plugin instance N, so any new plugin I run should use N as its previous_id."

4. Understanding Asynchronous Execution

Unlike local programs that block until complete, ChRIS plugin execution is asynchronous:

pl-dcm2niix-v2.1.1 --outputdir results
# Returns immediately with job ID
# Plugin runs on a compute node in the background

This is analogous to Unix job control with &:

long-running-program &
[1] 12345              # Job ID returned immediately

Why asynchronous?

ChRIS plugins run on remote compute resources. A single DICOM-to-NIfTI conversion might take seconds, but a brain segmentation could take hours. Rather than blocking your shell, ChELL:

Submits the job to ChRIS
Returns a job ID immediately
Lets you continue working
Allows you to check status later (Phase 2 feature: plugin status <id>)

Your working directory remains unchanged after execution, because you’re not "in" the job—you’re observing it, like a backgrounded process.

5. Parameter Syntax

5.1. Basic Plugin Parameters

Plugin parameters work just like command-line arguments:

pl-dcm2niix-v2.1.1 --outputdir results --compress true --output-format nifti

All flags after the plugin name are passed directly to the plugin.

5.2. Context Parameters with `--` Delimiter

ChELL extends the syntax with context parameters that control feed creation or instance naming, separated by a -- delimiter:

<plugin-name> <plugin-params> -- <context-params>

Before --: Parameters for the plugin itself After --: Parameters for ChELL’s execution context

5.3. Available Context Parameters

5.3.1. `feed_title`

Used when creating a new feed (non-feed directory context):

pl-dcm2niix-v2.1.1 --outputdir results -- feed_title="Brain MRI Study 2025-12"

If omitted, ChELL prompts interactively:

Feed title [SAG-anon]: Brain MRI Study 2025-12

You can accept the default (current directory name) by pressing Enter.

5.3.2. `instance_title`

Used in existing feeds to name this plugin instance in the DAG graph:

pl-segmentation-v1.0.0 --threshold 0.5 -- instance_title="Segment Cortex"

This title appears in ChRIS UI visualizations of your analysis graph, making it easier to understand complex workflows.

5.4. Interactive vs Non-Interactive Mode

The rule: Presence of -- signals non-interactive mode.

Interactive (no --):

pl-dcm2niix-v2.1.1 --outputdir results
# Prompts: "Feed title [SAG-anon]: "

Non-interactive (-- present):

pl-dcm2niix-v2.1.1 --outputdir results -- feed_title="MRI Study"
# No prompts, all parameters provided

This is critical for scripting and automation. Scripts can provide all parameters upfront and run without user interaction.

6. Complete Walkthrough

Let’s follow a complete analysis from raw data to final results.

6.1. Step 1: Prepare Data

You’ve acquired DICOM images on your local machine:

ls ~/local-data/patient-001/
# Output: IMG0001.dcm IMG0002.dcm ... IMG0512.dcm

6.2. Step 2: Upload to ChRIS

Use ChELL’s upload command to transfer data:

upload ~/local-data/patient-001 ~/uploads/patient-001
# Uploading [████████████████████] 100% | 512/512 files

6.3. Step 3: Start Analysis - DICOM to NIfTI Conversion

cd ~/uploads/patient-001
pl-dcm2niix-v2.1.1 --outputdir nifti -- feed_title="Patient 001 Analysis"

Output:

Feed created: 42
Job scheduled: pl-dcm2niix-v2.1.1 (ID: 103)
Output will be in: ~/feeds/feed_42/pl-dcm2niix_103/data/

What you now have:

~/feeds/feed_42/
  ├── pl-dircopy_102/        # Root: your uploaded DICOMs
  │   └── data/
  └── pl-dcm2niix_103/       # Conversion job (running)
      └── data/              # NIfTI files appear here when complete

6.4. Step 4: Continue Analysis - Brain Extraction

Once conversion completes (you can check with ls or wait for notification), continue:

cd ~/feeds/feed_42/pl-dcm2niix_103/data/
ls
# Output: brain.nii.gz

# Run brain extraction
pl-fsl-bet-v1.0.2 --fractional-threshold 0.5 -- instance_title="Extract Brain"

Output:

Job scheduled: pl-fsl-bet-v1.0.2 (ID: 104)
Output will be in: ~/feeds/feed_42/pl-fsl-bet_104/data/

Feed structure:

~/feeds/feed_42/
  ├── pl-dircopy_102/
  │   └── data/
  ├── pl-dcm2niix_103/
  │   └── data/
  └── pl-fsl-bet_104/         # Brain extraction (running)
      └── data/

6.5. Step 5: Parallel Processing

You can spawn multiple analyses from the same data:

cd ~/feeds/feed_42/pl-dcm2niix_103/data/

# Segmentation
pl-freesurfer-v7.3.2 -- instance_title="Full Segmentation"

# Registration to atlas
pl-fsl-flirt-v1.0.0 --reference MNI152 -- instance_title="Register to MNI"

Both jobs run in parallel, each reading from pl-dcm2niix_103:

~/feeds/feed_42/
  ├── pl-dircopy_102/
  ├── pl-dcm2niix_103/
  ├── pl-fsl-bet_104/
  ├── pl-freesurfer_105/      # Parallel branch 1
  └── pl-fsl-flirt_106/       # Parallel branch 2

6.6. Step 6: Downstream Analysis

Outputs from one step become inputs to the next:

cd ~/feeds/feed_42/pl-fsl-bet_104/data/
ls
# Output: brain_mask.nii.gz

# Apply mask to segmentation
pl-apply-mask-v1.0.0 --mask brain_mask.nii.gz

This creates a two-level deep graph:

pl-dircopy_102
  └── pl-dcm2niix_103
        ├── pl-fsl-bet_104
        │     └── pl-apply-mask_107
        ├── pl-freesurfer_105
        └── pl-fsl-flirt_106

7. Working in Subdirectories

ChRIS plugins can create complex output directory structures:

~/feeds/feed_42/pl-freesurfer_105/data/
  ├── mri/
  │   ├── T1.mgz
  │   └── brain.mgz
  ├── surf/
  │   ├── lh.pial
  │   └── rh.pial
  └── stats/
      └── aseg.stats

ChELL handles navigation intelligently:

cd ~/feeds/feed_42/pl-freesurfer_105/data/mri/
# Deep in a subdirectory

pl-volume-analysis-v1.0.0 brain.mgz
# ChELL walks UP to find pl-freesurfer_105, extracts ID 105, uses as previous_id

The algorithm: Walk up from CWD until finding a directory matching <plugin-name>_<id>, extract id, use as previous_id.

This means you can navigate naturally within plugin outputs without worrying about the graph topology.

8. Plugin Name Completion

ChELL provides tab completion for plugin names:

pl-<TAB>
# Shows all available plugins:
# pl-dcm2niix-v2.1.1
# pl-dircopy-v2.0.0
# pl-fsl-bet-v1.0.2
# pl-freesurfer-v7.3.2
# ...

pl-dcm<TAB>
# Completes to: pl-dcm2niix-v2.1.1

Full version required: You must type (or tab-complete) the full plugin name including version: pl-dcm2niix-v2.1.1, not just pl-dcm2niix.

This ensures deterministic execution—you always know exactly which plugin version you’re running.

9. Behind the Scenes: pl-dircopy

When you run a plugin in a non-feed directory, ChELL uses the special pl-dircopy plugin to create the feed:

# You type:
cd ~/uploads/SAG-anon
pl-dcm2niix-v2.1.1 --outputdir results

# ChELL executes (transparently):
# 1. Create feed with pl-dircopy of ~/uploads/SAG-anon
# 2. Run pl-dcm2niix with pl-dircopy output as input

pl-dircopy requirements:

Must be registered in your ChRIS instance
If multiple versions exist (e.g., v1.0.0, v2.1.0), ChELL automatically selects the highest version
If missing: Error message "pl-dircopy not found. Cannot create feeds."

Resolution if missing:

plugin add pl-dircopy
# Or with full Docker image:
plugin add fnndsc/pl-dircopy:latest

10. Error Scenarios

10.1. Plugin Not Found

pl-typo-v1.2.3
# Error: Plugin 'pl-typo-v1.2.3' not found in /bin

Resolution: Check available plugins with cd /bin && ls, or use tab completion.

10.2. Not in Valid Context

cd /tmp
pl-dcm2niix-v2.1.1
# Error: Cannot create feed from /tmp (not a ChRIS directory)

Resolution: Navigate to a ChRIS-managed location (~/uploads/, ~/feeds/, etc.).

10.3. Cannot Extract Context

cd ~/feeds/feed_42/some-manual-directory/
pl-segmentation-v1.0.0
# Error: Could not determine plugin context from current directory

Resolution: Navigate to a valid plugin output directory (matching pattern <plugin-name>_<id>/data/).

10.4. pl-dircopy Missing

cd ~/uploads/data
pl-dcm2niix-v2.1.1
# Error: pl-dircopy not found. Cannot create feeds.

Resolution: Install pl-dircopy: plugin add pl-dircopy

11. Scripting with ChELL

The non-interactive mode (-- delimiter) enables automation:

#!/bin/bash
# process-patient.sh

PATIENT_ID=$1

# Upload data
upload ~/raw-data/${PATIENT_ID} ~/uploads/${PATIENT_ID}

# Start analysis (non-interactive)
cd ~/uploads/${PATIENT_ID}
pl-dcm2niix-v2.1.1 --outputdir nifti -- feed_title="Patient ${PATIENT_ID} Analysis"

# Note: Job runs asynchronously, script continues immediately
echo "Analysis started for patient ${PATIENT_ID}"

Usage:

./process-patient.sh 001
./process-patient.sh 002
./process-patient.sh 003
# All analyses submitted in parallel

12. Comparison with Traditional chili Commands

ChELL’s plugin execution is syntactic sugar over verbose chili commands:

ChELL (Simple)	chili (Verbose)
cd ~/uploads/data pl-dcm2niix-v2.1.1 --arg val	chili feed create \ --dirs /uploads/data \ --params "title:My Feed" # Note plugin ID from output (e.g., 456) chili plugin run pl-dcm2niix-v2.1.1 \ --previous_id 456 \ --params "arg:val"

ChELL (Simple)

chili (Verbose)

cd ~/uploads/data
pl-dcm2niix-v2.1.1 --arg val

chili feed create \
  --dirs /uploads/data \
  --params "title:My Feed"

# Note plugin ID from output (e.g., 456)

chili plugin run pl-dcm2niix-v2.1.1 \
  --previous_id 456 \
  --params "arg:val"

ChELL collapses two commands, context management, and parameter marshaling into a single, intuitive invocation.

13. Phase 2 Features (Future)

The following features are not yet implemented but planned:

13.1. Job Status Monitoring

plugin status 789
# Output:
# Job ID: 789
# Plugin: pl-dcm2niix-v2.1.1
# Status: running
# Progress: 45%
# Started: 2025-12-05 14:32:10

13.2. Real-Time Progress

pl-freesurfer-v7.3.2 --follow
# Streams live output from compute node
# [14:35:12] Preprocessing...
# [14:36:45] Surface reconstruction...
# [14:42:03] Cortical parcellation...

pl-dcm2niix-v2.1.1 --outputdir results --cd-on-complete
# After job completes, shell automatically cd's to output directory

13.4. Dependency Specification

pl-registration-v1.0.0 --wait-for 103,104
# Waits for jobs 103 and 104 to complete before starting

14. Best Practices

14.1. Organizing Uploads

Structure your uploads directory logically:

~/uploads/
  ├── project-a/
  │   ├── patient-001/
  │   ├── patient-002/
  │   └── patient-003/
  └── project-b/
      ├── scan-alpha/
      └── scan-beta/

Each subdirectory can become a separate feed.

14.2. Naming Feeds

Use descriptive, searchable titles:

# Good
-- feed_title="2025-12-05 Patient 001 Structural MRI"

# Avoid
-- feed_title="test" # Too generic
-- feed_title="asdf" # Meaningless

14.3. Naming Plugin Instances

Use instance titles to document your workflow:

pl-dcm2niix-v2.1.1 -- instance_title="Convert T1 Structural"
pl-fsl-bet-v1.0.2 -- instance_title="Extract Brain (thresh=0.5)"
pl-freesurfer-v7.3.2 -- instance_title="Full Cortical Segmentation"

This creates a self-documenting analysis graph visible in ChRIS UI.

14.4. Keeping Context

When working on long-running analyses, document your feed ID:

echo "Patient 001 Analysis: feed_42" > ~/notes.txt
# Later:
cd ~/feeds/feed_42/

Or use descriptive feed titles that are easy to search.

14.5. Parallel vs Sequential

Parallel: Run multiple plugins from the same data simultaneously:

cd ~/feeds/feed_42/pl-dcm2niix_103/data/
pl-segmentation-v1.0.0 &
pl-registration-v1.0.0 &
# Both jobs submitted, running in parallel

Sequential: Wait for completion before next step (manual for now, automated in Phase 2):

pl-step1-v1.0.0
# Wait for completion (check with ls or Phase 2 `plugin status`)
cd ~/feeds/feed_42/pl-step1_104/data/
pl-step2-v1.0.0

15. Technical Details

15.1. Path Pattern Matching

ChELL detects feed directories by matching:

/home/<username>/feeds/feed_<numeric-id>/

Examples:

/home/chris/feeds/feed_42/           ✓ Feed directory
/home/chris/feeds/feed_42/pl-dircopy_103/data/ ✓ In feed
/home/chris/uploads/data/            ✗ Not a feed

15.2. Plugin Instance ID Extraction

ChELL walks up the directory tree from CWD, looking for:

<plugin-name>_<numeric-id>

Regex: ^(.)_(\d)$

Examples:

pl-dircopy_456        → ID: 456
pl-dcm2niix_789       → ID: 789
pl-freesurfer_1042    → ID: 1042

15.3. Parameter Parsing

ChELL splits arguments on the first ` — ` (space-dash-dash-space) occurrence:

Input: "pl-test --a 1 --b 2 -- title=Foo limit=99"
Split: ["--a 1 --b 2", "title=Foo limit=99"]
Plugin params: {a: "1", b: "2"}
Context params: {title: "Foo", limit: "99"}

If ` — ` is absent, all arguments are plugin parameters, and ChELL prompts for missing context parameters interactively.

16. Troubleshooting

16.1. "Plugin not found" but it exists

Symptom: You can see the plugin with ls /bin but ChELL says it’s not found.

Cause: Tab completion or typing error in version number.

Solution:

cd /bin
ls | grep dcm
# Output: pl-dcm2niix-v2.1.1

# Copy exact name
pl-dcm2niix-v2.1.1 --help

16.2. Job never completes

Symptom: You submitted a job hours ago, but the output directory is still empty.

Cause: Job may have failed, or compute resource is overloaded.

Solution (Phase 2):

plugin status 789
# Check job status and logs

Current workaround: Check ChRIS web UI for job status.

16.3. Wrong previous_id

Symptom: Plugin runs but uses unexpected input data.

Cause: Navigated to wrong directory, so ChELL extracted wrong plugin instance ID.

Solution: Verify you’re in the correct output directory:

pwd
# Should be: ~/feeds/feed_42/pl-dcm2niix_103/data/
# Not: ~/feeds/feed_42/pl-dircopy_102/data/

16.4. Cannot create feed

Symptom: "pl-dircopy not found. Cannot create feeds."

Cause: pl-dircopy plugin not installed.

Solution:

plugin add pl-dircopy
# Or:
plugin add fnndsc/pl-dircopy:latest --compute host

17. Summary

ChELL’s plugin execution model transforms ChRIS from a complex API-driven system into an intuitive, filesystem-based workflow. By encoding context in directory paths and treating plugins like local executables, it provides:

Simplicity: cd to data, type plugin name
Transparency: Filesystem structure = analysis graph
Flexibility: Interactive prompts or scriptable non-interactive mode
Power: Full ChRIS capabilities without API complexity

The key insight: your location is your context. Navigate to your data, run your analysis, and let ChELL handle the complexity of feeds, instances, and graph topology.

Last updated: 2025-12-05

FilesExpand file tree

pluginrun.adoc

Latest commit

History

pluginrun.adoc

File metadata and controls

Running Plugins in ChELL

1. Overview

1.1. The Vision

2. The Two Execution Contexts

2.1. Context 1: Starting a New Analysis

2.2. Context 2: Continuing an Existing Analysis

3. The Filesystem Topology = Analysis Graph

4. Understanding Asynchronous Execution

5. Parameter Syntax

5.1. Basic Plugin Parameters

5.2. Context Parameters with -- Delimiter

5.3. Available Context Parameters

5.3.1. feed_title

5.3.2. instance_title

5.4. Interactive vs Non-Interactive Mode

6. Complete Walkthrough

6.1. Step 1: Prepare Data

6.2. Step 2: Upload to ChRIS

6.3. Step 3: Start Analysis - DICOM to NIfTI Conversion

6.4. Step 4: Continue Analysis - Brain Extraction

6.5. Step 5: Parallel Processing

6.6. Step 6: Downstream Analysis

7. Working in Subdirectories

8. Plugin Name Completion

9. Behind the Scenes: pl-dircopy

10. Error Scenarios

10.1. Plugin Not Found

10.2. Not in Valid Context

10.3. Cannot Extract Context

10.4. pl-dircopy Missing

11. Scripting with ChELL

12. Comparison with Traditional chili Commands

13. Phase 2 Features (Future)

13.1. Job Status Monitoring

13.2. Real-Time Progress

13.3. Automatic Navigation

13.4. Dependency Specification

14. Best Practices

14.1. Organizing Uploads

14.2. Naming Feeds

14.3. Naming Plugin Instances

14.4. Keeping Context

14.5. Parallel vs Sequential

15. Technical Details

15.1. Path Pattern Matching

15.2. Plugin Instance ID Extraction

15.3. Parameter Parsing

16. Troubleshooting

16.1. "Plugin not found" but it exists

16.2. Job never completes

16.3. Wrong previous_id

16.4. Cannot create feed

17. Summary

5.2. Context Parameters with `--` Delimiter

5.3.1. `feed_title`

5.3.2. `instance_title`