Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# SCM syntax highlighting & preventing 3-way merges
pixi.lock merge=binary linguist-language=YAML linguist-generated=true
83 changes: 31 additions & 52 deletions .github/workflows/test-spras.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,29 +3,26 @@ name: Test SPRAS
on: [push, pull_request]

jobs:
# Installs the conda environment but does not run tests because the tests require Linux Docker images
conda-only:
name: Test conda environment
# Installs the pixi environment but does not run tests because the tests require Linux Docker images
pixi-only:
name: Test pixi environment
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [macos-latest, windows-latest]
steps:
- name: Checkout repository
uses: actions/checkout@v2
- name: Install conda environment
uses: conda-incubator/setup-miniconda@v2
uses: actions/checkout@v4
- name: Install pixi
uses: prefix-dev/setup-pixi@v0.9.0
with:
activate-environment: spras
environment-file: environment.yml
auto-activate-base: false
miniconda-version: 'latest'
- name: Log conda environment
# Log conda environment contents
shell: bash --login {0}
run: conda list
pixi-version: v0.53.0
cache: false
locked: false
- name: Log pixi environment contents
run: pixi list

# Runs the test code and Snakemake workflow in the conda environment
# Runs the test code and Snakemake workflow in the pixi environment
test:
name: Run tests
# The Docker images will be pulled in both the docker job and this test job
Expand All @@ -48,32 +45,20 @@ jobs:
sudo rm -rf /usr/share/dotnet /usr/local/lib/android
sudo docker image prune --all --force
sudo docker builder prune -a
- name: Install conda environment
uses: conda-incubator/setup-miniconda@v2
with:
activate-environment: spras
environment-file: environment.yml
auto-activate-base: false
miniconda-version: 'latest'
- name: Install spras in conda env
# Install spras in the environment using pip
shell: bash --login {0}
run: pip install .
- name: Log conda environment
# Log conda environment contents
shell: bash --login {0}
run: conda list
- name: Install pixi
uses: prefix-dev/setup-pixi@v0.9.0
- name: Log pixi environment contents
run: pixi list
- name: Install Apptainer
# Formerly used Singularity instead of Apptainer (see https://github.com/eWaterCycle/setup-singularity/issues/6)
uses: eWaterCycle/setup-apptainer@v2
with:
# Choose version from https://github.com/apptainer/apptainer/releases
apptainer-version: 1.3.6
- name: Run tests
shell: bash --login {0}
# Verbose output and disable stdout and stderr capturing
# [this pytest-split workflow was borrowed from Snakemake CI]
run: pytest -vs --splits 2 --group ${{ matrix.test_group }}
run: pixi run pytest -vs --splits 2 --group ${{ matrix.test_group }}
workflow:
name: Run workflow
runs-on: ${{ matrix.os }}
Expand All @@ -84,23 +69,19 @@ jobs:
# TODO: is it worth it to make this setup into a composite action?
- name: Checkout repository
uses: actions/checkout@v4
- name: Install conda environment
uses: conda-incubator/setup-miniconda@v2
- name: Install pixi
uses: prefix-dev/setup-pixi@v0.9.0
with:
activate-environment: spras
environment-file: environment.yml
auto-activate-base: false
miniconda-version: 'latest'
# Install spras in the environment using pip
- name: Install spras in conda env
shell: bash --login {0}
run: pip install .
pixi-version: v0.53.0
cache: false
locked: false
- name: Log pixi environment contents
run: pixi list
- name: Run Snakemake workflow
shell: bash --login {0}
# We enable high parallelization (cores 4) to test our way out of the experienced
# race conditions from #268 and #279
# We also enforce strict DAG evaluation to catch DAG problems before they appear as user errors. (#359)
run: snakemake --cores 4 --configfile config/config.yaml --show-failed-logs --strict-dag-evaluation cyclic-graph --strict-dag-evaluation functions --strict-dag-evaluation periodic-wildcards
run: pixi run snakemake --cores 4 --configfile config/config.yaml --show-failed-logs --strict-dag-evaluation cyclic-graph --strict-dag-evaluation functions --strict-dag-evaluation periodic-wildcards

# Run pre-commit checks on source files
pre-commit:
Expand All @@ -109,14 +90,12 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v3
- name: Install conda environment
uses: conda-incubator/setup-miniconda@v2
- name: Install pixi
uses: prefix-dev/setup-pixi@v0.9.0
with:
activate-environment: spras
environment-file: environment.yml
auto-activate-base: false
miniconda-version: 'latest'
pixi-version: v0.53.0
cache: false
locked: false
- name: Run pre-commit
shell: bash --login {0}
# https://github.com/pre-commit/action/blob/576ff52938d158a24ac7e009dfa94b1455e7df99/action.yml#L19
run: pre-commit run --all-files --show-diff-on-failure --color=always
run: pixi run pre-commit run --all-files --show-diff-on-failure --color=always
9 changes: 9 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -144,3 +144,12 @@ TempMat.mat

# Singularity cache
unpacked

# pixi environments
.pixi
pixi.lock
*.egg-info

# pixi environments
.pixi
*.egg-info
1 change: 1 addition & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
exclude: (^.pixi/|.snap)
default_language_version:
# Match this to the version specified in environment.yml
python: python3.11
Expand Down
27 changes: 8 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,31 +26,20 @@ SPRAS requires

First, download or clone this repository so that you have the Snakefile, example config file, and example data.

The easiest way to install Python and the required packages is with [Anaconda](https://www.anaconda.com/download/).
The Carpentries [Anaconda installation instructions](https://carpentries.github.io/workshop-template/#python) provide guides and videos on how to install Anaconda for your operating system.
After installing Anaconda, you can run the following commands from the root directory of the `spras` repository
The easiest way to install Python and the required packages is with [Pixi](https://pixi.sh/latest/installation/).
Optionally, after installing Pixi, you can run the following commands from the root directory of the `spras` repository
```
conda env create -f environment.yml
conda activate spras
pixi shell
```
to create a conda environment with the required packages and activate that environment.
If you have a different version of Python already, you can install the specified versions of the required packages in your preferred manner instead of using Anaconda.
to create a pixi environment that contains all of the packages you need to use SPRAS without having to prefix them with `pixi run`.
If you have a different version of Python already, you can install the specified versions of the required packages, as well as `go`, in your preferred manner instead of using Pixi.

While the `spras` conda environment comes bundled with all of Python dependencies needed for `spras` to run, it does not yet have a working installation of `spras` itself.
To install `spras` in the environment, finish by running the following from the root directory of the repository:
```bash
python -m pip install .
```
Use caution when pip installing directly to your computer without using some form of virtual/conda environment as this can alter your system's underlying Python modules, which could lead to unexpected behavior.
In most cases, you should only `pip install` spras if you're already working in the `spras` conda environment!

For developers, SPRAS can be installed via `pip` with the `-e` flag, as in `python -m pip install -e .`. This points Python back to the SPRAS repo so that any changes made to the source
code are reflected in the installed module.
The `spras` pixi environment comes bundled with all of Python dependencies needed for `spras` to run, including an installation of `spras` itself.

You also need to install [Docker](https://docs.docker.com/get-docker/).
After installing Docker, start Docker before running SPRAS.

Once you have activated the conda environment and started Docker, you can run SPRAS with the example Snakemake workflow.
Once you have activated the pixi environment and started Docker, you can run SPRAS with the example Snakemake workflow.
From the root directory of the `spras` repository, run the command
```
snakemake --cores 1 --configfile config/config.yaml
Expand Down Expand Up @@ -78,7 +67,7 @@ The Docker images are available on [DockerHub](https://hub.docker.com/orgs/reedc
These wrappers are in the `spras/` subdirectory.

**Test code**: Tests for the Docker wrappers and SPRAS code.
The tests require the conda environment in `environment.yml` and Docker.
The tests require the pixi environment via `pixi shell` and Docker.
Run the tests with `pytest -s`.

## Singularity
Expand Down
4 changes: 2 additions & 2 deletions Snakefile
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ import spras.config.config as _config
SEP = '/'

wildcard_constraints:
params="params-\w+",
dataset="\w+"
params=r"params-\w+",
dataset=r"\w+"

# Elsewhere we import this as config, but in the Snakefile, the variable config is already populated
# with the parsed config.yaml. This is done by Snakemake, which magically pipes config into this file
Expand Down
4 changes: 2 additions & 2 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,10 @@ these helpful tutorials that will prime you with some basics:

## Editing SPRAS Documentation

The `spras` conda environment currently comes pre-bundled with the tools you
The `spras` pixi environment currently comes pre-bundled with the tools you
need to edit and build SPRAS's docs website, so the first step is making sure
you have an active spras environment. If you don't, refer to this repo's
[root README.md](../README.md) for instructions on installing conda and
[root README.md](../README.md) for instructions on installing pixi and
building/activating the `spras` env.

### Building And Viewing Your Changes
Expand Down
23 changes: 7 additions & 16 deletions docs/contributing/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -223,11 +223,11 @@ Also test the functions available in the ``Dataset`` class.
0 True A

Note: If you get a 'no module named' error, make sure that you are
running your interactive python session inside the SPRAS conda
running your interactive python session inside the SPRAS pixi
environment (your terminal should begin with ``(spras)`` instead of
``(base)``, which can be done through ``conda activate spras``), and
your editor's interpreter is set to using the SPRAS environment over the
base environment (on VSCode and IntelliJ editors, this should be in the
``(base)``, which can be done through ``pixi shell``), and your editor's
interpreter is set to using the SPRAS environment over the base
environment (on VSCode and IntelliJ editors, this should be in the
bottom right.)

Note the behaviors of the ``get_node_columns`` function when there are
Expand Down Expand Up @@ -308,15 +308,7 @@ through SPRAS with

snakemake --cores 1 --configfile config/config.yaml

Make sure to run the command inside the ``spras`` conda environment.

If installing via ``pip`` instead of using conda, install with the ``-e
.[dev]`` options (the full command to run from the repo root is ``python
-m pip install -e .[dev]``) so that Python picks up any changes you make
and installs all optional development packages. Omitting the ``-e`` flag
will prevent your changes from being reflected unless you force
re-install, and omitting ``.[dev]`` will prevent pip from installing
``pre-commit`` and ``pytest``.
Make sure to run the command inside the ``spras`` pixi environment.

As a workflow manager, Snakemake will consider the work described in the
configuration file to be completed once the necessary output files have
Expand Down Expand Up @@ -498,9 +490,8 @@ automatically on every commit through the GitHub Actions. However,
developers will benefit from setting up their environment to run the
same tests locally while they modify the SPRAS source.

The ``pre-commit`` package is installed as part of the conda environment
in ``environment.yml``, or when installing SPRAS with ``python -m pip
install -e .[dev]``. From there, the pre-commit `quick start
The ``pre-commit`` package is installed as part of the pixi environment
in ``pyproject.toml``. From there, the pre-commit `quick start
<https://pre-commit.com/#quick-start>`__ guide explains two primary ways
to use it locally:

Expand Down
14 changes: 7 additions & 7 deletions docs/htcondor.rst
Original file line number Diff line number Diff line change
Expand Up @@ -97,14 +97,14 @@ To run in the local CHTC pool, omit the ``+WantGlideIn`` and

Parallelizing SPRAS workflows with HTCondor requires the same setup as
the previous section, but with two additions. First, it requires an
activated SPRAS conda environment with a ``pip install``-ed version of
activated SPRAS pixi environment with a ``pip install``-ed version of
the SPRAS module (via ``pip install .`` inside the SPRAS directory).

Second, it requires an experimental executor for HTCondor that has been
forked from the upstream `HTCondor Snakemake executor
<https://github.com/htcondor/snakemake-executor-plugin-htcondor>`__.

After activating your ``spras`` conda environment and ``pip``-installing
After activating your ``spras`` pixi environment and ``pip``-installing
SPRAS, you can install the HTCondor Snakemake executor with the
following:

Expand Down Expand Up @@ -190,8 +190,8 @@ jobs, but the workflow can use Snakemake's checkpointing to pick up any
jobs where they left off.

**Note**: If you encounter an error that says ``No module named
'spras'``, make sure you've ``pip install``-ed the SPRAS module into
your conda environment.
'spras'``, make sure you've enabled the ``pixi`` environment with ``pixi
shell``.

****************
Job Monitoring
Expand Down Expand Up @@ -255,9 +255,9 @@ containing:
raise CredsError("Credentials not found for this workflow")

it indicates you must upgrade the version of the HTCondor Snakemake
executor bundled with your conda environment.
executor bundled with your pixi environment.

To upgrade, from your activated ``spras`` conda environment run:
To upgrade, from your activated ``spras`` pixi environment run:

.. code:: bash

Expand All @@ -283,4 +283,4 @@ executor repository
inspecting the commit history.

If the preceding steps did not update the installed version, you may
need to delete and rebuild your ``spras`` conda environment.
need to delete and rebuild your ``spras`` pixi environment.
17 changes: 8 additions & 9 deletions docs/install.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,19 +5,18 @@
SPRAS provides several convenient ways to use the package, depending on
your needs

***************************
Using a Conda Environment
***************************
************
Using Pixi
************

If you want to run spras locally on your machine, you can use the
provided `environment.yml` file to create a conda environment with all
the necessary dependencies: 1. `Download and Install Conda`_ 2. Build
the `spras` conda environment and activate it:
provided `pyproject.toml` file to create a `pixi` environment with all
the necessary dependencies: 1. `Download and Install Pixi`_ 2. Activate
the pixi environment:

.. code:: bash

conda env create -f environment.yml
conda activate spras
pixi shell

*****************************
Installing SPRAS with `pip`
Expand Down Expand Up @@ -49,4 +48,4 @@ image tag name. For example, to get spras v0.6.0:

docker pull reedcompbio/spras:0.6.0

.. _download and install conda: https://conda-forge.org/download/
.. _download and install pixi: https://pixi.sh/latest/installation/
19 changes: 5 additions & 14 deletions docs/tutorial/beginner.rst
Original file line number Diff line number Diff line change
Expand Up @@ -29,26 +29,17 @@ Visit the `SPRAS GitHub repository
================================

From the root directory of the SPRAS repository, create and activate the
Conda environment and install the SPRAS python package:
Pixi environment and install the SPRAS python package:

.. code:: bash

conda env create -f environment.yml
conda activate spras
python -m pip install .
pixi shell

.. note::

The first command performs a one-time installation of the SPRAS
dependencies by creating a Conda environment (an isolated space that
keeps all required packages and versions separate from your system).

The second command activates the newly created environment so you can
use these dependencies when running SPRAS; this step must be done
each time you open a new terminal session.

The last command is a one-time installation of the SPRAS package into
the environment.
This performs a one-time installation of the SPRAS dependencies by
creating a Pixi environment (an isolated space that keeps all
required packages and versions separate from your system).

0.3 Test the installation
=========================
Expand Down
5 changes: 2 additions & 3 deletions docs/tutorial/introduction.rst
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,8 @@ Together, we will cover:

Required software:

- `Conda
<https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html>`__
: for managing environments
- `Pixi <https://pixi.sh/latest/installation/>`__ : for managing
environments

- `Docker <https://www.docker.com/get-started/>`__ : for containerized
runs
Expand Down
Loading
Loading