hpc.qmd

---
title: "High-Performance Computing (Argon Supercomputer)"
---

# Overview

Analysis scripts for High Performance Computing (HPC) at the University of Iowa.
Currently uses Argon.

## Links to Documentation

Workflow: <https://workflow.uiowa.edu/entry/new/3282/11927336>

Cluster Systems Documentation:
<https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513411/Cluster+Systems+Documentation> (archived at <https://perma.cc/EKB8-ZKR7>)

Argon Cluster Documentation:
<https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513466/Argon+Cluster> (archived at <https://perma.cc/6HST-VV6Y>)

## Data Storage

Significant amounts of data storage are provided on Argon, but data are not backed up in any way unless special arrangements are made.
It is the responsibility of the user to back up important information.

## Sensitive Data

User agrees to refrain from storing Restricted data on HPC resources.
Data is classified as Restricted when the unauthorized disclosure, alteration or destruction of that data could cause a significant level of risk to the University or its affiliates.
Examples of Restricted data include data protected by state or federal privacy regulations and data pertaining to identified human subjects that has not been deidentified.

## Fees

At present there are no fees for the use of the Argon cluster for low-priority usage.
For large users, or those who want access to dedicated resources, the option of purchasing or renting supplemental system hardware may be available.
If you are interested in dedicated hardware, contact [research-computing@uiowa.edu](mailto:research-computing@uiowa.edu).

# Accessing Lab Drive

`/Shared/lss_itpetersen`

# Map Argon Drive to Local Computer

## Windows

1. Right click on `This PC` and select `Map Network Drive`
2. Select `Y` and type `\\data.hpc.uiowa.edu\argon_home\Documents`

# Accessing Argon

## SSH

`argon.hpc.uiowa.edu`

Using [SecureCRT](https://its.uiowa.edu/securecrt) or [PuTTY](https://www.chiark.greenend.org.uk/~sgtatham/putty/) for an SSH connection:

1. Download [SecureCRT](https://its.uiowa.edu/securecrt) from UIowa Informational Technology Services: <https://its.uiowa.edu/securecrt>
1. In Hostname type `argon.hpc.uiowa.edu` and in username type HawkID
1. Click connect

## Windows

Windows Explorer:

`\\data.hpc.uiowa.edu\argon_home` (username: itpetersen@uiowa.edu or hawkid@uiowa.edu)

## Mac

Mac OS Terminal:

`ssh itpetersen@argon.hpc.uiowa.edu` (on campus)

`ssh -p 40 itpetersen@argon.hpc.uiowa.edu` (off campus without VPN)

# Access Project Directories

```bash
cd /Users/itpetersen/Documents/Projects/Bayesian_IRT/
cd /Users/itpetersen/Documents/Projects/EXT_pilot/
cd /Users/itpetersen/Documents/Projects/EXT_ParentReport/
cd /Users/itpetersen/Documents/Projects/Multiple_Imputation/
cd /Users/itpetersen/Documents/Projects/SelfRegulation_IRT/
cd /Users/itpetersen/Documents/Projects/Test/
```

# Using a Container {#sec-container}

<https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513442/Singularity+Containers>

Steps:

1. If you are on Windows, install Windows Subsystem for Linux (WSL)
    - In PowerShell (Admin):
    ```bash
    wsl --install
    ```
    Restart the computer.
    It will install Ubuntu by default.
1. Update Ubuntu
    - Open Ubuntu (may need to run as administrator)
    - Inside the Ubuntu terminal, run:
    ```bash
    sudo apt update
    sudo apt upgrade -y
    ```
1. Install Apptainer inside WSL
    - Inside the Ubuntu terminal, first install dependencies:
    ```bash
    sudo apt update
    sudo apt install -y \
    build-essential \
    libseccomp-dev \
    pkg-config \
    squashfs-tools \
    cryptsetup \
    curl \
    wget \
    git \
    uidmap \
    golang-go
    ```
    - Now, install Apptainer:
    ```bash
    sudo apt install -y apptainer
    ```
    - Verify installation:
    ```bash
    apptainer --version
    ```
    - If not available via apt (`E: Unable to locate package apptainer`), install from source (specify the version number for the latest release version available; see release versions [here](https://github.com/apptainer/apptainer/releases)):
    ```bash
    export VERSION=1.4.5
    wget https://github.com/apptainer/apptainer/releases/download/v${VERSION}/apptainer-${VERSION}.tar.gz
    tar -xzf apptainer-${VERSION}.tar.gz
    cd apptainer-${VERSION}
    ```
    - If you installed from source, build Apptainer:
    ```bash
    ./mconfig
    make -C builddir
    sudo make -C builddir install
    ```
    - Verify installation:
    ```bash
    apptainer --version
    ```
1. Go to your home directory:

    ```bash
    cd ~
    ```
1. View the files/folders in this directory:

    ```bash
    ls
    ```
1. Create a `containers` folder (if you haven't already created it):

    ```bash
    mkdir containers
    ```
1. Go to the `containers` folder:

    ```bash
    cd containers
    ```
1. If you want to install `Mplus` in the container:
    - Copy the `Mplus` installer file for Linux into the build directory:
    ```bash
    cp "/mnt/c/Users/itpetersen/Downloads/MplusLinux64.bin" \
       ~/containers/
    ```
    - Verify the `containers` folder has `MplusLinux64.bin`:
    ```bash
    lss
    ```
    - Generate the install properties file:
    ```bash
    ./MplusLinux64.bin -r ~/containers/mplus_install.properties
    ```
    This will launch the installer and record your choices to the properties file.
    Go through the installation steps, and when done you'll have `mplus_install.properties` in your containers directory.
    - Edit the install path in the install properties file to `/opt/mplus` throughout:
    ```bash
    nano mplus_install.properties
    ```
    Here are the contents of my ` mplus_install.properties` file:
    
    ```bash
    # Tue Feb 24 21:04:04 CST 2026
    # Replay feature output
    # ---------------------
    # This file was built by the Replay feature of InstallAnywhere.
    # It contains variables that were set by Panels, Consoles or Custom Code.
    #Choose Install Set
    #------------------
    CHOSEN_FEATURE_LIST=Application,Examples
    CHOSEN_INSTALL_FEATURE_LIST=Application,Examples
    CHOSEN_INSTALL_SET=Typical
    #Choose Install Folder
    #---------------------
    USER_INSTALL_DIR=/opt/mplus
    #Install
    #-------
    -fileOverwrite_/opt/mplus/uninstall/uninstall_mplus.lax=Yes
    -fileOverwrite_/opt/mplus/mplus=Yes
    -fileOverwrite_/opt/mplus/Documentation/Getting_Started.pdf=Yes
    -fileOverwrite_/opt/mplus/Documentation/Mplus_Users_Guide_v8.pdf=Yes
    -fileOverwrite_/opt/mplus/Documentation/Version_8.1_Language_Addendum.pdf=Yes
    -fileOverwrite_/opt/mplus/Documentation/Version_8.5_Language_Addendum.pdf=Yes
    ```
1. Create your `R` Container Definition File
    - Inside the Ubuntu terminal, run:
    ```bash
    nano r_argon.def
    ```
    - Paste this (for `Mplus` + `R` + `brms` + `cmdstanr` installation):
    ```bash
    Bootstrap: docker
    From: rocker/r-ver:latest

    %files
        MplusLinux64.bin /opt/MplusLinux64.bin
        mplus_install.properties /opt/mplus_install.properties

    %post
        # Install system dependencies
        apt-get update && apt-get install -y \
            libcurl4-openssl-dev \
            libssl-dev \
            libxml2-dev \
            libgit2-dev \
            libfontconfig1-dev \
            libfreetype6-dev \
            libharfbuzz-dev \
            libfribidi-dev \
            libpng-dev \
            libtiff5-dev \
            libjpeg-dev \
            libglpk-dev \
            libgmp3-dev \
            libmpfr-dev \
            make \
            g++ \
            cmake \
            git \
            sudo \
            && rm -rf /var/lib/apt/lists/*

        # Install Mplus
        chmod +x /opt/MplusLinux64.bin
        /opt/MplusLinux64.bin -i silent -f /opt/mplus_install.properties
        chmod -R 755 /opt/mplus

        # Install core CRAN packages
        R -e "install.packages(c(
            'tidyverse',
            'data.table',
            'rstan',
            'brms',
            'bayesplot',
            'mgcv',
            'lme4',
            'mirt',
            'lavaan',
            'semTools',
            'MplusAutomation',
            'SimDesign',
            'fitdistrplus',
            'sn',
            'parallelly',
            'renv',
            'remotes'
        ), repos='https://cloud.r-project.org')"

        # Install cmdstanr
        R -e "install.packages('cmdstanr', repos = c('https://stan-dev.r-universe.dev','https://cloud.r-project.org'))"

        # Install latest CmdStan into /opt/cmdstan
        mkdir -p /opt/cmdstan
        R -e "cmdstanr::install_cmdstan(cores = 2, dir='/opt/cmdstan')"

        # Enable threading for within-chain parallelization
        mkdir -p /root/.R
        echo 'STAN_THREADS=true' >> /root/.Renviron
        echo 'STAN_THREADS=true' >> /root/.R/Makevars

        # Install latest version of petersenlab package from GitHub
        R -e "remotes::install_github('DevPsyLab/petersenlab')"

    %environment
        export R_LIBS_USER=/usr/local/lib/R/site-library
        export CMDSTAN=/opt/cmdstan
        export STAN_THREADS=true
        export PATH=/opt/mplus:$PATH

    %runscript
        exec R "$@"
    ```
    - Paste this (for `R` + `brms` + `cmdstanr` installation):
    ```bash
    Bootstrap: docker
    From: rocker/r-ver:latest

    %post
        # Install system dependencies
        apt-get update && apt-get install -y \
            libcurl4-openssl-dev \
            libssl-dev \
            libxml2-dev \
            libgit2-dev \
            libfontconfig1-dev \
            libfreetype6-dev \
            libharfbuzz-dev \
            libfribidi-dev \
            libpng-dev \
            libtiff5-dev \
            libjpeg-dev \
            libglpk-dev \
            libgmp3-dev \
            libmpfr-dev \
            make \
            g++ \
            cmake \
            git \
            sudo \
            && rm -rf /var/lib/apt/lists/*

        # Install core CRAN packages (specify the packages to install here)
        R -e "install.packages(c(
            'tidyverse',
            'data.table',
            'rstan',
            'brms',
            'bayesplot',
            'mgcv',
            'lme4',
            'mirt',
            'lavaan',
            'semTools',
            'MplusAutomation',
            'SimDesign',
            'fitdistrplus',
            'sn',
            'parallelly',
            'renv',
            'remotes'
        ), repos='https://cloud.r-project.org')"

        # Install cmdstanr
        R -e "install.packages('cmdstanr', repos = c('https://stan-dev.r-universe.dev','https://cloud.r-project.org'))"

        # Install latest CmdStan into /opt/cmdstan
        mkdir -p /opt/cmdstan
        R -e "cmdstanr::install_cmdstan(cores = 2, dir='/opt/cmdstan')"

        # Enable threading for within-chain parallelization
        mkdir -p /root/.R
        echo 'STAN_THREADS=true' >> /root/.Renviron
        echo 'STAN_THREADS=true' >> /root/.R/Makevars

        # Install latest version of petersenlab package from GitHub
        R -e "remotes::install_github('DevPsyLab/petersenlab')"

    %environment
        export R_LIBS_USER=/usr/local/lib/R/site-library
        export CMDSTAN=/opt/cmdstan
        export STAN_THREADS=true

    %runscript
        exec R "$@"
    ```
    - Press `Ctrl+O` to Save (i.e., "Write Out")
    - Press Enter
    - Press `Ctrl+X` to Exit
1. Build the Container
    - Inside the Ubuntu terminal, run:
    ```bash
    sudo apptainer build r_argon.sif r_argon.def
    ```
1. Test the Container Locally
    - Inside the Ubuntu terminal, run:
    ```bash
    apptainer exec r_argon.sif R
    ```
    - Check `R` version:
    ```r
    version
    ```
    - Check `R` packages:
    ```r
    library("brms")
    library("cmdstanr")
    library("petersenlab")
    ```
    - Exit `R`
    ```r
    q()
    ```
    - If you attempted to install `Mplus`, verify the `Mplus` installation:
    ```bash
    apptainer exec r_argon.sif /opt/mplus/mplus
    ```
1. Copy the Container to Your Computer
    - Inside the Ubuntu terminal, run (updating your path; this is equivalent to: `E:/Documents/OneDrive - University of Iowa/Research/Containers/`):
    ```bash
    cp ~/containers/r_argon.sif "/mnt/e/Documents/OneDrive - University of Iowa/Research/Containers/"
    ```
1. Copy the Container to Argon
    - Use WinSCP to transfer the `.sif` file to Argon (updating your path): `/old_Users/itpetersen/Documents/Containers/`
1. Test Container on Argon
    - Inside the Ubuntu terminal, run (updating your path):
    ```bash
    cd /Users/itpetersen/Documents/Containers
    apptainer exec r_argon.sif R
    ```
    - Check `R` version:
    ```r
    version
    ```
    - Check `R` packages:
    ```r
    library("brms")
    library("cmdstanr")
    library("petersenlab")
    ```
    - Exit `R`
    ```r
    q()
    ```
    - If you attempted to install `Mplus`, verify the `Mplus` installation:
    ```bash
    apptainer exec r_argon.sif /opt/mplus/mplus
    ```
1. Copy the `cmdstan` installation from the container to a writable location:
    - Inside the Ubuntu terminal, run (updating your path):
    ```bash
    mkdir -p $HOME/cmdstan_cache
    apptainer exec \
    /Users/itpetersen/Documents/Containers/r_argon.sif \
    cp -r /opt/cmdstan/. $HOME/cmdstan_cache/
    ```
1. Submit a Job Using the Container on Argon
    - e.g.:
    ```bash
    # Run R inside your container
    /usr/bin/apptainer exec /Users/itpetersen/Documents/Containers/r_argon.sif \
        Rscript my_script.R
    ```
    For an example job script using a container, see @sec-jobScript.

# Linux

## General Commands

- `pwd` print current working directory
- `cd path/to/directory` sets working directory
- `ls` print files in directory
- `module load moduleName` loads modules required for analyses; necessary to complete prior to submitting jobs involving R
- `module list` lists downloaded modules
- `qsub` submit a job script to Argon processing
- `R CMD BATCH` submit a single R script to Argon processing
Argon requires Linux-compatible file endings, which is problematic for files created outside Argon (using DOS or CRLF file endings). Use the following commands to check if your files are Argon compatible (i.e., Unix LF) and resolve incompatible file endings:
- `file myfile.job | grep CRLF` checks if a file ending uses the incompatible CRLF format. Remember to check the ending of your file; this script assumes it is .job
- `dos2unix myfile.job` converts CRLF file endings to Argon-compatible LF endings
- `find ~/DirectoryName -name '*job' -exec dos2unix "{}" \;` converts all files in a directory that end with .job to LF format; change `*job` to change other file types

## Initial run

### Load the Latest Stack

`module load stack`

### Stack 2022.2

```bash
module load stack/2022.2
module load r/4.2.2_gcc-9.5.0
cd path
```

### Installing Linux Packages to Install `R` Packages

```bash
module load stack/2022.2
module load r/4.2.2_gcc-9.5.0
module load geos/3.9.1_gcc-9.5.0
module load gdal/2.4.4_gcc-9.5.0
module load proj/5.2.0_gcc-9.5.0
module load gsl/2.7.1_gcc-9.5.0
module load nlopt/2.7.0_gcc-9.5.0
module load jags/4.3.0_gcc-9.5.0-dev
module load zlib/1.2.13_gcc-9.5.0-dev
module load cmake/3.25.0_gcc-9.5.0
module load glpk/4.65_gcc-9.5.0-dev
module load libxml2/2.10.3_gcc-9.5.0
module load r-nloptr
module load r-zlibbioc/1.44.0_gcc-9.5.0
module load r-data-table/1.14.4_gcc-9.5.0
module load r-stringi/1.7.8_gcc-9.5.0
module load r-selectr/0.4-2_gcc-9.5.0
module load r-generics/0.1.3_gcc-9.5.0
module load r-fansi/1.0.3_gcc-9.5.0
module load r-utf8/1.2.2_gcc-9.5.0
module load r-pkgconfig/2.0.3_gcc-9.5.0
module load r-gtable/0.3.1_gcc-9.5.0
module load r-scales/1.2.1_gcc-9.5.0
module load r-tzdb/0.3.0_gcc-9.5.0
module load r-timechange/0.1.1_gcc-9.5.0
module load r-dbi/1.1.3_gcc-9.5.0

```

## Load Software Stacks

<https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513440/Argon+Software+List> (archived at <https://perma.cc/WJ4Q-GDUS>)

`module load stack/2022.2`

## Job Script {#sec-jobScript}

### Using A Container

```bash
#!/bin/sh

# Set working directory
cd /Users/itpetersen/Documents/Projects/Bayesian_IRT/

# Specify qsub options
#$ -pe smp 4
#$ -M isaac-t-petersen@uiowa.edu
#$ -m eas
#$ -l mf=512G
#$ -l h_vmem=512G
#$ -cwd
#$ -q UI-HM
#$ -e /Users/itpetersen/Documents/Projects/Bayesian_IRT/Output/
#$ -o /Users/itpetersen/Documents/Projects/Bayesian_IRT/Output/

# Load the environment modules
module purge

# Run the R script inside container
apptainer exec \
    --bind $HOME/cmdstan_cache:/opt/cmdstan \
    --cleanenv \
    /Users/itpetersen/Documents/Containers/r_argon.sif \
    Rscript ./Analyses/factorScores.R
```

### Using Software Stacks

```bash
#!/bin/sh

# Set working directory
cd /Users/itpetersen/Documents/Projects/Bayesian_IRT/

# Specify qsub options
#$ -pe smp 4
#$ -M isaac-t-petersen@uiowa.edu
#$ -m eas
#$ -l mf=512G
#$ -l h_vmem=512G
#$ -cwd
#$ -q UI-HM
#$ -e /Users/itpetersen/Documents/Projects/Bayesian_IRT/Output/
#$ -o /Users/itpetersen/Documents/Projects/Bayesian_IRT/Output/

# Load the environment modules
module purge
module load stack/2022.2
module load r/4.2.2_gcc-9.5.0

# Run the R script
Rscript ./Analyses/factorScores.R
```

### qsub options {#sec-qsubOptions}

`-pe smp 4`: specify a parallel environment and number of slots (CPU cores) to be used (`smp` = shared memory parallel environment); increasing the number of slots increases the memory available for the job.
If your job gets killed due to insufficient memory, you should increase the number of slots requested (and, as needed, the `mf` and `h_vmem` flags, below).
You can estimate the memory available per slot by dividing a node's total memory by its total core count.
You can find the specific configurations for each node type on the wiki: <https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513468/Queues+and+Policies>
For instance, if you are using the UI queue that has 512GB of memory spread across 128 cores, each core comes with 4GB of memory (i.e., $512 / 128 = 4$).
Thus, if your job needs 20GB of memory, you would need to request at least 5 cores (i.e., $20 / 4 = 5$).

- The `OMP_NUM_THREADS` variable is set to '1' by default.
If your code can take advantage of the threading then specify `OMP_NUM_THREADS` to be equal to the number of job cores per node requested.

`-M isaac-t-petersen@uiowa.edu`: Set the email address to receive email about jobs.
This must be your University of Iowa email address.

`-m eas`: Specify when to send an email message (; ; ; ; )

- `b` = beginning of job
- `e` = end of job
- `a` = when job is aborted
- `s` = when job is suspended
- `n` = no mail is sent

`-l mf=512G`: request a particular quantity of memory you expect to use (to be available for your computation to start; the request is only applicable at scheduling time.)
The `mem_free` flag tells the scheduler to find a node that has at least that much memory available at the moment your job starts.
However, it does not reserve or lock that memory for your job's exclusive use while it is running.
To ensure your job has a guaranteed memory allocation, you should increase the number of slots requested (see above: i.e., `-pe smp <slots>`).

`-l h_vmem=512G`: request the maximum quantity of virtual memory you expect to need; `h_vmem` is a hard limit.
It acts as a ceiling.
If your job exceeds this amount of memory, the system will terminate the process to protect the node's stability.
However, it does not reserve or lock that memory for your job's exclusive use while it is running.
To ensure your job has a guaranteed memory allocation, you should increase the number of slots requested (see above: i.e., `-pe smp <slots>`).

`-cwd`: Determines whether the job will be executed from the current working directory.
If not specified, the job will be run from your home directory.

`-q UI-HM`: specify queue

`-e /Users/itpetersen/Documents/Projects/Bayesian_IRT/Output/`: Name of a file or directory for standard error.

`-o /Users/itpetersen/Documents/Projects/Bayesian_IRT/Output/`: Name of a file or directory for standard output.

## Submit Job

<https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513450/Basic+Job+Submission> (archived at <https://perma.cc/2SS2-LEJR>)

<https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513452/Advanced+Job+Submission> (archived at <https://perma.cc/8H6G-2M2F>)

`cd path/to/dataSet123`

[cd /Users/itpetersen/Documents/Projects/Test/Jobs]

`qsub myscript.job`

Job dependency (run `Job B` when `Job A` is finished):

`qsub -hold_jid JOB_ID test_B.job`

### Bayesian IRT

`cd /Users/itpetersen/Documents/Projects/Bayesian_IRT/Jobs`

`qsub bayesianIRT.job`

`qsub -hold_jid JOB_ID factorScores.job`

### Multiple Imputation

`cd /Users/itpetersen/Documents/Projects/Multiple_Imputation/Jobs`

`qsub srs_selfRegulation.R`

## Queues

<https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513468/Queues+and+Policies> (archived at <https://perma.cc/UUR7-XLBZ>)

`UI`

`UI-HM`

`UI-GPU-HM`

`UI-DEVELOP`

`all.q`

## Queue Status

`qstat -g c`

`qstat -g c -q QUEUE_NAME`

`qstat -g c -q UI`

## Job Status

`qstat -u itpetersen`

`qstat -j JOB_ID`

## Job Time and Memory

- Terminated jobs: `qacct -j JOB_ID`
- Jobs in progress: `qstat -j JOB_ID | grep usage`

## Cancel Job

`qdel JOB_ID`

## Run `R` Script

See [here](#sec-runRscript)

# R

<https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76514707/R+Programs+in+Batch+mode+for+HPC> (archived at <https://perma.cc/99JQ-43ZG>)

## Find which versions of `R` are installed

`module spider R`

But, there may be more recent version of `R` installed in the "Additional Software Stacks" (<https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513440/Argon+Software+List>; archived at <https://perma.cc/WJ4Q-GDUS>)

## Compile `R`

If you want to compile a more recent version of `R` than is available in the software stacks, see [here](https://brandonlebeau.org/2022/06/02/compile-hpc-iowa/) (archived at <https://perma.cc/C6EX-EZL4>).
However, we now compile the most recent version of `R` (and packages) using a [container](#sec-container).

## Install `R` packages

<https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76514803/Adding+R+programs+to+a+personal+library> (archived at <https://perma.cc/3SRR-2JE7>)

1. first load the `R` environment module
    - `module load r/4.2.2_gcc-9.5.0`
1. launch R
    - `R`
1. then install the package, (the code below uses a generic `package_name`):
    - `install.packages("package_name", repos = "http://cran.r-project.org", dependencies = TRUE, type = "source", Ncpus = 40)`
    - If using `packrat`, it is preferable to install packages by source, if possible, but you can remove `type = source` if you want to install binaries
    - Assuming that you do not have a personal library directory you will see

    ```r
    Warning in install.packages("package_name", repos = "http://cran.r-project.org") :
    lib = "/opt/R/3.0.2/lib64/R/library"' is not writable

    Would you like to use a personal library instead?  (y/n)
    ```

1. Select `y`
1. Select `y` again when prompted to create the directory; your package should download and install into your personal library directory

Example:

```{r}
#| eval: false

install.packages(c(
    "renv","psych","tidyverse","data.table","nlme","lme4","mirt","TeachingDemos","Amelia","mice","miceadds","abind","future","lavaan","blavaan","Rcpp","igraph","shinystan","StanHeaders","brms","rstan","rjags"),
    repos = "http://cran.r-project.org",
    dependencies = TRUE,
    Ncpus = 40)
```

### Install `R` package locally from source

`install.packages(path_to_file, repos = NULL, type = "source", Ncpus = 40)`

Example:

```{r}
#| eval: false

install.packages(c(
    "renv","psych","tidyverse","data.table","nlme","lme4","mirt","TeachingDemos","Amelia","mice","miceadds","abind","future","lavaan","blavaan","Rcpp","igraph","shinystan","StanHeaders","brms","rstan","rjags"),
    repos = "http://cran.r-project.org",
    type = "source",
    dependencies = TRUE,
    Ncpus = 40)
```

### Managing packages for a project using `renv` {#sec-renv}

1. Create a directory where you want you want to store the project.
1. `cd` to the above directory
1. Start R
1. Load the `renv` package: `library("renv")`
1. At the `R` prompt, initialize the `renv` project on the local repository of `R` packages with:
    - `renv::init(project = "/Users/itpetersen/Documents/Projects/SelfRegulation_IRT/")`
    - or `renv::init()` if you are in the intended working directory
1. You must restart your `R` session in the given project directory after running init in order for the changes to take effect!
1. From this point on you are working in a `renv` project.
Installed packages will go into a library within this project.
After initializing the `renv` project on the local repository of `R` packages, packages from the local repository can be installed with `renv::install()`:
    - `renv::install("package_name")`
1. To restart a `renv` project, simply start `R` from the directory created in step (1).
The project will initialize automatically.

To update version of `renv`: `renv::upgrade()`

To install packages: `renv::install("package_name")`

To update packages: `renv::update()`

To save the current state of your library: `renv::snapshot()`

To restore the state of your library from the lock file: `renv::restore()`

To disable `renv` on a project: `renv::deactivate()`

#### To control which packages to install with `renv` using a `DESCRIPTION` file

If you want to control which packages are installed in a `renv` project, you can use a `DESCRIPTION` file to specify the packages that should be installed:

1. Create a `DESCRIPTION` file in the project directory with the following format:

    ```yaml
    Type: project
    Description: My project.
    Depends:
        packageName1,
        packageName2,
        packageName3
    ```

1. Run `renv::settings$snapshot.type("explicit")` to suppress dependency discover and to enable "explicit" mode: <https://rstudio.github.io/renv/reference/dependencies.html#explicit-dependencies>
1. Run `renv::init(bare=TRUE)` to initialize the project without attempting to discover and install `R` package dependencies

### To install a new package for a project after `renv` has been initialized

1. First, load the `R` environment module (see `Install R packages` section above)
1. cd to the directory of the `R` project
1. launch R
    - `R`
1. then install the package, (the code below uses a generic `package_name`):
    - `install.packages("package_name", repos = "http://cran.r-project.org", type = "source", dependencies = TRUE, Ncpus = 40)`
    - `install.packages(c("renv","psych","tidyverse","data.table","nlme","lme4","mirt","TeachingDemos","Amelia","mice","miceadds","abind","future","lavaan","blavaan","Rcpp","igraph","shinystan","StanHeaders","brms","rstan","rjags","renv"), repos = "http://cran.r-project.org", type = "source", dependencies = TRUE, Ncpus = 40)`
1. `renv::snapshot()`

## Update Packages

```{r}
#| eval: false

update.packages(ask = FALSE)
```

## Run `R` Script {#sec-runRscript}

`module load r/4.0.5_gcc-9.3.0`

`cd path/to/dataSet123`

[cd /Users/itpetersen/Documents/Projects/Test/Analyses]

`Rscript path/to/program.R`

[Rscript test.R]

## Quit R

`q()`

# Stan

<https://github.com/stan-dev/rstan/wiki/Installing-RStan-on-Linux> (archived at <https://perma.cc/89H9-L8S6>)

Makevars:

`\\data.hpc.uiowa.edu\argon_home\.R\Makevars`



# Troubleshooting

## Running out of Memory {#sec-outOfMemory}

May need to request more memory, either by requesting more slots (to ensure more memory; i.e., `-pe smp <SLOTS>`), by increasing how much memory is available for the node at the moment your job starts (i.e., `mf=<SIZE>`), and/or by increasing the hard limit for virtual memory (i.e., `h_vmem=<SIZE>`); see @sec-qsubOptions.

To see how much memory was used by a killed job, run `qacct -j JOBID` and look at: `maxvmem`.

## `job [NUMBER] died through signal TERM (15)`

May have run out of memory.
See @sec-outOfMemory.

## `job [NUMBER] died through signal KILL (9)`

May have run out of memory.
See @sec-outOfMemory.

## Exit Status: 143

May have run out of memory.
See @sec-outOfMemory.

## Exit Status: 137

May have run out of memory.
See @sec-outOfMemory.

## Having trouble installing packages?

Sometimes Argon will fail to install packages in an R workspace.
We advise the following steps:

1. Sometimes R provides an error stating something like "fatal error: modulename: No such file or directory".
In this instance, you may wish to exist R with q() and load all the modules that begin with the listed module name, as listed above in "Installing Linux Packages to Install R Packages".
Then reopen R and try to install the package again.
1. You can also try installing the package from binary using the RStudio Package Manager (RSPM):

    ```r
    remotes::install_github("cran4linux/rspm")
    rspm::enable()
    install.packages("PACKAGE_NAME")
    ```

1. If that does not work, you can try downloading the .tar file directly from the CRAN repository. Then copy the file into your Argon folder and type 'install.packages("/Users/path/to/directory/package_name", repos = NULL, type = "source")'.
Note that you may need to download an older version of the package (e.g., from the [CRAN Archive](https://cran.r-project.org/src/contrib/Archive/)), such that it is compatible with the version of R you are running on Argon (which it typically not the most recent R version):

    ```r
    install.packages("http://cran.r-project.org/src/contrib/Archive/MASS/MASS_7.3-60.0.1.tar.gz", repos = NULL, type = "source")
    ```

1. Check if you are installing the necessary modules within your job script: 

    ```r
    module load stack/2022.2
    module load r/4.2.2_gcc-9.5.0
    ```
1. Add a line to your R script to make sure it is accessing the same repository where you installed the packages manually:

    ```r
    .libPaths(c("/old_Users/HAWKID/Rlibs", 
            "/cvmfs/argon.hpc.uiowa.edu/2022.2/apps/linux-centos7-broadwell/gcc-9.5.0/r-4.2.2-lv7yirk/rlib/R/library"))
    ```

1. If the prior steps fail, email research-computing@uiowa.edu with the error, asking for help to figure out how to install the packages.


## Error When Installing Packages from Source

You may have to set environment variables in each module file to help the compiler find headers and libraries.
Note that if you run into any C++ code, you will need to set the `CPLUS_INCLUDE_PATH` variable.

### `nloptr`

1. Run the following commands

```bash
module load stack/2022.1
module load r/4.1.3_gcc-9.4.0
module load nlopt
```

2. Set `LIBRARY_PATH` so linker can find library while launching `R` session (single line below):

```
LIBRARY_PATH=$ROOT_NLOPT/lib64:$LIBRARY_PATH R
```

3. In `R` console, install `nloptr` (two lines below)

```{r}
#| eval: false

install.packages(verbose = 1, "nloptr")
```

### `tkrplot`

1. Run the following commands

```bash
module load stack/2022.1
module load r/4.1.3_gcc-9.4.0
```

2. Set LIBRARY_PATH so linker can find library while launching `R` session (single line below):

```
C_INCLUDE_PATH=$ROOT_XPROTO/include LIBRARY_PATH=$ROOT_LIBXEXT/lib:$ROOT_LIBXSCRNSAVER/lib R
```

3. In `R` console, install `tkrplot` (two lines below)

```{r}
#| eval: false

install.packages(verbose = 1, "tkrplot")
```

## Error Reading from Connection

```r
Error in unserialize(node$con) : error reading from connection
Calls: parlmice ... FUN -> recvData -> recvData.SOCK0node -> unserialize
```

There likely wasn't sufficient memory for a given core.
Try increasing the max memory available and decreasing the number of cores and/or slots, so there is more memory available per core:

<https://stackoverflow.com/questions/46186375/r-parallel-error-in-unserializenodecon-in-hpc> (archived at <https://perma.cc/MF6V-NAVS>)

<https://stackoverflow.com/questions/17015598/error-calling-serialize-r-function> (archived at <https://perma.cc/3Q75-DA2D>)

<https://gforge.se/2015/02/how-to-go-parallel-in-r-basics-tips/#Memory_load> (archived at <https://perma.cc/2JRF-8Y5F>)

## Error: `$'\r': command not found`

```bash
/opt/sge/default/spool/argon-itf-bx47-36/job_scripts/5685660: line 2: $'\r': command not found
```

That often means your job script has Windows (CRLF) line endings instead of Unix (LF).
You can fix the line endings with the following command:

```bash
dos2unix your_job_file.job
```

# Appendix {#sec-appendix}

## Old Stacks

### Stack 2022.1

```bash
module load stack/2022.1
module load r/4.1.3_gcc-9.4.0
cd path
```

### Stack 2021.1

```bash
module load stack/2021.1
module load r/4.0.5_gcc-9.3.0
cd path
```

### Stack 2020.2

```bash
module load stack/2020.2
module load r/4.0.2_gcc-8.4.0
cd path
```

### Stack 2020.1

```bash
module load stack/2020.1
module load r/3.6.2_gcc-9.2.0
cd path
```

## Managing packages for a project using `packrat`

Please note: we now use [`renv`](#sec-renv) rather than `packrat` for package management

<https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76514803/Adding+R+programs+to+a+personal+library> (archived at <https://perma.cc/3SRR-2JE7>)

1. Create a directory where you want you want to store the project.
1. `cd` to the above directory
1. Start R
1. Load the `packrat` package: `library("packrat")`
1. At the `R` prompt, initialize the `packrat` project on the local repository of `R` packages with: `packrat::init(project = "/Users/itpetersen/Documents/Projects/INSERT_PROJECT_NAME/", options = list(local.repos = "/Users/itpetersen/R/x86_64-pc-linux-gnu-library/4.0"))`
1. You must restart your `R` session in the given project directory after running init in order for the changes to take effect!
1. From this point on you are working in a `packrat` project.
Installed packages will go into a library within this project.
After initializing the `packrat` project on the local repository of `R` packages, packages from the local repository can be installed with `packrat::install_local()`:
    - `packrat::install_local("package_name")`
1. To restart a packrat project simply start `R` from the directory created in step (1). The project will initialize automatically.

To save the current state of your library: `packrat::snapshot()`; if that command fails due to an error when fetching sources, try `packrat::snapshot(snapshot.sources = FALSE)` 

To disable `packrat` on a project: `disable(restart = FALSE)`

### To install a new package for a project after `packrat` has been initialized

1. First, load the `R` environment module (see `Install R packages` section above)
1. cd to the directory of the `R` project
1. launch R
    - `R`
1. then install the package, (the code below uses a generic `package_name`):
    - `install.packages("package_name", repos = "http://cran.r-project.org", type = "source", dependencies = TRUE, Ncpus = 40)`
    - `install.packages(c("packrat","psych","tidyverse","data.table","nlme","lme4","mirt","TeachingDemos","Amelia","mice","miceadds","abind","future","lavaan","blavaan","Rcpp","igraph","shinystan","StanHeaders","brms","rstan","rjags","renv"), repos = "http://cran.r-project.org", type = "source", dependencies = TRUE, Ncpus = 40)`
1. `packrat::snapshot()`; if that command fails due to an error when fetching sources, try `packrat::snapshot(snapshot.sources = FALSE)`

### Build `packrat` environment on a compute node (only if necessary if package load fails due to issues building packages)

1. `qlogin` -- for more info, see here: <https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513454/Qlogin+for+Interactive+Sessions> (archived at <https://perma.cc/Y3A9-WQ3W>)
1. `module load stack/2020.2-base_arch` -- this will ensure that the modules point to the lowest common multiarchitecture on Argon and will run on all nodes
1. `module load r`
1. cd to the directory of the `R` project
1. `R`
1. Create `packrat` project (see above)
1. Install packages
1. Take a package snapshot using `packrat`
1. `exit` -- to exit the qlogin session

Note: Your `packrat` environment will then be linked to the proper `glpk` library and will run on any Argon node.
You do not need to use the `2020.2-base_arch` module at run time, only build time.