Holistic QC Pipeline

A modular rewrite of the BOOST behavioral quality-control (QC) pipeline. The repo pulls raw JATOS exports, normalizes them into tidy data frames, applies construct-specific QC, and persists both participant-level artifacts and aggregate dashboards for downstream analysts.

Highlights

• Single entrypoint (beh/main_handler.py) that coordinates pulling raw studies, CSV conversion, QC, persistence, and plotting.
• Domain-specific QC modules for the core cognitive constructs: cognitive control (CC), psychomotor speed (PS), memory (MEM), and word learning (WL).
• Automatic artifact management: raw outputs land under data/, aggregated summaries in meta/, and generated plots in per-subject folders (with exemplar group views retained in group/plots/).
• Ready to automate: python code/main_handler.py all mirrors the GitHub Action and is safe to schedule.

Repository Layout

beh/
  main_handler.py        # Orchestrates end-to-end QC for a task or the full battery
  data_processing/
    pull_handler.py      # Pulls fresh JATOS exports by study IDs
    utils.py             # Shared helpers (CSV normalization, accuracy/RT math, WL fuzzy matching)
    save_utils.py        # Writes subject artifacts (CSV + plots) into the data lake structure
    cc_qc.py             # CC task QC rules (AF/NF/NTS/ATS/NNB/VNB)
    ps_qc.py             # PS task QC rules (PC/LC/DSST)
    mem_qc.py            # Working memory QC rules (FN/SM)
    wl_qc.py             # Word learning QC rules (WL/DWL + delay reconciliation)
    plot_utils.py        # Matplotlib/seaborn helpers for construct-specific visualizations
  transfer/
    path_logic.py        # Optional helper to mirror generated outputs onto the BOOST file server

data/                   # Subject-level caches (obs/int sites, then subject/task/data|plot)
meta/                   # Aggregate CSVs rebuilt via META_RECREATE (cc_master, mem_master, ps_master, wl_master[_wide])
group/plots/            # Example construct plots for quick reference
requirements.txt        # Python dependencies for QC + plotting
run.py                  # Flask placeholder (not yet active)

Data & QC Flow

1. Pull – Pull in pull_handler.py requests study metadata + data blobs from JATOS for the study IDs defined in Handler.IDs. days_ago defaults to 127 but can be overridden when calling load().
2. Normalize – CONVERT_TO_CSV flattens newline-delimited JSON into tidy Pandas frames ready for QC.
3. QC & Metrics – Handler.choose_construct() routes each task to its construct-specific QC class:
   • CCqC enforces max RT checks, per-condition accuracy thresholds, and task-switching rules.
   • PS_QC scores psychomotor speed blocks and tallies correct counts.
   • MEM_QC inspects FN/SM performance with RT + accuracy rollups.
   • WL_QC orchestrates fuzzy matching against version-specific keys, handling WL and DWL simultaneously.
4. Visualize – plot_utils generates construct-appropriate figures (per-condition counts, RT distributions, WL learning curves, etc.).
5. Persist – SAVE_EVERYTHING stores per-participant CSVs and plots under data/<study>/<site>/<subject>/<task>/. Once the task artifacts are saved, META_RECREATE is invoked for every domain so the aggregate CSVs in meta/ stay synchronized with the subject-level cache.

Supported Tasks

Construct	Tasks	Notes
CC (Cognitive Control)	AF, NF, ATS, NTS, NNB, VNB	Shared QC thresholds at 50% accuracy, optional task-switching logic for ATS/NTS
PS (Psychomotor Speed)	PC, LC, DSST	Separate RT limits for LC/PC vs DSST; exports accuracy and correct-count masters
MEM (Face/Scene Memory)	FN, SM	Captures per-condition accuracy, mean RT, and counts into mem_master.csv
WL (Word Learning + Delayed)	WL, DWL	Combines learning/distraction/immediate blocks with delayed recall; masters upsert rows per subject/session

To target a single task, run python beh/main_handler.py WL. To mirror the nightly sweep, use python beh/main_handler.py all.

Setup

1. Create a virtual environment and install dependencies:

   python -m venv .venv
   source .venv/bin/activate
   pip install -r requirements.txt

2. (Optional) If you are on Nix, nix develop provisions the toolchain.
3. Configure secrets:
   • Handler.pull() currently references a token inline. Replace with an environment variable (e.g., JATOS_TOKEN) and export it before running.
   • Proxy credentials (tease) should also come from the environment or an .env file that is not committed.

Running QC Locally

# QC the full battery (mirrors CI)
python beh/main_handler.py all

# QC a single construct
python beh/main_handler.py AF

Outputs land under data/ using the subject -> task folder pattern enforced by SAVE_EVERYTHING. Every run also refreshes the aggregated CSVs in meta/ via META_RECREATE:

• cc_master.csv: condition-level accuracy + mean RT for CC tasks.
• mem_master.csv: joined counts/RT/accuracy for FN/SM.
• ps_master.csv: per-block correct counts for PS tasks.
• wl_master_wide.csv & wl_master.csv: wide vs flattened WL summaries combining WL + DWL submissions.

Visual Artifacts

• Participant plots are co-located with their data under data/.../plot/.
• Shared reference visuals live in group/plots/ (e.g., flanker.png, task_switching.png) for quick distribution in slide decks.

Transferring Data to the Server

beh/transfer/path_logic.py discovers local subject folders and mirrors them to /mnt/lss/Projects/BOOST (observational vs intervention sites routed automatically). Use PathLogic.copy_subjects_to_server(max_workers=?, dry_run=True) inside a Python shell to preview the copy plan before executing.

Development Workflow

• Lint with python -m flake8 tests before committing.
• Run pytest --collect-only from repo root as a quick smoke check for test discovery.
• Run pytest (tests live under tests/) to cover threshold logic, expected artifact names, and any new utilities.
• Run pytest tests/e2e -q to target only e2e coverage.
• Run bash scripts/ci_check.sh to execute the same local checks used by CI.
• Keep notebooks or ad-hoc experiments outside tracked directories, or convert them into reproducible scripts.

Testing Commands (Local)

# lint
python -m flake8 tests

# discovery smoke check
pytest --collect-only

# full local test run
pytest

# e2e-only run
pytest tests/e2e -q

# CI-equivalent local checks
bash scripts/ci_check.sh

Test Naming & Extension Pattern

• Domain starter tests follow tests/data_processing/<domain>/test_<domain>_starter.py (for example: cc, mem, ps, wl).
• New domain tests should use test_<domain>_<behavior>.py so discovery remains predictable.
• Fixtures live under tests/fixtures/ with task-specific folders (for example: tests/fixtures/af_known_task/).
• Reuse shared builders/assertions from tests/data_processing/helpers/ to avoid duplicating fixture shaping and dataframe checks.

Extending the Pipeline

1. Add the new task code and study IDs to Handler.IDs.
2. Implement construct logic under beh/data_processing/ (reuse helpers in utils.py when possible).
3. Register the new branch in Handler.choose_construct() and extend META_RECREATE if new aggregate metrics are required.
4. Document the task behavior and update tests/fixtures to reflect the new data expectations.

Troubleshooting

• No data returned from JATOS: confirm the study IDs in Handler.IDs and that your token has access; adjust the days_ago window if you are backfilling.
• Missing session folders: ensure input CSVs include session or session_number. SAVE_EVERYTHING uses those columns to label artifacts.
• WL metrics look stale: Rerun both WL and DWL so their subject CSVs exist before META_RECREATE rebuilds the wide/flat summaries.

License & Data Privacy

This repository processes sensitive participant responses. Keep tokens, raw exports, and downstream artifacts off public machines. Add new temp/output folders to .gitignore as needed to avoid leaking data.