SLB Behavioral Data Preprocessing Pipeline

End-to-end behavioral data pipeline for SLB fMRI studies.

This pipeline converts raw PsychoPy outputs into structured, analysis-ready datasets with deterministic preprocessing, task-aware quality control, and automated export.

---

Overview

This pipeline implements a fully reproducible behavioral data workflow, starting from raw ZIP ingestion and ending with:

• Clean per-run datasets
• Task-level merged datasets
• Behavioral QC reports
• Synced outputs for downstream fMRI analysis

The pipeline enforces strict separation between:

• Raw data (immutable)
• Preprocessed data (derived)
• QC artifacts
• Exported deliverables

---

Pipeline Architecture

Stage	Script	Responsibility
0	cookiecutter (template)	Directory templating
1	extract_and_structure.py	Raw ingestion + structuring
2	preprocess_individual.py	Cleaning + feature engineering
3	merge_group_level.py	Dataset aggregation
4	qc_missed_keys_report.py	Behavioral QC
5	export_to_box.py	Data delivery

Each stage:

• Has clearly defined inputs/outputs
• Does exactly one thing
• Can be rerun independently

---

0. Template (Cookiecutter)

• Enforces participant directory structure before any data is processed
• Prevents ad-hoc folder creation
• Ensures downstream scripts never guess paths

Generated structure:

data/SLB_XXX/
  th/
  tm/
  ol/
  socialRA/

<task>/
  raw/
  preprocessed/

---

1. Extraction (extract_and_structure.py)

• Unzips participant data
• Detects task + run
• Populates Cookiecutter structure

Output:

data/SLB_XXX/<task>/raw/<run>/

---

2. Preprocessing (preprocess_individual.py)

• Cleans CSVs using task dictionaries
• Preserves all required variables
• Handles missing data safely

Output:

data/SLB_XXX/<task>/preprocessed/*_preprocessed.csv

---

3. Group Merge (merge_group_level.py)

• Aggregates across participants
• Normalizes metadata

Output:

group_level/*.csv

---

4. QC (qc_missed_keys_report.py)

• Detects missed responses only when expected
• Task-aware logic (no false positives)

Output:

qc/missed_keys_master.csv
qc/missed_keys_report.pdf

---

5. Export (export_to_box.py)

• Syncs outputs to Box

---

Running

Full pipeline:

The entire behavioral pipeline is orchestrated via DVC as a single DAG from raw ZIPs → final export.

dvc repro

Run from a specific stage:

dvc repro stage_name

Force full rerun:

dvc repro -f

Manual:

python extract_and_structure.py
python preprocess_individual.py
python merge_group_level.py
python qc_missed_keys_report.py
python export_to_box.py