Skip to content

Add DEAP loader with market-outcome route - #1

Merged
aaygan29 merged 1 commit into
mainfrom
deap-loader
Aug 7, 2026
Merged

Add DEAP loader with market-outcome route#1
aaygan29 merged 1 commit into
mainfrom
deap-loader

Conversation

@aaygan29

@aaygan29 aaygan29 commented Aug 7, 2026

Copy link
Copy Markdown
Owner

Wires DEAP into the framework as the first real-data path. The loader is built against the verified DEAP file format and needs only the licensed download to run.

What's here

io/deap.py loads the preprocessed-python pickles into three ModalityBlocks (eeg, peripheral, behavior) and handles six format traps that each silently corrupt a result:

  1. The preprocessed data is bandpass filtered 4-45 Hz, so delta is not in it. Default bands omit delta; requesting an out-of-band frequency raises rather than returning filter roll-off.
  2. Label column order is (valence, arousal, dominance, liking), pinned in one constant.
  3. Each trial is 384 baseline + 7680 trial samples; the baseline is split off and (by default) used for a per-trial, leak-safe correction.
  4. Pickles are Python-2 / latin1 encoded.
  5. Channels 33-40 are not EEG (GSR, respiration, EMG, ...) and get their own block.
  6. A 60 s music video has no ERP, so those features are disabled.

It recovers the per-participant randomised trial order from Experiment_id in participant_ratings.csv. Without it, trial index is not a stimulus key and the aggregate arm is impossible, so the loader refuses rather than inventing keys.

Frontal alpha asymmetry is included as the theory-specified feature, the DEAP analogue of the NAcc/MPFC ROIs on the fMRI side.

Circularity guard. behavior_mode defaults to "noncircular" (familiarity + trial order only). DEAP's SAM ratings are same-instrument, same-session self-report, not the choice-based comparator the neuroforecasting papers beat, so a plot of ratings-beating-EEG would not mean what it appears to. "ratings" mode is available and stamps circular=True into provenance.

io/deap_market.py turns DEAP's YouTube stimuli into a real aggregate outcome via view counts, the move that makes DEAP a neuroforecasting dataset rather than affect decoding. It does the id extraction and keying but does not fetch (network + ToS), enforces log-transform, and flags the 2012-recording-vs-later-count temporal confound.

Validation

  • tests/deap_fixture.py writes a synthetic download in the true on-disk format, with a latent per-video valence driving both EEG asymmetry and a synthetic market outcome.
  • 31 new tests, one per trap plus the end-to-end path. Full suite now 97 passing.
  • scripts/run_deap.py --demo runs the whole pipeline on the fixture and checks EEG forecasts the market better than the noncircular behaviour block (EEG oos R² 0.47 vs -0.09). Added to CI.
  • docs/deap.md documents the traps, the circularity problem, the market route, and the analytic choices that make published DEAP accuracies non-comparable.

Still needs

The licensed DEAP download to run for real, and (for the aggregate arm) YouTube view counts fetched via the Data API. No real recording has touched this yet, so expect to fix things on first contact.

DEAP is the fastest path to real recordings for this framework: 32
participants x 40 music videos, EEG + peripheral physiology on all, fully
crossed subject-by-stimulus. The loader is built against the verified file
format and needs only the licensed download to run.

io/deap.py
- Loads the preprocessed-python pickles (latin1, Python-2 origin) into three
  ModalityBlocks: eeg, peripheral, behavior.
- Handles six format traps that each silently corrupt a result: the 4-45 Hz
  bandpass (delta absent; requesting it raises), the
  (valence, arousal, dominance, liking) label order, the 384-sample pre-trial
  baseline split, the latin1 pickle encoding, the non-EEG peripheral channels
  (own block), and the absence of ERPs in a 60 s design.
- Frontal alpha asymmetry as the theory-specified feature, the DEAP analogue of
  the NAcc/MPFC ROIs on the fMRI side.
- Recovers the per-participant randomised trial order from Experiment_id in
  participant_ratings.csv; without it, trial index is not a stimulus key and the
  aggregate arm is impossible, so the loader refuses rather than inventing keys.
- behavior_mode defaults to "noncircular" (familiarity + trial order only),
  because the SAM ratings are same-instrument, same-session self-report and are
  not the choice-based comparator the neuroforecasting papers beat. "ratings"
  mode is available and stamps circular=True into provenance.
- binarise_ratings offers fixed-threshold and subject-median splits, the single
  largest analytic degree of freedom in DEAP work, with the choice recorded.

io/deap_market.py
- Turns DEAP's YouTube stimuli into a real aggregate outcome via view counts,
  the move that makes DEAP a neuroforecasting rather than affect-decoding
  dataset. Does the id extraction and keying; does not fetch (network/ToS).
- Enforces log-transform (view counts are heavily skewed) and flags the 2012
  recording vs later-count temporal confound in provenance.

Validation
- tests/deap_fixture.py writes a synthetic download in the true on-disk format,
  with a latent per-video valence driving both EEG asymmetry and a synthetic
  market outcome. 31 tests cover every trap and the end-to-end path.
- scripts/run_deap.py --demo runs the whole pipeline on that fixture and checks
  EEG forecasts the market better than the noncircular behaviour block
  (EEG oos R2 0.47 vs -0.09). Added to CI alongside the synthetic check.
- Full suite: 97 tests passing.

docs/deap.md documents the traps, the circularity problem, the market route,
and the analytic choices that make published DEAP accuracies non-comparable.
@aaygan29
aaygan29 merged commit 355932f into main Aug 7, 2026
2 checks passed
@aaygan29
aaygan29 deleted the deap-loader branch August 7, 2026 21:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant