Add DEAP loader with market-outcome route - #1
Merged
Merged
Conversation
DEAP is the fastest path to real recordings for this framework: 32 participants x 40 music videos, EEG + peripheral physiology on all, fully crossed subject-by-stimulus. The loader is built against the verified file format and needs only the licensed download to run. io/deap.py - Loads the preprocessed-python pickles (latin1, Python-2 origin) into three ModalityBlocks: eeg, peripheral, behavior. - Handles six format traps that each silently corrupt a result: the 4-45 Hz bandpass (delta absent; requesting it raises), the (valence, arousal, dominance, liking) label order, the 384-sample pre-trial baseline split, the latin1 pickle encoding, the non-EEG peripheral channels (own block), and the absence of ERPs in a 60 s design. - Frontal alpha asymmetry as the theory-specified feature, the DEAP analogue of the NAcc/MPFC ROIs on the fMRI side. - Recovers the per-participant randomised trial order from Experiment_id in participant_ratings.csv; without it, trial index is not a stimulus key and the aggregate arm is impossible, so the loader refuses rather than inventing keys. - behavior_mode defaults to "noncircular" (familiarity + trial order only), because the SAM ratings are same-instrument, same-session self-report and are not the choice-based comparator the neuroforecasting papers beat. "ratings" mode is available and stamps circular=True into provenance. - binarise_ratings offers fixed-threshold and subject-median splits, the single largest analytic degree of freedom in DEAP work, with the choice recorded. io/deap_market.py - Turns DEAP's YouTube stimuli into a real aggregate outcome via view counts, the move that makes DEAP a neuroforecasting rather than affect-decoding dataset. Does the id extraction and keying; does not fetch (network/ToS). - Enforces log-transform (view counts are heavily skewed) and flags the 2012 recording vs later-count temporal confound in provenance. Validation - tests/deap_fixture.py writes a synthetic download in the true on-disk format, with a latent per-video valence driving both EEG asymmetry and a synthetic market outcome. 31 tests cover every trap and the end-to-end path. - scripts/run_deap.py --demo runs the whole pipeline on that fixture and checks EEG forecasts the market better than the noncircular behaviour block (EEG oos R2 0.47 vs -0.09). Added to CI alongside the synthetic check. - Full suite: 97 tests passing. docs/deap.md documents the traps, the circularity problem, the market route, and the analytic choices that make published DEAP accuracies non-comparable.
This was referenced Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wires DEAP into the framework as the first real-data path. The loader is built against the verified DEAP file format and needs only the licensed download to run.
What's here
io/deap.pyloads the preprocessed-python pickles into threeModalityBlocks (eeg,peripheral,behavior) and handles six format traps that each silently corrupt a result:It recovers the per-participant randomised trial order from
Experiment_idinparticipant_ratings.csv. Without it, trial index is not a stimulus key and the aggregate arm is impossible, so the loader refuses rather than inventing keys.Frontal alpha asymmetry is included as the theory-specified feature, the DEAP analogue of the NAcc/MPFC ROIs on the fMRI side.
Circularity guard.
behavior_modedefaults to"noncircular"(familiarity + trial order only). DEAP's SAM ratings are same-instrument, same-session self-report, not the choice-based comparator the neuroforecasting papers beat, so a plot of ratings-beating-EEG would not mean what it appears to."ratings"mode is available and stampscircular=Trueinto provenance.io/deap_market.pyturns DEAP's YouTube stimuli into a real aggregate outcome via view counts, the move that makes DEAP a neuroforecasting dataset rather than affect decoding. It does the id extraction and keying but does not fetch (network + ToS), enforces log-transform, and flags the 2012-recording-vs-later-count temporal confound.Validation
tests/deap_fixture.pywrites a synthetic download in the true on-disk format, with a latent per-video valence driving both EEG asymmetry and a synthetic market outcome.scripts/run_deap.py --demoruns the whole pipeline on the fixture and checks EEG forecasts the market better than the noncircular behaviour block (EEG oos R² 0.47 vs -0.09). Added to CI.docs/deap.mddocuments the traps, the circularity problem, the market route, and the analytic choices that make published DEAP accuracies non-comparable.Still needs
The licensed DEAP download to run for real, and (for the aggregate arm) YouTube view counts fetched via the Data API. No real recording has touched this yet, so expect to fix things on first contact.