# Cross-modal Sequences

A cross-modal sequence memorization task for MEG/EEG/ECG. Participants
learn three sequences of six items each, drawn from six semantic
categories (animals, body parts, characters, colors, landscapes, shapes)
and presented in two modalities — **images** and **words** — that
alternate across repetitions. Each sequence is shown six times per block
and is paired with a unique background sound played at every item. After
each block, participants answer six drag-to-target recall questions
(given the first item as a hint, place the probed item in its correct
slot) and rate their vigilance / focus / off-task thoughts on three
7-point Likert sliders. The experiment runs twice, ~5 min apart, with a
different set of three categories each time.

Heavy randomization (sequence order, category-to-sequence mapping,
modality alternation, target positions, rewarded vs. non-rewarded
sequences) is applied per participant to minimize cross-sequence
associations.

## Design at a glance

- **3 sequences per run × 2 runs = 6 sequences total** (named A1-F1 run 1,
  A2-F2 run 2)
- Each sequence: 6 items, each from a different category, unique
  transition structure, unique paired sound
- **4 blocks × 3 trials per block**; each sequence repeated 6× per block
  before the recall phase
- **6 recall questions per block** (one per sequence item beyond the
  first)
- Half the sequences are rewarded (CHF payout); reward mapping is
  counterbalanced
- In-lab only — triggers via parallel port (Linux `/dev/parport0`) +
  photodiode flashes synced with stimulus onset

## Requirements

- Python **3.10** (pinned in `pyproject.toml`)
- PsychoPy **2024.2.4** (pinned in `env.yml`; aligned in `pyproject.toml`)
- `byte-triggers` — parallel-port MEG/EEG triggers
- `stimuli` — low-level audio helpers (PTB backend)
- `numpy`, `pandas`, `scipy`, `matplotlib`, `seaborn`
- A parallel port on Linux for real trigger output; Windows/macOS fall
  back to a mock port

## Install

```bash
git clone git@gitlab.unige.ch:ttf-digital-data-generation/nccr-evolang-tasks/crossmodal-sequences.git
cd crossmodal-sequences
```

Either conda (recommended, uses `env.yml`):

```bash
conda env create -f env.yml
conda activate exp3
```

or pip (`pyproject.toml`):

```bash
python3.10 -m venv .venv
source .venv/bin/activate
pip install -e .
```

On Linux, system libraries for the PsychoPy audio/graphics backends are
also needed (`libportaudio2`, `libasound2-dev`, `libgl1-mesa-glx`,
`libsdl2-2.0-0`).

## Run

Entry scripts (each is independently runnable with `python <file>`):

| Script | Purpose |
|---|---|
| `main.py` | Main experiment — call twice, once per run |
| `demo.py` | Short familiarization demo for participants |
| `localizer.py` | Category localizer run (independent of the main task) |
| `bonus_question.py` | End-of-experiment memory test across all sequences |
| `resting.py` | 3-minute resting-state block |
| `sound_cal.py` | Audio volume calibration helper |
| `recover_crash.py` | Rebuild a TSV from the log after a crash |

A `DlgFromDict` dialog at startup collects participant ID, run number,
language (`fr` / `en`), and an optional starting block (for resumes).

## Controls

The response-pad keys are remapped in `sequences/params.py`:

| Action | Key |
|---|---|
| Left | `1` |
| Confirm | `2` |
| Right | `3` |
| Up | `4` |
| Down | `5` |
| Pause | `b` |
| **Master key (unblock a stuck screen)** | `m` |

Bonus-question keys differ slightly (see `key_bq` in `params.py`): up/down
are `8` / `9`, remove is `6`.

## Outputs

Main behavioural TSV written per run to
`data/output/sub-<ID>_ses-01_task-xmodseq_run-<run>_beh.tsv`
(BIDS-style filename):

| Column | Description |
|---|---|
| `ID` | Participant identifier |
| `run` | Run number (1 or 2) |
| `date` | Timestamp |
| `block` | Block number within the run |
| `trial` | Trial within the block |
| `question` | Question index within the block (1-6) |
| `modality` | `img` (image/normal font) or `txt` (word/bold font) |
| `sequence_name` | Sequence label (A1, B1, …, F2) |
| `cue` | Stimulus shown as the 5 s hint |
| `target` | Stimulus the participant was asked to place |
| `correct_answer` | Ground-truth slot index |
| `response` | Slot index chosen |
| `distance` | Slot-distance between response and correct answer |
| `correct` | 1/0 |
| `rt` | Response time (s) |

Additional files written per session: the Likert-slider ratings
(vigilance / focus / think) per block, localizer results, bonus-question
results, and a verbose log under `output/` (gitignored).

Parallel-port triggers follow a 4-band scheme defined in
`sequences/params.py` (`trig1`-`trig4`): 11-126 for sequence × position,
131-246 for category × modality, 201-232 for events (responses, pauses,
rewards), 233-234 for the localizer.

## Files

```
main.py                # main experiment (2 runs)
demo.py                # participant familiarization
localizer.py           # category localizer
bonus_question.py      # end-of-experiment recall test
resting.py             # 3 min resting state
sound_cal.py           # volume calibration
recover_crash.py       # rebuild TSV from logs after a crash
seq_structure.py       # precompute sequence structures (pickle)
seq_structure.pkl      # precomputed structures
sequences/             # reusable experiment modules
  params.py            # hardware, timings, keys, triggers, layout
  flow.py              # trial/block flow helpers
  stimuli_manager.py   # stimulus sampling, randomization, distribution
  instr.py             # instruction rendering
  common.py            # window & photodiode setup
  utils.py             # misc utilities
  bonus_q.py           # bonus-question helpers
data/input/            # stimuli (images, sounds, word lists, instructions)
env.yml                # conda env (canonical)
pyproject.toml         # pip / PEP 621 metadata
TODO.md                # author notes
```

## Attribution

Original task by **Nicolas Piron** (<nicolas.piron@unige.ch>),
University of Geneva. Upstream code:
<https://github.com/NicolasPiron/crossmodal-sequences>.

This repo was forked cleanly from upstream commit `3230b8b`; git history
was not carried across.
