Full guide#

Every stage of the segmentation pipeline with the choices put back in: how a workflow is split into a search and a cross-validation, what a sample is, how the materialised dataset is laid out, which architectures exist and how to compare them, and how predictions come back to the GUI. Quickstart is this page with every choice made for you; Config reference lists every key.

import ethograph as eto  # after installing PyTorch, then `uv pip install "ethograph[model]"`

project = eto.segment.Project("project.yaml")
project.materialise()  # feature engineering → materialised dataset

# Stage 1 — find good settings on a 60/20/20 split of the trials
best = project.search()

# Stage 2 — cross-validate them, one fold per session, and review the
# predictions in the GUI
eto.segment.Project(best.config_path).cross_validate()

The two stages of a workflow#

The pipeline has one set of stages you run (materialise, train, infer) and two ways to use them. Which one you are in decides how the trials are divided, and that is the only thing that changes between them.

Stage 1 — search

Stage 2 — cross-validate

Question

What settings work?

Where is the model still wrong?

Split

The trials of every session, pooled and cut 60/20/20 by train.split.

One whole session held out per fold; every other session trains.

Chooses on

The validation trials — that is all validation is for.

Nothing. The settings are already fixed.

Call

project.search()

project.cross_validate()

Gives you

searches/{name}/best.yaml — a config inheriting yours with the winning parameters pinned.

A prediction set beside every session, each written by a model that never saw it. Load them in the GUI.

In stage 2 (cross-validation), we recommend taking an entire behavioural session as the held-out test set. This is convenient as you can use the GUI to open any trial in that session and compare your ground truth labels with predictions visually.

The stages themselves#

Stage

What it does, and what it writes

Feature creation

project.materialise()

Selects every configured feature column of every sample (one trial × one individual), applies the fixed preprocessing chain, encodes the branch’s curated labels per frame.

Writes {root}/data/{name}/, the materialised dataset.

Train

project.train()

Fits an architecture on the training samples, validates every few epochs, keeps the best checkpoint, evaluates the test samples once (raw and post-processed). Materialises first if needed. Returns a RunResult (run_dir, best_epoch, best_score, test_metrics).

Writes {root}/runs/{run}/: config, layout, stats, weights, metrics.

Search

project.search()

Optuna over search.params: each trial is a training run, scored by train.select_on on the validation trials. Resumable — the study lives in a SQLite file. Returns a SearchResult.

Writes {root}/searches/{name}/: study.db, trials.tsv, best.yaml.

Cross-validate

project.cross_validate()

One fold per session: train on the rest, predict the held-out one. folds= runs only some of them. Returns one DataFrame row per fold.

Writes {root}/cross_validation/{name}/folds.tsv, plus a prediction set per held-out session.

Inference

project.inference()

Runs a run over the sessions and post-processes the predictions (purge → stitch → snap to changepoints → purge). Returns the prediction paths; run= picks another run, sessions= narrows to a few.

Writes labels/predictions_{run}_{timestamp}/ in each session folder: {stem}_predictions.tsv plus _probs.npz.

project.config is the resolved config and project.root the project directory; project.sessions() opens every session, project.runs() names the runs already trained, and project.load_run() returns a trained run ready to predict with.

Feature engineering is done by user, not the pipeline#

The pipeline only selects variables already in the session file, so anything a model should see is a variable you add (and can plot in the GUI). Four config sections are the exception, derived at session open:

Section

Reads

Generates

features.sin_cos

an angle

(sin, cos), never z-scored

features.changepoint_features

a changepoint mask

proximity / offset / segment-length columns

features.neural

a pynapple TsGroup of spikes

a binned TsdFrame

features.label_inputs

curated labels of another branch

an indicator per state class, a Laplacian bump per point class

A changepoint mask is a binary variable with attrs["changepoint_mask"] = 1, as the GUI’s Detect button writes. features.changepoint_features and features.label_inputs are currently supported for .nc sessions only, not pynapple / NWB.

Reuse the labels you already have. If one question is answered — a behaviour labelled and curated across every session — and the next one is finer (a moment inside a state, an event that only follows another), put the new labels on a new branch and list the old branch under features.label_inputs: its labels become input columns, so the new model learns when the new event can happen from work already done. The input branch is never the target branch; the config refuses the overlap. See features.label_inputs.

Feature engineering may look like:

import ethograph as eto
from ethograph.features.changepoints import add_changepoint_features
from ethograph.features.geometry import egocentric_position, heading, intra_distances

ds["position_ego"] = egocentric_position(ds["position"], "body", heading_keypoint="head")
ds["intra"] = intra_distances(ds["position"])
ds["heading"] = heading(ds["position"], "body", "head")  # unit vector: attrs normalise=0
ds = add_changepoint_features(ds, sigmas=[2, 3, 5])  # *_cp_binary, *_cp_sigma3, …

Every function in ethograph.features.geometry takes and returns xarray (the movement convention: position (time, space, keypoint, individual)), works in 2-D and 3-D, and keeps the individual dim. Outputs that must never be z-scored (unit vectors, angles, binary flags, segment ids) carry attrs["normalise"] = 0; the pipeline honours it.

See Config reference for each section’s keys.

Video features are extracted by the pipeline too, but written into the session rather than derived at open — a pretrained network (S3D [Xie et al., 2018] by default, a timm backbone such as DINOv2 [Oquab et al., 2024] by name) is expensive enough to run once per video and cache. See Video features; the short version is

# a folder of videos, before any session exists
eto.segment.extract_videos(["/data/videos"], "/data/features", stack_s=0.5)

# or: the videos this config's sessions already name, then merge them in
project.video_features(merge=True)

which leaves an ordinary s3d (time, s3d_dims) variable on each trial — plottable in the GUI, and named in features.columns like anything else.

A network that cannot run in this environment is a video feature too: extractor: feral exports the sessions’ videos and curated labels for FERAL [Skovorodnikov et al., 2025], which fine-tunes in its own environment, and its per-frame embeddings attach as the variable feral when they come back. Its sessions the model may not have seen are named in train.split.holdout_sessions, which is what keeps the test score honest — see FERAL.

A config#

root: .                       # data/ and runs/ live here (default: this file's folder)

sessions:                     # a session is a folder; labels_path only when the file is elsewhere
  - source: ../sub-01/ses-01/behav                      # the session folder: .ethograph/, labels.tsv, its .nc layers
    video_dir: ../videos
  - source: ../sub-01/ses-02/behav
    video_dir: ../videos
  - source: ../sub-01/ses-03/behav
    video_dir: ../videos
  - source: ../sub-02/ses-01/behav
    video_dir: ../videos

trials:
  where: {num_pellets: [1, 2]}     # the metadata-table filter, every stage

features:
  name: kin_cp                     # → data/kin_cp/
  columns:                         # feature → dim → values; the individual dim is never listed
    position_ego: {space: [x, y, z], keypoint: [beakTip, stickTip]}
    speed:        {keypoint: [beakTip]}
    speed_cp_sigma3: {keypoint: [beakTip]}
    inter_beak:   {other: "*"}     # a second individual dim: the others, in order
  preprocess:
    likelihood_threshold: 0.6      # keypoint frames with `confidence` below this → NaN → interpolated
    clip_percentiles: [2, 98]
    zscore: true                   # statistics from the training samples only
  labels:
    mapping: mapping.txt
    branch: 0                      # one model per branch

model:
  architecture: c2f_tcn            # eto.segment.architectures()
  params: {num_f_maps: 128}

train:
  run_name: c2f_kin_cp
  epochs: 100
  eval_every: 5
  select_on: f1@50               # what val is scored on, and what a search maximises
  augment: {noise_std: 0.05, stretch: [0.8, 1.2], mirror: false, rotate_deg: 0}
  # the three ratios, drawn by whole trial across every session; they must sum to 1
  split: {train_fraction: 0.6, val_fraction: 0.2, test_fraction: 0.2}

search:                          # stage 1 — keys are the same dotted paths an override uses
  n_trials: 30
  params:
    train.loss.alpha: {type: float, low: 1.0e-5, high: 1.0e-2, log: true}
    train.loss.focal: {type: categorical, choices: [true, false]}
    train.loss.weights: {type: categorical, choices: [null, dataset_inverse_weights]}
    model.params.num_f_maps: {type: int, low: 32, high: 128, log: true}

infer:
  postprocess:
    min_duration_s: 0.05           # drop predicted labels shorter than this (0 = off)
    stitch_gap_s: 0.0              # merge same-label predictions separated by less than this (0 = off)

Every key is documented in Config reference. Two conveniences: base: other.yaml merges a file over another, and any key can be overridden without editing the file, using the same dotted spelling the YAML has — passed to the constructor or accumulated with update():

project = eto.segment.Project("project.yaml", "model.architecture=mstcn")
project.update("train.run_name=mstcn", "train.loss.gamma=2")

Values are parsed as YAML, and the config is rebuilt from the file each time, so a typo is caught there rather than half-way through a run. That is how a benchmark is written — one base file, a loop over overrides, then compare():

for architecture in ("c2f_tcn", "mstcn", "mlp"):
    eto.segment.Project(
        "project.yaml",
        f"model.architecture={architecture}",
        f"train.run_name={architecture}",
    ).train()

print(eto.segment.Project("project.yaml").compare())

The session lines#

sessions: is read the same way by both pipelines (SessionSpec, shared with pixel event spotting): a session is a source plus whatever the stage you run needs from it.

  • source — the session file. Always.

  • labels_path — its curated labels TSV. Unset, it is the session folder’s labels.tsv beside source (the GUI’s own convention; the log says what was assumed). Training and scoring read it. A session you only predict into needs none — a labels_path naming a file that does not exist simply means the session has no labels, and it contributes nothing to a training set.

  • video_dir — the folder searched for a trial’s video when the alignment does not already resolve it. Optional here: this pipeline reads feature columns, and only the video features (Video features) ever open a video.

  • name — the session id in every output (fold names, prediction sources, log lines). Optional: unset, it is the file’s stem, and sessions whose stems collide — Trial_data3.nc in every behav/ folder — are named by the nearest folder that tells them apart, ses-000_date-20250309_01_Trial_data3, listed in the log. Quote a name that is all digits, or YAML reads it as a number.

So a session to train on and one to predict into sit side by side:

sessions:
  - source: ../sub-01/ses-000_date-20250503_02/behav
    labels_path: ../sub-01/ses-000_date-20250503_02/behav/Trial_data_labels.tsv
  - source: ../sub-01/ses-000_date-20250506_02/behav     # no labels: predict into it

inference() covers every listed session unless sessions= narrows it, so that is the stage an unlabelled session is listed for. cross_validate() is for labelled sessions only: each fold scores the session it held out, which takes labels to score against.

What a sample is#

A sample is one (trial, individual) pair: an input x (F, T) and a target y (T,).

Input. Each entry of features.columns is an xarray variable with a time dim. The sample’s individual is selected (.sel(individual=…)), the listed dim values are selected, and whatever dims remain are flattened into columns. The columns of every feature are stacked into F:

position_ego (time, space, keypoint, individual)
  .sel(individual=ind, space=[x, y, z], keypoint=[beakTip, stickTip])
  → (T, 6)
speed        → (T, 1)
                                     → x: (F=7, T)

A pair feature (other: "*") keeps the remaining individuals as columns in dataset order (other1, other2, …), so every session must have the same number of individuals. Column names are written to columns.yaml.

A pynapple or NWB session works the same way: a Tsd is one column, and a TsdFrame (or TsdTensor) selects the listed columns over time, giving (T, n_columns).

Target. Per frame, the class index of that individual’s labels as actor; 0 is background. Only manual and curated labels count, never automated ones. Point events are skipped.

A trial with two individuals therefore gives two samples. With features.labels.branches listing several branches, y becomes multi-label (C, T): one binary channel per class. Labels within one branch never overlap; labels in different branches can (see Config reference).

The materialised dataset#

{root}/data/{features.name}/ uses the layout of the action-segmentation literature for interopability with newer architectures.

features/{key}.npy       (F, T) float32, session-level preprocessed
groundTruth/{key}.txt    one class name per frame
mapping.txt              "{index} {name}", contiguous, 0 = background
index.tsv                key → session, source, trial, individual, n_frames, fs, n_labelled
columns.yaml             the input layout: names, normalise flags, vector groups
classes.yaml             class index ↔ label id

key is {session_id}_trial{trial}_{individual}; session_id is the source’s stem plus a path hash, so two Trial_data.nc never collide.

The dataset does not record which samples are for training, validation or testing, and it is not z-scored. Both depend on the split, and the split is chosen per run. So one materialised dataset can serve many runs (a search, each cross-validation fold). Each run writes its own:

  • runs/{run}/splits/{train,val,test}.bundle: the sample keys in each role.

  • runs/{run}/stats.npz: the per-column mean and std, computed on that run’s training samples only and applied when the run loads its data.

Architectures#

Eleven networks are available; eto.segment.architectures() lists them. Switching between them is a one-line change, and project.compare() puts the runs side by side, so trying two or three is cheap.

Name

Shape

When to reach for it

c2f_tcn [Singhania et al., 2021]

U-Net over time

The default. Fast, and sees long-range context cheaply. Needs trials of at least 384 frames.

c2f_transformer [Kozlova et al., 2025]

c2f_tcn + attention

Same size limit; worth a run when c2f_tcn misses long-range structure.

mstcn (MS-TCN3) [Kozlova et al., 2025]

Dilated TCN, refined in stages

Works at any trial length. The usual baseline.

asformer [Yi et al., 2021]

Sliding-window attention + decoders

Strongest context modelling, several times slower per epoch.

edtcn [Lea et al., 2017]

Encoder–decoder, wide kernels

rnn

Bidirectional GRU or LSTM over the trial

Ours: a recurrent baseline, any trial length. Defaults in ethograph/segment/models/config/rnn.yaml.

mlp [Kozlova et al., 2025]

Per-frame, no temporal context

A floor to compare against: how much is temporal context buying you?

motionbert [Zhu et al., 2023]

Attention across joints, then across time

For pose columns that factor into joints: set model.params.num_joints (it has no default and must divide the column count). Reads a fixed 128-frame window at a time.

mp_transformer [Kozlova et al., 2025]

Max-pool → transformer encoder → upsample

DLC2Action’s MP-Transformer: attention over a pooled timeline. Reads a fixed 128-frame window at a time (model.params.len_segment); num_pool: 0 is a plain transformer encoder.

specscalpel [Ji et al., 2026]

Skeleton graph + frequency-selective filtering

Pose-native. Reads a skeleton of keypoints and sharpens the boundaries between adjacent behaviours in the frequency domain. Needs model.params.keypoints and (optionally) skeleton.

lady [Ji et al., 2026]

specscalpel + a learned Lagrangian-dynamics stream

Pose-native. Adds a physics-informed stream (torque, power, energy) over the skeleton’s generalised coordinates. Needs keypoints, a skeleton with edges, and the root-frame landmarks root/spine (+ left/right in 3D); reads raw positions only.

eto.segment.tunable_params(name) lists each one’s hyperparameters; set only the ones you want to change under model.params. For the vendored models they come, with a comment on each and its default, from ethograph/segment/dlc2action/config/model/{file}.yaml (mstcn reads ms_tcn3.yaml); specscalpel and lady read their skeleton-graph defaults. The loss is configured the same way under train.loss, from config/losses.yaml. See Config reference.

Note

The networks and the loss come from DLC2Action (AGPLv3, compatible with this project’s GPLv3 — see ethograph/segment/dlc2action/NOTICE.md). To plug in your own, register a builder with @register_architecture("name"), or ship one from another package through the ethograph.segment.architectures entry-point group. It takes (x (B,F,T), mask (B,1,T)) and returns logits (S,B,C,T), finest stage last — or a ModelOutput carrying those logits.

Stage 1: find the settings#

project.search() tries search.n_trials settings. Each one is a full training run, scored by train.select_on on the validation trials; the test trials are never read, so they still give an honest number at the end.

result = project.search()  # or search(n_trials=50)
print(result.best_params, result.best_score)

The winner is written to searches/{name}/best.yaml (your config with the winning values pinned), which is what stage 2 reads. Calling search() again adds trials to the same study rather than starting over.

The one choice you make is what goes under search.params, the ranges to search over. A parameter is named by its dotted config path, as in an override.

We search a subset of DLC2Action’s default parameter search (dlc2action.options.model_hyperparameters), with its ranges: the weight of the consistency term (alpha), focal or plain cross-entropy (focal), unweighted or inverse-frequency class weights (weights) — all three described under train.loss — and the network’s width, num_f_maps, under model.params:

search:
  n_trials: 30
  params:
    train.loss.alpha: {type: float, low: 1.0e-5, high: 1.0e-2, log: true}
    train.loss.focal: {type: categorical, choices: [true, false]}
    train.loss.weights: {type: categorical, choices: [null, dataset_inverse_weights]}
    model.params.num_f_maps: {type: int, low: 32, high: 128, log: true}  # c2f_tcn

The last line depends on the architecture. Swap in the row for yours:

Architecture

model.params ranges

c2f_tcn

num_f_maps: int 32–128, log

c2f_transformer

num_f_maps: [32, 64, 128]; heads: [1, 2, 4, 8]

mstcn

num_f_maps: int 32–128, log; num_layers_PG: 5–20; num_layers_R: 5–10; shared_weights: [true, false]

asformer

num_f_maps: [32, 64, 128]; num_decoders: 1–4; num_layers: 5–10; channel_masking_rate: 0.2–0.4

mp_transformer

N: 5–12; heads: [1, 2, 4, 8]; num_pool: 0–4

mlp

dropout_rates: 0.3–0.6

edtcn

loss settings only

DLC2Action also searches len_segment and temporal_subsampling_size. They are left out because a sample here is a whole trial rather than a fixed-length window.

Pick the parameters and ranges yourself. There are three kinds of range, matching Optuna’s three suggest calls:

search:
  n_trials: 30
  params:
    train.learning_rate:     {type: float, low: 1.0e-5, high: 1.0e-2, log: true}
    model.params.num_f_maps: {type: int, low: 32, high: 256, step: 32}
    train.augment.mirror:    {type: categorical, choices: [true, false]}

eto.segment.tunable_params(name) lists what each architecture accepts.

Architectures share almost no parameter names, so each one gets its own search, in a loop. Give each its own train.run_name, because that also names its study. Then cross-validate only the winner.

import ethograph as eto

SHARED = {  # means the same thing to every architecture
    "train.learning_rate": {"type": "float", "low": 1.0e-5, "high": 1.0e-2, "log": True},
    "train.loss.alpha": {"type": "float", "low": 0.0, "high": 0.5},
}
SPACES = {
    "asformer": {"model.params.num_decoders": {"type": "int", "low": 1, "high": 3}},
    "mstcn": {"model.params.num_R": {"type": "int", "low": 1, "high": 3}},
}

eto.segment.Project("project.yaml").materialise()  # once, shared by every search

results = []
for architecture, space in SPACES.items():
    overrides = eto.segment.as_overrides(
        {
            "model.architecture": architecture,
            "train.run_name": architecture,  # → searches/search_{architecture}/
            "search.params": {**SHARED, **space},
        }
    )
    result = eto.segment.Project("project.yaml", *overrides).search()
    results.append((result.best_score, result.config_path))

best_score, best_config = max(results)
eto.segment.Project(best_config).cross_validate()  # stage 2 on the winner only

eto.segment.as_overrides turns a dict into the key=value strings Project takes, so nested dicts and floats like 1.0e-5 survive intact.

Stage 2: cross-validate, and look at the mistakes#

With the settings settled, hold out a whole session per fold:

best = eto.segment.Project(result.config_path)
folds = best.cross_validate()  # one fold per session

Fold i trains on every session but the i-th and then predicts that one, so each session ends up with a prediction set from a model that never saw a frame of it. folds is one row per fold — its run, its metrics on the held-out session, and the path of the prediction set:

session

run

postprocessed.f1@50

predictions

ses-01

fold-ses-01_…

0.71

…/ses-01/behav/labels/predictions_fold-ses-01_…_{timestamp}/Trial_data_predictions.tsv

Folds are independent, so you can run some of them:

best.cross_validate(folds=["ses-01", "ses-02"])  # two folds, not all four

which is how you compare two parameter sets at a fraction of the cost. When you actually want to inspect a session in the GUI, run its own fold — a model that trained on the session it is predicting tells you nothing.

One session: fold by trial#

A session cannot be held out when it is the only one — which is every neural decoding project (features.neural), since units exist in one recording only. Fold by trial instead:

folds = project.cross_validate(n_folds=5)

Every trial is dealt into exactly one of the five folds (seeded by train.split.seed, so the folds are the same for every transform you compare); fold k trains on the other four and predicts its own, through train.split.holdout_trials; and the five prediction sets are merged into one per session, under labels/predictions_cv_{run_name}_{timestamp}/, so the whole session opens in the GUI with every trial predicted once by a model that never saw it — cross_validation/{name}/predictions.tsv lists it. folds.tsv still has one row per fold, with the trials it held out and its metrics on them; each row’s prediction_source says which fold wrote it.

Comparing binnings is then a loop over configs that share this one (base:), one materialised dataset each:

from ethograph.segment import as_overrides

for name, steps in {
    "rate_5ms_boxcar25ms": ["x.count(0.005) / 0.005", "sliding_window(x, window_size=0.025)"],
    "sqrt_count_10ms": ["x.count(0.01)", "np.sqrt(x)"],
}.items():
    eto.segment.Project(
        "decoding.yaml",
        f"features.name={name}",
        f"train.run_name={name}",
        *as_overrides({"features.neural.transform": steps}),
    ).cross_validate(n_folds=5)

giving cross_validation/cv_{name}/folds.tsv per transform — the same folds, so the numbers are paired.

Reviewing predictions in the GUI#

A prediction set is a labels TSV in the GUI’s own format, every row labeling_method = automated with the model’s confidence. Load it with File ▸ Import labels… and it enters the curation workflow: automated labels draw dotted, the grid views rank them by confidence, and every label you confirm becomes curated.

This is what makes stage 2 worth its cost: the fold’s predictions and the labels you drew are the same kind of object on the same axis, so “60% F1” becomes which class, which trials and how far off the boundaries are.