User Manual

ethograph is a GUI for visualizing and segmenting multimodal behavioural recordings.

Install

uv tool install --python 3.12 --compile-bytecode "ethograph[gui,audio]"

Launch with ethograph launch, or the desktop shortcut. See Installation for detailed instructions and troubleshooting.

Multimodal visualization

Ethograph allows you to visualize a number of different data modalities:

Data	Shown as	File formats / loaders
Video	multi-panel video	.mp4, .avi, .mov, .mkv (via pyav and pynaviz)
Pose / BoundingBox	Overlay on video	.h5, .csv, .slp (DLC, SLEAP, LightningPose, … via movement)
Audio	Waveform + spectrogram	.wav, .flac, .ogg, .mp3 (via audioio)
Electrophysiology	Multi-channel trace	.dat, .bin, .raw via phylib, and many more formats via Neo
Spike-sorted units	Raster / PSTH	Kilosort folder, .nwb file, pynapple.TsGroup
Features	Line plots / heatmaps	Kinematics, firing rates, latent variables, model outputs, etc.

Feature data can be loaded via three backends:

Backend	Object	File format
xarray	xarray.Dataset / TrialTree	.nc
Pynapple	Tsd / TsdFrame	.npz or folder
NWB	.nwb (loaded via pynapple)	.nwb
numpy	plain array	.npy

Unlike a plain numpy array, the top three formats carry explicit timestamps with each value. This allows ethograph to automatically align data of different sampling rates and modalities (video, audio & electrophysiology).

Note

.npy only supports (T, D) dimensional data. It can be loaded via drag & drop, you’ll be asked for the sampling rate of each timestamp (T), and it is converted internally into a timestamped object.

Loading your data

Launching opens the start page. It lists the supported file types along the bottom and offers three ways in, coloured below as in the screenshot:

Which entry point?

New here? Start with 1 Template datasets — example data of cool animal behaviours (🐦‍⬛🐦🎶🦗🐀) that loads in one click. See more here: examples gallery.

For your own data, the question is not how many files you have — drag & drop takes many — but how they are arranged across devices and trials.

Your data	Use
One trial, many devices. Files that start together and cover the same stretch of time — four cameras, three microphones, pose per camera.	2 Drag & drop, Several cameras/mics filming one trial
Many trials, one device. One file per trial per stream — trial001.mp4, trial002.mp4, …, optionally with matching pose and audio files.	2 Drag & drop, Several trials of one device
Many trials and many devices — e.g. 20 trials filmed by 2 cameras with 1 microphone (40 videos + 20 audio files); streams needing offsets — ephys on its own clock, trial windows carved out of a continuous recording; or an existing session folder (holding a .nc with trials, a .nwb, or pynapple files).	3 Custom set-up

Drag & drop

Pick a layout in the drop card’s dropdown, drop the files, click Load. Frame rates, sample rates and durations are read from the files themselves. You are only asked for what cannot be inferred: the source software of an ambiguous .h5/.csv pose file, the sampling rate of a .npy, or the camera order when several videos and pose files arrive together.

Several cameras/mics filming one trial (default): several videos become cam-1, cam-2, … (each its own view, one shared playhead); several audio files become mic-1, mic-2, … A multichannel .wav is split into channels automatically.

Several trials of one device: the files of each stream are natural-sorted and paired one per trial, so trial001.mp4 goes with trial001.h5. Every stream must hold the same number of files; the result is a multi-trial session you navigate trial by trial.

Tip

Long video, short moments of movement? Tick Compute video motion (video only) or Compute speed per keypoint (video + pose) on the drop card. Both give you a trace of how much is moving over time, from pixel changes or from the pose, so hours of a sleeping animal collapse to a handful of peaks you can jump to and label.

Project folder

The bar above the cards names your project folder: one folder per research project, holding everything that spans sessions. The most important file in it is mapping.txt, the label vocabulary (label id, name, key) every session of the project labels with. Beside it live the configs you may want to edit:

• skeleton/ and space/: pose skeletons and reference geometries for the plots.

• calibration/: camera calibrations for 3D.

• workflows/: curation workflows.

• segment/segment.yaml and spot/spot.yaml: the configs of the segmentation and event-spotting models.

A fresh install starts with a ready-made starter project, so you can look at data before choosing one of your own. The full layout is in Folder layout; the models that read the configs are under Models.

Drops are kept here too: while a project is set, every drag & drop lands in its sessions/ folder and shows up under Reopen a previous drop… on the drop card. Nothing is copied into the project; your media and data stay where they are.

Custom set-up

The loader takes a session folder, not a file. A session is a folder that holds the alignment (.ethograph/alignment.nwb, or a .nwb at its root) and at most one dataset (.nc, or pynapple .npz files). Browse to that folder in Session folder and click Load; the alignment, the dataset, labels.tsv and the panel layout are found from the folder.

The other fields add to what the folder already knows, and can stay empty:

• Alignment and Metadata name those files explicitly when they do not live in the folder (Template writes an example metadata TSV to fill in).

• Video / Pose / Audio folder point at media kept elsewhere; the alignment’s per-trial filenames are looked up there.

• Ephys file and Units add a raw recording and spike-sorted units (Kilosort folder or a pynapple file).

• Import labels loads the folder’s labels.tsv; Downsample thins the dataset for a faster first look.

If your data is not aligned yet, Data wizard builds the session folder for you: it pairs videos, audio and pose files per trial, reads their rates and durations, and writes the alignment and a session.nc. For anything the wizard does not cover, write the folder from Python instead.

See Bring your own data for the folder layout, the dataset schema and the alignment, and Ephys recordings for ephys formats and Kilosort.

Labelling

Ethograph is a labelling GUI for marking the onset and offset of behavioural events. Press a number key to select a label class, click on the timeseries plot to mark the onset, click again to mark the offset, then press V to play back the segment you just labelled.

The labels documentation has more on how to:

• label in the GUI

• define new label classes (mapping.txt)

• deal with labels overlapping in time (label branches)

• import labels from other formats (Pynapple, NWB, Audacity, Raven, …)

• migrate a BORIS project (observations become sessions)

• export labels

The interface

The layout is responsive and panel-based: every plot, video and trace lives in its own panel that you can drag, stack, tab, float and resize freely.

Add panels with the ➕ Add panel button (bottom-left of the window, or Shift+N): pick a data source, then drag it onto the plot area or press Enter to drop it in the default spot. Remove any panel with its ✕. You can open as many as you like, e.g. the same feature as both a line plot and a heatmap.

From the camera alone, ➕ Add panel adds a 3D space plot and a speed line plot.

Click a panel to edit it. The right sidebar is context-sensitive: clicking a panel shows the settings for that panel, and the panel gets a green edge.

Click the speed plot, the space plot, the video: the Data section follows.

Where things are

The right sidebar has three section buttons: Data, Labels and Trials. Click one to open it (the other closes). The crops below are those sections, taken from the window above.

Data, after clicking the speed plot.

Individual is always on top, whatever panel is clicked. It picks whose data the panels show and whom the next label belongs to.

Feature and dimensions are the clicked panel’s: which variable it draws and which keypoint, axis or individual.

Axes Controls are the clicked panel’s too: Y min / Y max, the percentile y-limit and the horizontal lines are stored per panel, so two speed plots can be scaled differently. Only Autoscale Y and Lock Axes are global.

Labels: the classes you can press, grouped by branch.

Mode and overlay: whether a label’s boundaries come from clicks on the plots (Graph based) or from the frame on screen (Frame by frame), and how existing labels are drawn.

Branch: select the branch first. Only the branch whose name is highlighted (editing) accepts new labels, Ctrl+D and Ctrl+E; click a branch name to make it the active one. A label of another branch can still be clicked and played back, but editing it is refused until its branch is active. The checkbox only shows or hides that branch’s overlay. See Label branches.

Classes and their keys: press a key to arm that class.

Important

The Individual dropdown lives in Data, not in Labels, and it does two things at once: it chooses whose data the panels show and whom the next label belongs to. Switch it to bird_2 and every panel following the sidebar shows bird_2’s traces, and a label you place is bird_2’s. The bottom bar echoes labelling: <individual> so you always know. Pinned panels (📌) override it, see below.

Changepoint detection and correction, which snap label boundaries to a feature’s troughs or turning points, are in the top bar under Changepoints (the feature itself is the one picked in Data). See Changepoints.

Which individual a panel shows

With more than one individual in the dataset, every panel is in one of two modes, shown in its title:

speed — bird_1 (sidebar)    follows the Individual dropdown
speed — bird_2 (pinned)     ignores it

• The Individual dropdown sets what every (sidebar) panel shows, and the default for new panels.

• 📌 next to the dropdown pins the last clicked panel — any panel — to one individual, or sets it back to Follow sidebar. The same entries end a panel’s ⠿ menu.

• A pinned panel shows and creates only that individual’s labels. If its data has an individual dimension, the pin selects that individual’s data too (a space or radial plot’s own Individual dropdown is its pin).

To compare two birds side by side: add two duplicate lineplots via ➕ Add panel. Then for one of them, click 📌 → bird_2. Switching the dropdown now moves only the original.

Trial metadata

Per-trial conditions (stimulus, reward, pellet count, …) live in a small table — a {stem}_metadata.tsv next to your data, or the NWB trials table — and show up as the trials table in the sidebar’s Trials section (the third section button), where each column becomes a filter. See Trial metadata.

Important

The trials table is the one place trials are included or excluded — and it applies to everything. Filter num_pellets to 1, 2 (not 0) and every operation sees only those trials: navigation, label/sequence jumps, changepoint correction, curation (Ctrl+C, inspect mode, frame-by-frame review, the label and video grids), model training and model inference. A label in a filtered-out trial is not visited, curated, trained on or predicted over until you widen the filter. No dialog has a trial filter of its own; each one tells you how many trials it will run over.

Two more things worth knowing: Ethograph keeps a curated column in that table itself (yes once every label of a trial is manual or curated, otherwise no — see Curating labels), and metadata edits saved automatically.

Video & audio playback

Two things are easy to miss when you first play back a recording:

• The red timeline marker tracks exact time, not the video frame. Video is discrete frames, but audio and other signals have far finer resolution, so clicking a plot places the marker (and any label) at the exact time you clicked — the video just seeks to the nearest frame as a visual reference.

• Only “Audio-synced” playback plays sound. The playback mode dropdown in the bottom bar also offers silent “Smooth” and “Real-time” modes; pick Audio-synced whenever you need to hear the audio locked to the video.

See Video & Audio playback for the full details, including playback speed/pitch, video proxies for smooth scrubbing, and which audio channel Play uses.

Feature dropdowns

Above, clicking the speed plot filled the Data section with Lineplot settings: a Features dropdown, then one dropdown per dimension (Keypoint, Space, …). This section explains where those dropdowns come from, so you know what to expect from your own data.

They are built from the dataset. We use pose data from a carrion crow performing a tool-use task [Moll et al., 2025]: one behavioural trial as an xarray.Dataset, with position, velocity, speed and acceleration for three keypoints tracked in 3D. Every variable becomes an entry of the Features dropdown, and every dimension of the chosen variable becomes a dropdown of its own.

import ethograph as eto

ds = eto.sample_data()
ds[["position", "velocity", "speed"]]

<xarray.Dataset> Size: 190kB
Dimensions:     (time: 1077, space: 3, keypoint: 3, individual: 1)
Coordinates:
  * time        (time) float64 9kB 0.0 0.005 0.01 0.015 ... 5.37 5.375 5.38
  * space       (space) <U1 12B 'x' 'y' 'z'
  * keypoint    (keypoint) <U8 96B 'beakTip' 'stickTip' 'pellet'
  * individual  (individual) <U5 20B 'Crow1'
Data variables:
    position    (time, space, keypoint, individual) float64 78kB ...
    velocity    (time, space, keypoint, individual) float64 78kB ...
    speed       (time, keypoint, individual) float64 26kB ...
Attributes:
    source_software:  DeepLabCut
    ds_type:          poses
    fps:              200.0
    time_unit:        seconds
    source_file:      c:/Users/aksel/Documents/Code/ethograph/data/Moll2025/2...
    trial:            41
    pellet_position:  right

xarray.Dataset

• Dimensions:
  • time: 1077
  • space: 3
  • keypoint: 3
  • individual: 1
• Coordinates: (4)
  • time
    (time)
    float64
    0.0 0.005 0.01 ... 5.37 5.375 5.38

    array([0.000e+00, 5.000e-03, 1.000e-02, ..., 5.370e+00, 5.375e+00, 5.380e+00],
          shape=(1077,))

  • space
    (space)
    <U1
    'x' 'y' 'z'

    array(['x', 'y', 'z'], dtype='<U1')

  • keypoint
    (keypoint)
    <U8
    'beakTip' 'stickTip' 'pellet'

    array(['beakTip', 'stickTip', 'pellet'], dtype='<U8')

  • individual
    (individual)
    <U5
    'Crow1'

    array(['Crow1'], dtype='<U5')

• Data variables: (3)
  • position
    (time, space, keypoint, individual)
    float64
    ...

    [9693 values with dtype=float64]

  • velocity
    (time, space, keypoint, individual)
    float64
    ...

    [9693 values with dtype=float64]

  • speed
    (time, keypoint, individual)
    float64
    ...

    [3231 values with dtype=float64]

• Attributes: (7)

  source_software :

  DeepLabCut

  ds_type :

  poses

  fps :

  200.0

  time_unit :

  seconds

  source_file :

  c:/Users/aksel/Documents/Code/ethograph/data/Moll2025/2024-12-18_041_Crow1_DLC_3D.csv

  trial :

  41

  pellet_position :

  right

Notice the dimensions:

Feature	Dimensions
position	(time, space, keypoint, individual)
velocity	(time, space, keypoint, individual)
speed	(time, keypoint, individual)

With xarray.DataArray, you can use sel() to pick a specific keypoint, spatial axis, and individual:

# Standard xarray .sel — works when all dimensions exist
print(f"Dims: {ds['position'].dims}, Shape: {ds['position'].shape}")
beak_x = ds["position"].sel(keypoint="beakTip", space="x", individual="Crow1")
print(f"Dims: {beak_x.dims}, Shape: {beak_x.shape}")

Dims: ('time', 'space', 'keypoint', 'individual'), Shape: (1077, 3, 3, 1)
Dims: ('time',), Shape: (1077,)

---

In the GUI, you can switch between features using the Feature dropdown and select (sel()) a unique combination of feature dimensions. In the video below, the user switches from the feature speed to velocity, with the selection dimensions individual=crow1, keypoint=beakTip and space=["x", "y", "z"] (All checkbox ticked). Thanks to sel_valid(), any invalid dimensions are ignored (e.g. it’s not problematic that speed does not have a space dimension).

Video from an older GUI version; the idea is unchanged.

At the end of the video and in the plot below, one can see that particularly velocity for keypoint=beakTip and space=z (green line) defines segmentation boundaries of the crows downward movement, inserting the stick tool into the dispenser.

Show code cell source

Hide code cell source

import matplotlib.patches as mpatches
import matplotlib.pyplot as plt
import numpy as np

from ethograph.labels.intervals import add_interval, empty_intervals
from ethograph.labels.plots import plot_label_segments

sel = {"keypoint": "beakTip", "individual": "Crow1"}

# 3 labelled intervals (real onsets from the Moll2025 dataset)
label_color = np.array([0, 204, 204]) / 255.0  # _LABEL_COLORS[10]
labels_df = empty_intervals()
labels_df = add_interval(labels_df, 3.87, 4.035, 10, "Crow1")
labels_df = add_interval(labels_df, 4.315, 4.435, 10, "Crow1")
labels_df = add_interval(labels_df, 4.588757756, 4.674700055, 10, "Crow1")

label_mappings = {10: {"name": "stickInDisp", "color": label_color, "order": 10, "branch": 0}}

fig, axes = plt.subplots(2, 1, figsize=(10, 4), sharex=True)

# velocity x, y, z
for i, (axis_name, color) in enumerate(zip(["x", "y", "z"], ["#1f77b4", "#ff2a0e", "#2ca02c"])):
    vel, _ = eto.sel_valid(ds["velocity"].sel(space=axis_name), sel)
    axes[0].plot(ds.time, vel, color=color, label=axis_name)
axes[0].set_ylabel("beakTip velocity \n (cm/s)", fontsize=11)
axes[0].legend(loc="upper right", fontsize=11, ncol=2)
plot_label_segments(axes[0], labels_df, label_mappings, individual="Crow1", is_main=True, alpha=0.3)

# speed
spd, _ = eto.sel_valid(ds["speed"], sel)
axes[1].plot(ds.time, spd, color="#000305")
axes[1].set_ylabel("beakTip speed \n (cm/s)", fontsize=11)
plot_label_segments(axes[1], labels_df, label_mappings, individual="Crow1", is_main=True, alpha=0.3)

axes[1].set_xlabel("time (s)")
insert_stick_patch = mpatches.Patch(
    facecolor=label_color,
    alpha=0.3,
    edgecolor="none",
    label="insert stick tool",
)

handles, labels = axes[0].get_legend_handles_labels()
axes[1].legend(
    [insert_stick_patch],
    ["insert stick tool"],
    loc="upper right",
    fontsize=11,
    ncol=4,
)

for ax in axes:
    ax.set_xlim(3.5, 5.0)


plt.tight_layout()
plt.show()

Changepoints

Besides looking at the z-velocity, one can also identify good candidates for segmentation boundaries by looking at where there are minima or turning points in the beakTip speed curve. We call these kinematic changepoints.

Show code cell source

Hide code cell source

import matplotlib.patches as mpatches
import matplotlib.pyplot as plt
import numpy as np

from ethograph.labels.intervals import add_interval, empty_intervals
from ethograph.labels.plots import plot_label_segments

sel = {"keypoint": "beakTip", "individual": "Crow1"}

# 3 labelled intervals (real onsets from the Moll2025 dataset)
label_color = np.array([0, 204, 204]) / 255.0  # _LABEL_COLORS[10]
labels_df = empty_intervals()
labels_df = add_interval(labels_df, 3.87, 4.035, 10, "Crow1")
labels_df = add_interval(labels_df, 4.315, 4.435, 10, "Crow1")
labels_df = add_interval(labels_df, 4.588757756, 4.674700055, 10, "Crow1")

label_mappings = {10: {"name": "stickInDisp", "color": label_color, "order": 10, "branch": 0}}

fig = plt.subplots(1, 1, figsize=(10, 4), sharex=True)

# speed
spd, _ = eto.sel_valid(ds["speed"], sel)
plt.plot(ds.time, spd, color="#000305")

troughs = ds["speed_troughs"].sel(keypoint="beakTip", individual="Crow1").values
trough_idx = np.where(troughs == 1)[0]
turning_points = ds["speed_turning_points"].sel(keypoint="beakTip", individual="Crow1").values
turning_points_idx = np.where(turning_points == 1)[0]
changepoints_idx = np.sort(np.concatenate([trough_idx, turning_points_idx]))
plt.scatter(ds.time[changepoints_idx], spd[changepoints_idx], s=20, c="r", label="changepoints")
plt.ylabel("beakTip speed \n (cm/s)", fontsize=11)
plot_label_segments(plt, labels_df, label_mappings, individual="Crow1", is_main=True, alpha=0.3)

plt.xlabel("time (s)")
insert_stick_patch = mpatches.Patch(
    facecolor=label_color,
    alpha=0.3,
    edgecolor="none",
    label="insert stick tool",
)

handles, labels = plt.gca().get_legend_handles_labels()
plt.legend(
    [insert_stick_patch, *handles],
    ["insert stick tool", *labels],
    loc="upper right",
    fontsize=11,
)

plt.xlim(3.5, 5.0)
plt.tight_layout()
plt.show()

Storing trials: Dataset vs TrialTree

Back at the start we split loading into one recording versus trials. That choice decides how you load; this section shows what it looks like on disk for the xarray backend. A session.nc can hold either shape, and the GUI accepts both.

Plain Dataset — a single xr.Dataset saved with xarray.Dataset.to_netcdf(). Loaded as one trial:

import xarray as xr

ds = xr.Dataset(...)
ds.to_netcdf("session.nc")
# -> GUI loads this as one trial

TrialTree — wraps one xr.Dataset per trial into a single .nc, and is how multi-trial data is stored on the xarray side (pynapple uses an IntervalSet; NWB uses its nwb.trials table):

import ethograph as eto

# Multiple trials: one Dataset per trial
dt = eto.from_datasets([ds_trial1, ds_trial2, ds_trial3])
dt.save("session.nc")

# Access individual trials
ds = dt.trial(1)      # by ID
ds = dt.itrial(0)     # by index

One session-long xr.Dataset plus trial start/stop times? Pass both to from_continuous(). The times can be a pd.DataFrame with trial, start_time and stop_time columns (in seconds, on the dataset’s time axis) or a pynapple.IntervalSet. Each trial becomes its own window, with time starting at 0:

trials = pd.DataFrame({"trial": [1, 2], "start_time": [0.0, 120.0], "stop_time": [100.0, 250.0]})
dt = eto.from_continuous(ds, trials)
dt.save("session.nc")

Storing trials is only half the job: the GUI also needs to know which video, audio and pose files belong to each trial, which lives in a separate alignment file. Bring your own data covers both ends — the dataset schema and the alignment — and TrialTree documents the full TrialTree API.

If a model will learn from your trials, see Trial windows for how long to make them.

Editing the dataset while the GUI is open

There are two ways to change the data the GUI shows without restarting it: compute something on the fly in the console panel, or rewrite the file on disk and reload it.

In the console panel. Add it with ➕ Add panel like any other panel. Click a feature panel and its plotted values are bound to a Python name (a plain numpy array, as drawn: the current feature, selections and trial); anything you assign becomes a new feature you can plot. Good for a quick look — speed_z = (speed - np.nanmean(speed)) / np.nanstd(speed) — but it lives for the current trial only and is never written to the dataset.

On disk, then reload. The GUI reads the dataset into memory and lets go of the file, so a notebook can sit beside it: change the data, save, and pull the new version in with File ▸ Reload data from disk. A panel showing a variable you rewrote redraws with the new values; a variable you added appears in the Features dropdown and in ➕ Add panel.

import ethograph as eto

dt = eto.open("session.nc")
dt.update_trial(1, lambda ds: ds.assign(speed_smooth=ds["speed"].rolling(time=5).mean()))
dt.save()
# -> File ▸ Reload data from disk

A pynapple session works the same way: overwrite its .npz files and reload.

Reload moves only the data. Media, alignment (so trial timing), labels, metadata and the panel layout are not re-read; a file whose trial ids differ from the loaded session is refused. For those, go back to the start page.

Tip

Smoothing and changepoints have dedicated tools. Both preview the result on the plot you clicked and hand you the code once you like what you see: Tools ▸ Data: Find smoothing parameters… and Changepoints ▸ Detect changepoints….

Template datasets

Ready-made datasets with video, features and labels already in place, downloaded from the start page. The notebooks that built them are in the examples gallery.

Tool-using crows [Moll et al., 2025] — Learned precision tool use in carrion crows. Two trials with video, 3D pose (DeepLabCut), kinematics, and video features.

Lockbox [Reiske et al., 2026] — Behaviour recognition for mice solving mechanical puzzle lockboxes. Six trials with multi-camera video and pose.

Zebra finches in BirdPark [Rüttimann et al., 2025] — Multimodal dataset of zebra finch groups with synchronized video, microphone arrays, and backpack-mounted accelerometers.

Cricket stridulation — Sound production in Pholidoptera littoralis with synchronized video, audio, and DeepLabCut pose.