Video features#

A pretrained network turns each frame of a video into a vector: what the animal looks like it is doing, which pose keypoints alone do not capture. The network is an extractor, chosen by name; every extractor writes the same kind of file, and downstream nothing cares which one made it.

`extractor`	Kind	What one frame’s feature is	Docs
`s3d` (default)	clip-wise	the `stack_s` window of frames centred on it, embedded by S3D [Xie et al., 2018] (Kinetics-400 [Kay et al., 2017]) — motion is in the feature	video_features › S3D
`timm`	frame-wise	the frame on its own, embedded by an image backbone — DINOv2 [Oquab et al., 2024] ViT-B/14 (`vit_base_patch14_reg4_dinov2.lvd142m`) unless `model_name` says otherwise; needs `pip install 'ethograph[timm]'`	video_features › timm

Both follow the recipes of v-iashin/video_features [Iashin, 2020], which documents each network and links on to its weights; our config exposes only what changes the features, so read there for the rest.

A frame-wise feature carries no motion; the temporal model downstream reads it off the sequence. A clip-wise feature carries the window’s motion but needs the window to be long enough for the network. Which is better for a given behaviour is an empirical question — the point of one registry is that the comparison is a config line and a compare_runs.

They are expensive — a forward pass per frame — so they are computed once into a sidecar file per video and merged into your sessions afterwards.

Important

Every temporal setting is in seconds and resolved against each video’s own rate. analysis_fps says how many frames per second the network sees (frames are skipped, never interpolated up); it is the one cost lever, and at 200 fps you will want it. S3D’s stack_s needs at least 13 frames at that rate, so its 0.5 s default works down to 26 fps but fails below it — the error names the shortest window that works.

Starting from a folder of videos#

No config, no session, no alignment — just videos in, sidecars out:

import ethograph as eto

eto.segment.extract_videos(["/data/videos"], "/data/features", stack_s=0.5)                       # S3D (default)
eto.segment.extract_videos(["/data/videos"], "/data/features", extractor="timm", analysis_fps=25)  # timm / DINOv2

The first argument takes any mix of files, folders (searched recursively) and globs. Each video becomes /data/features/{video stem}_{extractor}.nc, holding a (time_video, {extractor}_dims) array on the video’s own clock (frame 0 at t=0), with the resolved plan and the model in its attrs. The written paths are returned. Videos that already have a sidecar are skipped unless you pass overwrite=True, so re-running after adding footage is cheap.

Parameter	Meaning
`extractor`	`s3d` (default) or `timm`.
`model_name`	`timm` only: any timm model with pretrained weights [Wightman, 2019] — valid names are the timm model list (or `timm.list_models(pretrained=True)`). `None` = DINOv2 ViT-B/14.
`stack_s`	`s3d` only: window length in seconds. Must be ≥ 13 frames at the effective rate.
`analysis_fps`	Rate the network sees; `None` = every frame.
`crop`	A pixel box cut from every frame before the network sees it (`{x0, y0, x1, y1}`, the crop tool’s numbers).
`include`	Regular expressions; keep only videos whose path matches one.
`overwrite`	Re-extract videos that already have a sidecar (default `False`).

A setting that belongs to the other extractor is refused by name — stack_s means nothing to a frame-wise model, model_name nothing to S3D — rather than silently ignored. Those are the only settings that change the features. Batch size, decode chunk, fp16 and the device live on the extractor’s own config (S3DConfig, TimmConfig).

Cropping to the animal#

Resolution on the animal matters more than resolution on the arena. crop cuts one rectangle from every decoded frame before the resize, in the same numbers the GUI’s crop tool reports (Tools ▸ Video: Pick a crop for a config…), so they copy straight across:

video_features:
  crop: {x0: 240, y0: 80, x1: 880, y1: 720}

The network takes a square, and you choose which one. Every extractor has a fixed square input — 224×224 for S3D, the model’s own side for a timm backbone (518 for DINOv2) — and gets there by scaling the box’s shorter side to it and taking the centre square, the Kinetics / ImageNet evaluation transform. So:

No crop: the centre square of the full frame, scaled. The default, and fine when the animal sits mid-frame.
A square crop: exactly what you drew, only scaled. The crop tool’s Square option (ticked by default) grows the dragged rectangle to a square about its centre, so the numbers it prints are the numbers the network sees. Draw it 224×224 for S3D and there is no resampling at all.
A non-square crop: the long side is cut off — a 203×164 box loses 19 % of its width. The extraction logs a warning saying how much; it does not refuse.

One box per video, applied to every trial’s video alike. A box that follows an individual — the per-individual feature a multi-animal recording needs — is the planned extension; the sidecar already records the crop in its attrs so that step needs no new format.

Taking one camera out of a folder#

Two cameras pointed at the same arena give nearly identical features, so extracting both is an hour of GPU time for nothing. include narrows what is found:

eto.segment.extract_videos(["/data/videos"], "/data/features", include=["cam-1"])

Each pattern is a regular expression matched against the whole path with re.search, so a plain substring works, and it finds the camera wherever it sits in the layout — cam-1/trial003.mp4 and trial003_cam-1.mp4 alike. Several patterns are a union (["cam-1", "cam-3"]).

Note

A filter that matches nothing raises rather than extracting zero videos — silently doing nothing looks too much like success. The message says how many videos were found and shows a few, so you can see what to match against.

Working from a config instead? Use video_features.camera — the alignment already knows which stream is which, so there is nothing to pattern-match.

Starting from a config#

If your sessions already have an alignment naming each trial’s video, name each session’s video_dir and let the alignment resolve the file within it:

sessions:
  - source: ../sub-01/ses-01/behav/Trial_data.nc
    labels_path: ../sub-01/ses-01/behav/Trial_data_labels.tsv
    video_dir: /data/videos

video_features:
  extractor: s3d                  # or timm
  analysis_fps: 25
  camera: cam-1                   # regex pattern so video features only for camera 1

project = eto.segment.Project("project.yaml")

project.video_features()              # extract only
project.video_features(merge=True)    # extract, then merge into the sessions

Sidecars go to {root}/video_features/; the paths written are returned, and overwrite=True re-extracts. Any setting can also be overridden without editing the file: eto.segment.Project("project.yaml", "video_features.extractor=timm", "video_features.analysis_fps=25").

Merging into a session#

Extraction alone does not make the embedding a feature — the sidecar is on the video’s clock, and the pipeline reads features from the session. Merging samples the sidecar onto each trial’s own time axis (nearest neighbour, applying that trial’s video offset) and writes a session copy carrying a variable named after the extractor, s3d (time, s3d_dims) or timm (time, timm_dims):

project.video_features(merge=True)
# → /data/sub-01/ses-01/behav/Trial_data_s3d.nc

The merge never overwrites your session file: it writes a sibling {stem}_{extractor}.nc and logs the path — point your config’s sessions: at it (or call merge_video_features() yourself with in_place=True if you would rather overwrite). Merging is xarray-only; for pynapple and NWB sessions the sidecar exists but you carry it in yourself. Sidecars written before the registry existed (time_s3d) still merge: the time dim is found by name.

Then name it like any other feature:

features:
  columns:
    s3d: {s3d_dims: [0, 1, 2, 3]}      # or a shortlist you selected
    speed: {keypoint: [beakTip]}

Choosing which dimensions to keep#

1024 (S3D) or 768 (DINOv2 ViT-B) columns is a lot next to a handful of kinematic ones, and they dominate the input — in practice a small, well-chosen subset does better than all of them. Two tools, both leaning on kind="video_feature", which every extractor stamps for you.

Is the whole group pulling its weight? That is a question about the trained model, so it takes two runs — the same materialised dataset, one extra fit:

eto.segment.Project("project.yaml", "train.run_name=full").train()
eto.segment.Project(
    "project.yaml",
    "train.run_name=no_video",
    "train.drop_kinds=[video_feature]",
).train()

print(eto.segment.Project("project.yaml").compare())

Important

compare() reads each run’s test_metrics.yaml, which is written only when the split leaves something in test. With train.split.test_fraction: 0 the table comes back empty — and an ablation judged on a random trial split is flattered anyway, so the honest version of this benchmark is cross_validate() with and without the video columns.

Which extractor? The same two-run shape: merge both sidecars into the session, list one or the other under features.columns, compare.

Which individual features are stereotypic? Rank them by Cohen’s d: for each feature and each behaviour class, how far apart the feature’s distribution is during that behaviour versus outside it, in pooled standard deviations. A feature with a large d for some class is one the model can act on; a feature with a small d everywhere is noise the model has to learn to ignore.

ranking, names = project.rank_video_features()
# the 20 most discriminating dims
print([names[i] for i in ranking.top(20)] )

This reads the materialised dataset, so it ranks exactly the columns a model would see and costs no re-extraction. It picks them out by kind="video_feature", and raises if no column declares it — describe your features when you build the session (describe()), then materialise again. min_frames drops a class that barely occurs in a trial.

The library call underneath takes the trials directly, for a bank that is not in a project yet:

from ethograph.video_features import rank_features

ranking = rank_features(trials)         # [(values (T, F), labels (T,)), ...]

labels are dense per-frame class ids with 0 = background; background is excluded from the comparison by default (background=0). Scores are averaged over trials.

In the GUI the same ranking is a heatmap — Model ▸ Video features: rank by Cohen’s d… — of classes against the top-k features, so you can see which behaviours each feature actually separates, and copy the top-k straight into your config as a features.columns line, keyed by the feature and its own dim:

features:
  columns:
    s3d: {s3d_dims: [492, 734, 671, 640, 585, ...]}

One video, no ethograph at all#

from ethograph.video_features import S3DConfig, build_extractor, extract_s3d

da = extract_s3d("clip.mp4", S3DConfig(stack_s=0.5))         # (time_video, s3d_dims)
da.to_netcdf("clip_s3d.nc")

da = build_extractor("timm").extract("clip.mp4")             # (time_video, timm_dims)

Adding an extractor#

An extractor is an entry in ethograph.video_features.EXTRACTORS — a name mapped to a class with name, plan(video_fps) and extract(video) — whose extract returns to_dataarray(...). Its package is pip-installed, never copied into the tree (notes/adr/0009-extractors-pip-installed-not-vendored.md); if the package cannot share the GUI environment, the extractor runs it by subprocess and reads the file back.