FERAL#

FERAL [Skovorodnikov et al., 2025] fine-tunes a video foundation model (V-JEPA 2) to predict a class per frame from the pixels alone: no pose, no features. There are two ways to use it from ethograph, and both start the same way: the labels you curate in the GUI become FERAL’s labels.json, without the chunking, class-name and split bookkeeping being done by hand.

FERAL’s labels — the simple one. Label in ethograph → export → train and predict in FERAL → import its predicted segments → curate them in the GUI. No segmentation model, no features; the released FERAL is enough.
FERAL as a video feature — label in ethograph → export → train FERAL and save its per-frame embeddings → they attach to every trial as a variable feral, optionally next to pose, changepoint or sensor columns → train a segmentation model on them → curate its predictions in the GUI.

Why the second, for two reasons:

Other features join the video. Exported as an embedding, FERAL is one input among several: pose, IMU or any other session variable sits next to it in the segmentation model.
Temporal context without losing temporal precision. A FERAL chunk is always 64 frames: either 64 consecutive ones, or with chunk_step > 1 every n-th frame over a longer span, which gives up temporal precision. On high-frame-rate video of behaviours that need a long context, one solution is to export at a stride of 1: FERAL learns short-context embeddings on every frame, and the segmentation model bridges the longer context.

Start with the first; move to the second when either applies.

The two environments#

FERAL runs in its own environment: its dependency pins clash with the GUI’s, so it is never installed alongside ethograph. What crosses is files. The project writes FERAL’s inputs into {root}/feral/; you train and infer there with FERAL’s own commands; what FERAL writes is read back from the same folder.

FERAL’s, once, following FERAL’s own README (a CUDA GPU of compute capability 8.0 or newer, PyTorch 2.5 or later):

conda create -n feral python=3.11 && conda activate feral
pip install torch torchvision          # a CUDA build; see FERAL's README for the version table
pip install feral

ethograph’s is where everything else happens: the config, the export, the import, the GUI — and, for the second workflow, the segmentation model.

Workflow 1: FERAL’s labels#

1. A config, naming the sessions, the labels and the camera:

sessions:
  - source: D:\data\ses-01\behav
    video_dir: D:\data\ses-01\videos          # if the alignment names the files without a folder
  - source: D:\data\ses-02\behav
    video_dir: D:\data\ses-02\videos
individual: bird1                                # a FERAL video is one trial of one animal
features:
  columns: {feral: {}}                           # never materialised here; a config names at least one column
  labels: {mapping: ../mapping.txt, branch: 0}
video_features:
  extractor: feral
  camera: cam-1                                  # which camera's file, when the alignment holds several
  context_s: null                                # seconds one FERAL chunk spans; null = every frame
  preset: lite                                   # FERAL's recipe: lite, max, rare; null = its default
train:
  run_name: crows

context_s and preset set FERAL’s own parameters (chunk_step and --mode); FERAL’s config documentation explains them and every other one, and Tuning FERAL says where to change them.

2. Export, in ethograph. With labels curated in the GUI:

import ethograph as eto

project = eto.segment.Project("config/feral.yaml")
project.export_feral()

writes {root}/feral/:

feral/
    labels.json        # FERAL's label file: a class per frame of every video, the splits
    config.yaml        # a complete FERAL config: feral train-config reads it
    videos.tsv         # which session, trial and individual each video is
    export.yaml        # FERAL class → label id, the rate, the stride, the held-out sessions
    export.log

and logs the commands for the next step. Each trial’s curated labels are rasterised on its own video’s frames through the alignment (trial = video

offset), so the label list is exactly as long as the video. The train / val / test roles are drawn by whole trial from train.split; inference is every video. A class no training video holds is background for FERAL (its inverse-frequency class weight would otherwise swamp the loss) and export.yaml lists it.

3. Train and predict, in FERAL’s environment. The log prints these with your paths filled in:

conda activate feral
feral train-config D:\project\feral\config.yaml
feral infer D:\project\feral\checkpoints\crows_best_checkpoint.pt D:\data\ses-01\videos --output D:\project\feral\predictions\ses-01__videos.json
feral infer D:\project\feral\checkpoints\crows_best_checkpoint.pt D:\data\ses-02\videos --output D:\project\feral\predictions\ses-02__videos.json

feral infer takes one flat folder of videos, so it runs once per video folder, each writing its own JSON into feral/predictions/. Copy the --output paths from the log: the import looks for exactly those names. FERAL’s own scores on its val and test splits are in feral/answers/.

4. Import, back in ethograph:

project.import_feral_predictions()

Every video’s per-frame probabilities are placed on its trial’s clock through the alignment (frames outside the trial are dropped), the most probable class per frame becomes intervals with the project’s own label ids, and infer.postprocess (minimum duration, stitching, snapping to changepoints — all off unless you set them) runs over them as it does for any model. Each session gets a prediction set labels/predictions_feral_{run_name}_{timestamp}/, in the GUI’s labels format with labeling_method=automated, FERAL’s probabilities beside it for the confidence overlay.

5. Curate in the GUI. Open the session and use File → Import predictions → Import as Labels… on that folder; the predictions draw dotted until you accept or edit them (Importing labels). Curated labels are training targets for the next export.

Workflow 2: FERAL as a video feature#

Here FERAL is a video feature like S3D or DINOv2 in Video features: its per-frame embedding becomes the variable feral on every trial, and a segmentation model reads it.

Note

Exporting embeddings needs FERAL’s --save_embeddings option, which is a pull request against FERAL at the time of writing. Until it is merged, install FERAL from that branch instead of the release: pip install git+https://github.com/Akseli-Ilmanen/feral@export-embeddings.

1. The config is the same one, with the columns the segmentation model reads and the sessions to hold out:

sessions:
  - source: D:\data\ses-01\behav
    video_dir: D:\data\ses-01\videos
  - source: D:\data\ses-02\behav
    video_dir: D:\data\ses-02\videos
  - source: D:\data\ses-03\behav
    video_dir: D:\data\ses-03\videos

individual: bird1

video_features:
  extractor: feral
  camera: cam-1
  context_s: 0.64
  preset: lite

features:
  columns:
    feral: {feral_dims: 0..767}                  # the embedding, once FERAL has written it
    speed_changepoints_prox: {}                  # ... next to whatever else the session holds
  labels: {mapping: ../mapping.txt, branch: 0}

train:
  run_name: feral_c2f
  split:
    holdout_sessions: [D:\data\ses-03\behav]   # the true test set — see Leakage

2. Export as above; project.video_features() is the same export for extractor: feral.

3. Train and infer in FERAL’s environment, adding --save_embeddings to every feral infer (the log names the folder):

conda activate feral
feral train-config D:\project\feral\config.yaml
feral infer D:\project\feral\checkpoints\feral_c2f_best_checkpoint.pt D:\data\ses-01\videos --output D:\project\feral\predictions\ses-01__videos.json --save_embeddings D:\project\feral\embeddings
feral infer D:\project\feral\checkpoints\feral_c2f_best_checkpoint.pt D:\data\ses-02\videos --output D:\project\feral\predictions\ses-02__videos.json --save_embeddings D:\project\feral\embeddings
feral infer D:\project\feral\checkpoints\feral_c2f_best_checkpoint.pt D:\data\ses-03\videos --output D:\project\feral\predictions\ses-03__videos.json --save_embeddings D:\project\feral\embeddings

Every call adds {video stem}.npy files to the same feral/embeddings/. The --output JSONs still import as in workflow 1: FERAL alone, the video-only baseline to compare the segmentation model against.

4. Back in ethograph, nothing to import: a session opened under this config finds its videos’ files in feral/embeddings/ and carries the variable feral (a (time, feral_dims) array, memory-mapped and sampled onto the trial clock the way every video feature is). Then the pipeline is the usual one:

project.materialise()  # feral + changepoint columns → the materialised dataset
result = project.train()  # holdout_sessions is the test set
project.compare()
project.inference()  # a prediction set beside every session

5. Curate in the GUI, as in workflow 1. Drop feral/embeddings/ onto the GUI window to see the embedding as a heatmap under the video, exactly as Video features describes for any folder of .npy files.

Leakage: which sessions FERAL may see#

A FERAL fine-tuned on a session’s labels embeds that session better than one it never saw, so any downstream score on a session FERAL trained on is inflated. train.split.holdout_sessions is the guard: the sessions it names are FERAL’s test and inference only, never train or val, and downstream they are the test set. Name them before exporting, and train()’s test score is one you can report, with FERAL’s own score in feral/answers/ as the video-only baseline on the same sessions. cross_validate() over FERAL features warns about this: it still compares feature sets on the same inflated footing, but it is not a held-out score (that would take one FERAL training and export per fold, which the project does not do for you).

Tuning FERAL#

FERAL’s parameters are FERAL’s, and its config documentation is where they are explained. export_feral() writes feral/config.yaml as a complete FERAL config: its packaged default, with only the paths, the run name and the two settings below filled in. Edit that file as you see fit before feral train-config; FERAL reads it verbatim, and the import does not depend on anything in it. Exporting again rewrites it, so keep a copy of your edits.

The ethograph config sets two of FERAL’s parameters, both refused with any other extractor:

video_features.preset is FERAL’s --mode (lite, max, rare; null = its default), laid over the default as feral train --mode does.
video_features.context_s is the seconds one chunk spans. The export reads the videos’ rate, resolves FERAL’s chunk_step from it and scales the chunk shifts by the same stride; null is every frame (chunk_step: 1), whatever the rate.