S3D video features: extract, rank, inspect#

A pretrained video network (S3D, Kinetics-400) turns each frame’s surroundings into a 1024-d vector — what the animal looks like it is doing, which pose keypoints alone do not capture. This notebook goes from a session .nc plus its labels TSV to a session file carrying both the full 1024 dimensions and a ranked subset, ready to plot in the GUI and to name in a segmentation config.

  1. Extract S3D and merge it into the session — one call

  2. Rank the dimensions against your labels by Cohen’s d

  3. Plot the classes × top-k heatmap

  4. Write s3d (all 1024) and s3d_top (the subset) into one file

You need an alignment.nwb#

There is no filename guessing here. Which video belongs to which trial, and where in that video the trial starts, both come from .ethograph/alignment.nwb beside your .nc — the same file the GUI reads.

That second part is the whole reason the alignment is involved: a trial generally does not begin where its video file begins. S3D features are written on the video’s clock (frame 0 at t = 0), the session lives on the trial clock, and stream_offset_for_trial is the bridge between them. If you have no alignment yet, build one in the GUI: cover page ▸ Data wizard.

Important

CAMERA must name a device the alignment actually has (cam-1, cam-2, …), spelled exactly. It is not optional and has no default: the lookup builds the trials-table column name as video_{CAMERA}, so None asks for a column called plain video, finds nothing, and reports “no video found” for every trial. The cell below prints the alignment’s camera list and refuses to continue if CAMERA is not in it.

Cut your footage to trials if you can#

Not for correctness — the offset handles either layout — but for cost. S3D is a forward pass per frame and a sidecar covers the whole video file, so one long session recording means paying for every inter-trial second you will never label. Splitting is usually the biggest saving available:

# Frame-accurate: re-encodes, slower, exact cut
ffmpeg -ss 12.40 -to 47.85 -i session.mp4 -c:v libx264 -crf 18 -preset fast -an trial001.mp4

# Fast: stream copy, but cuts snap to the nearest keyframe (can be seconds off)
ffmpeg -ss 12.40 -to 47.85 -i session.mp4 -c copy -an trial001.mp4

Use the re-encoding form unless you know your GOP is short — a keyframe-snapped cut shifts a trial’s features in time and nothing downstream can detect it. Re-run the Data wizard afterwards so the alignment names the new clips; its offsets then become 0.0 and nothing in this notebook changes.

Settings#

from pathlib import Path

NC_PATH = r"C:\Users\aksel\Documents\AK_data\derivatives\sub-03_id-Freddy\ses-000_date-20250526_02\behav\Trial_data3.nc"
LABELS_PATH = r"C:\Users\aksel\Documents\AK_data\derivatives\sub-03_id-Freddy\ses-000_date-20250526_02\behav\Trial_data_labels.tsv"

VIDEO_DIR = r"C:\Users\aksel\Documents\VidData\20250526_02_Freddy"
CAMERA = "cam-1"         
INDIVIDUAL = None        # If None -> first in the TSV

STACK_S = 0.5            # S3D window in seconds; >= 13 frames at the effective rate
ANALYSIS_FPS = 50        # rate S3D sees; None = every frame. Halving ~halves the compute time.
TOP_K = 20               # how many dimensions to keep in `s3d_top`

ROOT = Path.cwd()        # sidecars land in ROOT/video_features/

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

import ethograph as eto
from ethograph.io.schema import VIDEO_FEATURE, describe
from ethograph.io.trialtree import TrialTree
from ethograph.labels.intervals import load_mapping
from ethograph.labels.ml import intervals_to_dense
from ethograph.labels.tsv_store import load_labels_tsv
from ethograph.segment.config import SegmentConfig, SessionSpec, VideoFeaturesConfig
from ethograph.utils.xr_utils import get_time_coord
from ethograph.video_features import rank_features

# A project needs no YAML file — a SegmentConfig built here does just as well.
config = SegmentConfig(
    sessions=[SessionSpec(source=Path(NC_PATH), labels_path=Path(LABELS_PATH), video_dir=Path(VIDEO_DIR))],
    root=ROOT,
    video_features=VideoFeaturesConfig(stack_s=STACK_S, analysis_fps=ANALYSIS_FPS, camera=CAMERA),
)
project = eto.segment.Project(config)

session = project.sessions()[0]
cameras = session.result.nwb_alignment.cameras
if CAMERA not in cameras:
    raise ValueError(
        f"CAMERA={CAMERA!r} is not in this alignment, which has {cameras}. "
        "Set CAMERA to one of those. Leaving it None does NOT pick a default - "
        "it looks for an un-suffixed 'video' column and finds nothing."
    )

alignment = session.result.nwb_alignment


first = session.trial_ids[0]
offsets = [alignment.stream_offset_for_trial(t, "video", CAMERA) for t in session.trial_ids]
print(f"{len(session.trial_ids)} trials | cameras {cameras} | using {CAMERA!r}")
print(f"trial {first} video: {session.media_path(first, 'video', CAMERA)}")

1. Extract and merge#

One call does all of it: resolve each trial’s video from the alignment, run S3D once per video file (ten trials cut from one recording extract once, not ten times), then sample each sidecar onto its trial’s own time axis applying that trial’s offset.

Sidecars go to ROOT/video_features/ and are skipped if they already exist — pass overwrite=True after changing STACK_S or ANALYSIS_FPS. The merge never touches your source file; it writes the sibling {stem}_s3d.nc.

Important

S3D needs at least 13 frames per window. STACK_S = 0.5 works down to 26 fps; below that the call raises and names the shortest window that does work.

sidecars = project.video_features(merge=True)

MERGED_NC = Path(NC_PATH).with_name(Path(NC_PATH).stem + "_s3d.nc")
print(f"{len(sidecars)} sidecar(s) newly extracted in {config.video_features_dir}")
print(f"merged session -> {MERGED_NC}  ({MERGED_NC.stat().st_size / 1e6:.0f} MB)")

Warning: Feature 'labels_confidence' listed in catalog but not found in dataset. Skipping validation for this feature.
Extracting S3D video features into c:\Users\aksel\Documents\Code\ethograph\examples\video_features\91830111 (overwrite=False)

2. Dense per-frame labels#

The ranking is supervised, so it needs one class id per frame. Two rules, both deliberate:

  • Only manual and curated rows count — ranking on another model’s automated output would select the dimensions that reproduce that model, not the ones that describe the behaviour.

  • Background (0) is the contrast, not a class — an unlabelled frame is what every class is measured against.

A trial with no curated labels, or one entirely background, carries no contrast and is left out; the cell reports how many.

dt = eto.open(str(MERGED_NC))

curated = load_labels_tsv(LABELS_PATH)
curated = curated[curated["labeling_method"].isin(["manual", "curated"])]
if curated.empty:
    raise ValueError(f"{LABELS_PATH} holds no manual/curated rows — nothing to rank against.")

individual = INDIVIDUAL or sorted(curated["individual"].dropna().unique())[0]
print(f"Ranking against individual {individual!r}")

collected: list[tuple[np.ndarray, np.ndarray]] = []
skipped_unlabelled = skipped_background = 0
for trial in dt.trials:
    rows = curated[curated["trial"] == trial]
    if rows.empty:
        skipped_unlabelled += 1
        continue
    s3d = dt.trial(trial)["s3d"]
    time = np.asarray(get_time_coord(s3d).values)
    fs = 1.0 / float(np.median(np.diff(time)))
    y = intervals_to_dense(rows, fs, [individual], len(time))[:, 0].astype(np.int64)
    if not np.any(y):
        skipped_background += 1
        continue
    collected.append((np.asarray(s3d.values), y))

print(
    f"{len(collected)} trial(s) usable; skipped {skipped_unlabelled} unlabelled "
    f"and {skipped_background} all-background."
)

3. Rank by Cohen’s d#

For each dimension f and class label c: how far apart is f’s distribution during label vs outside. How much is the activation function of this artificial unit (in S3D network) different during vs rest?

\[d_{f,c} = \frac{|\bar{x}_{f,c} - \bar{x}_{f,\lnot c}|}{\sqrt{(s^2_{f,c} + s^2_{f,\lnot c})/2}}\]

A dimension’s score is its best class, averaged over trials. So if a feature is meaningful for at least one label, it gets a high rank.

ranking = rank_features(collected)
top_indices = ranking.top(TOP_K)

label_names = [f"label {int(c)}" for c in ranking.class_ids]

print(f"Ranked {ranking.n_features} dimensions over {ranking.n_trials} trial(s) "
      f"and {len(ranking.class_ids)} class(es).\n")
print(pd.DataFrame({
    "dim": top_indices,
    "cohens_d": ranking.scores[top_indices].round(3),
    "best_class": [label_names[i] for i in ranking.per_class[top_indices].argmax(axis=1)],
}).to_string(index=False))

4. Classes × top-k heatmap#

The ranking collapses each dimension to its best class; the heatmap does not. It shows which behaviours each dimension actually separates — how you spot one that is mediocre on average but the only one catching a rare class.

fig, ax = plt.subplots(figsize=(14, 8))

heatmap_data = ranking.per_class[top_indices, :].T   # (classes, top_k)
im = ax.imshow(heatmap_data, aspect="auto", cmap="YlOrRd", interpolation="nearest")

ax.set_xlabel("S3D feature", fontsize=24)
ax.set_ylabel("Motifs", fontsize=24)

ax.set_yticks(range(len(label_names)))
ax.set_yticklabels(label_names, fontsize=14)

ax.set_xticks(range(len(top_indices)))
ax.set_xticklabels(top_indices, rotation=45, ha="right", fontsize=12)

cbar = plt.colorbar(im, ax=ax, fraction=0.046, pad=0.04)
cbar.set_label("Cohen's d", fontsize=24)
cbar.ax.tick_params(labelsize=12)

plt.tight_layout()
plt.show()

5. Write the subset alongside the full set#

s3d is already in the merged file. This adds s3d_top next to it, so you can switch between them in the GUI and in a config without re-running anything.

Important

s3d_top needs its own dim name — two variables cannot share a dim at two different lengths in one xarray Dataset. Its coord values are the original 1024-space ids (492, 734, …), not 0…k, so every column keeps its provenance.

The trials are pulled into memory and the file handle closed before writing, so this saves back over the same path rather than leaving you with a third file.

TOP_DIM = "s3d_top_dims"

def with_top(ds):
    subset = ds["s3d"].isel(s3d_dims=top_indices).rename({"s3d_dims": TOP_DIM})
    subset = subset.assign_coords({TOP_DIM: np.asarray(top_indices, dtype=int)})
    subset.attrs = {
        **ds["s3d"].attrs,
        "description": f"Top {len(top_indices)} S3D dims by Cohen's d (ids in the 1024-d space)",
        "selected_from": "s3d_dims",
        "selection": "cohens_d",
    }
    describe(subset, VIDEO_FEATURE, is_egocentric=False)
    return ds.assign({"s3d_top": subset})


trials_out = [with_top(dt.trial(t).load()) for t in dt.trials]
dt.close()
TrialTree.from_datasets(trials_out).save(str(MERGED_NC))

print(f"wrote {MERGED_NC}")
print(f"  s3d      {trials_out[0]['s3d'].shape}")
print(f"  s3d_top  {trials_out[0]['s3d_top'].shape}  dims {list(top_indices)}")

6. Inspect in the GUI, then use it#

ethograph launch

Drop {stem}_s3d.nc on the cover page with your video folder. s3d and s3d_top are ordinary features — both appear in the add-panel popup (➕ / Shift+N), so you can open a heatmap panel for one and a line plot for the other and step through trials against the video. Pin a single column (s3d_top_dims = 492) to get a line plot of one dimension.

Naming them in a segmentation config — dim values, not positions, so the same list means the same thing against either variable:

dims = ", ".join(str(int(i)) for i in top_indices)
print(f"""features:
  columns:
    s3d_top: {{{TOP_DIM}: [{dims}]}}
    # or, equivalently, straight out of the full set:
    # s3d: {{s3d_dims: [{dims}]}}
""")

Important

The ranking read your curated labels, so it is only as good as the trials you labelled — and selecting on a session you later report on leaks the test set. Rank on your training sessions only.

Keeping all 1024 is ~4 KB/frame (≈720 MB per video-hour at 50 fps). Once you have settled on the subset, ds.drop_vars("s3d") makes the file a hundredth of that. To check the subset was worth it, fit twice on the same materialised dataset and compare — see the train.drop_kinds=[video_feature] ablation in docs/source/models/segment/video_features.md.