Curator feedback#

Where you overruled the model, it should pay extra attention next time. That is the whole idea of Model ▸ Curator feedback…: a trial you had to correct is flagged hard in the metadata table’s difficulty column, and the next training run draws it more often. The column is a checklist in the trials table’s filter like curated, so “show me the trials the model got wrong” is one click, and the segmentation pipeline reads it back: train.oversample: {column: difficulty, weights: {hard: 3.0}} draws a hard trial three times as often (Config reference). Auto-flagging only ever adds: a trial you flagged by hand stays hard whatever it scored.

This is not confidence

A low confidence tells you where to look (Model confidence, the grids’ Flag confidence below). It never flags a trial hard. The model already works on the classes it is unsure of; what it cannot know is where it was confident and wrong, and only a curator can tell it. Keep the two apart: confidence guides the review, disagreement guides the training.

By hand#

Ctrl+T flags the current trial, and again puts it back to normal; the difficulty column of the trials table shows where you stand. Use it for a trial that is simply difficult — an odd posture, an occlusion, a rare variant — whether or not the model got it right this time.

By score, once, by your hand#

Which trials the model got wrong is measured, not guessed — but which of them the next run should see more often is decided, not thresholded by default. Every prediction run keeps the labels it wrote in its folder under labels/ ({stem}_predictions.tsv, beside onset_curves.npz), and each trial’s final, curated labels are scored against what its run said — an F1 per trial and event type:

  • State labels match at IoU ≥ 0.5, the f1@50 the pipeline selects its checkpoints on.

  • Point labels match within the run’s own tolerance — the precision the model was trained to, read from the run folder (the onset model’s tolerance_s, half of spot’s focus_window_ms). Nothing to pick; set the popup’s Tolerance only to judge every run at one number, or for an old run folder that carries none.

The scores land in the metadata table as review_f1_state and review_f1_point (blank where a trial had nothing of that type) and nothing else happens. A trial no run predicted into is not scored. The comparison runs by itself, once, the moment the last trial the table shows becomes curated; Score now runs it without waiting, and a workflow step (Score trials against the model) presses it.

Then Histogram… shows the scores as a distribution, one histogram per event type. What you are looking for is a split: a cluster of trials the model handled and a tail, or a second bump, it did not. Put the threshold in the gap — the line and the red bars follow it — and press Flag N trials hard. That is the one moment the scores touch the difficulty column, and it is your press, remembered for next time but never repeated for you. There is no default cut-off because 0.5 means nothing in general: a clean behaviour on a good camera may split at 0.8, a hard one at 0.3.

Flag few. The count under the histogram turns red past a quarter of the scored trials. With train.oversample on, that many hard trials would be most of what the model sees, and a flag most trials carry says nothing. If the whole distribution is low, the answer is a better model or more training data, not more flags.

The scores are anchored to the model

A curated label is a prediction a human accepted, so a session confirmed with N fifty times in a row scores near 1 whatever the model did. The F1 is honest in proportion to how much you actually moved or deleted. Read it as “how much the curator had to change”, not as an independent benchmark — the pipeline’s own evaluation on a held-out split is that.