Building self-critiquing video systems

TL;DR: Internal model probes flagged candidate physical failures for human verification, while an agent ran data preparation, training and evaluation. Fine-tuning Qwen3-VL-8B with LoRA increased glitch detection from 20% to 36% at a 10% false-positive rate on 1,223 clips with scenes and prompts held out. Timing and generalization remain limited, and adding probe scores did not consistently improve overall detection. We’re now testing whether verified failure examples improve our World-Action Model.

At Hedra, we’re working toward models that can identify their own failures and learn from verified feedback. For video models, that starts with detecting when an object warps, disappears or moves without a physical cause. In this work, we use a vision-language model (VLM) to turn those failures into descriptions and timestamps that researchers can check.

We used human-labeled examples from Physion-Eval to identify failure signals and develop probes. The probes flagged possible defects in our generated videos, which researchers checked and labeled for training a glitch verifier. An agent prepared the data and ran training and evaluation; researchers reviewed the results and approved each training round.

The sequence below shows why automated alerts still need human verification.

Certain visual glitches persist for only a single frame, whereas others manifest gradually as an object deforms or interacts with its environment. Both types can easily evade detection during standard playback. Manual frame-by-frame review is inefficient, and conventional object detection tools often fail to categorize physical anomalies correctly.

Why object tracking misses physical glitches

We initially evaluated standard object detection and tracking frameworks, including YOLO11x and GroundingDINO, to monitor object boundaries and trajectories. However, a stable bounding box can easily mask structural deformations within an object, while an unstable tracking trajectory may trigger false alarms in the absence of an actual physical failure.

Across our benchmark tests, tracking metrics, visual feature extractors and segmentation models performed near chance levels for glitch classification.

Elephant

The mask follows the elephant as its trunk changes length.

Color-change ball

Only the hand is selected.

Tracking alone wasn’t enough to identify physical failures reliably. We next examined whether a vision-language model’s internal signals could reveal failures that its final answer missed.

Finding failure signals inside the VLM

We used lightweight linear classifiers, or probes, to read failure signals from the VLM’s internal activations. Each probe targeted a specific failure type, such as an abrupt color change, geometric warping or unnatural bending.

Physion-Eval provided human descriptions of physical glitches and timestamps showing when they occurred. We also created edited videos with known defects to test what the probes detected. These comparisons showed that the probes responded to inconsistencies over time, rather than just visual differences in individual frames.

This is an interpretability approach to model development: use internal signals to identify failures worth investigating. The probes reveal associations with specific failure types; activation-patching experiments test whether those signals influence the verifier’s score.

Using probes to guide human annotation

We used the probes to highlight candidate defects in generated videos for human review. Reviewers started with the flagged frames, checked for deformations or state changes, and marked when each defect began and ended. The probes supplied candidate detections; people verified the failures and supplied the training labels.

The agent organized datasets, generated controlled video edits, and ran training and evaluation scripts. Researchers reviewed candidate failures and approved each training round.

Fine-tuning Qwen3-VL-8B to detect video glitches

We fine-tuned Qwen3-VL-8B via LoRA on human-verified annotations of our generated videos, while keeping the visual encoder frozen. We evaluated the verifier on Physion-Eval, which we also used to develop the probes. The model reports whether a failure is present, its category and severity (how noticeable and physically implausible it is), timestamps, and an explanation.

Text explanations and timestamps let researchers check predictions against the video and identify targets for the next training round, including missed deformations, false alarms and inaccurate explanations.

Measuring glitch detection on Physion-Eval

On the held-out Physion-Eval test set, the fine-tuned verifier detected 36% of glitched clips at a 10% false-positive rate, compared with 20% for the zero-shot Qwen model. That is a 1.8× increase in glitch detection at the same false-positive rate.

The test set includes held-out scenes and text prompts comprising 1,053 clips rated 4 or 5 on a 5-point glitch severity scale, alongside 170 clean baseline clips. Mild artifacts were excluded to focus strictly on severe physical defects. At a 30% false-positive rate, detection rose to 65%, trading more false alarms for higher recall.

In the color-change scenario, standard tracking isolates the person while failing to track the ball. Conversely, the fine-tuned VLM correctly pinpoints the color transition at 4.2 seconds, closely aligning with the human annotation timestamp of 4.4 seconds.

Tracking pipeline

Person detected; ball missed.

Post-trained VLM

“At 4.2 s, the yellow sphere in the hand suddenly turns blue without any change in camera angle.”

In the basketball example below, conventional tracking generates numerous false alerts. Similarly, in the tablecloth sequence, standard tracking alarms fail to correlate with the actual annotated failure event.

Basketball: tracking pipeline

15 alarms; 3 overlap the glitch.

Basketball: post-trained VLM

“The basketball moves on its own without any external force, and bounces too smoothly.”

Vanishing cloth: tracking pipeline

96 alarms; 3 overlap the glitch.

Vanishing cloth: post-trained VLM

“At 2.2 s, the tablecloth on the table suddenly disappears into thin air.”

The cloth detection's timestamp matches the human annotation.

What improves glitch detection, and what still fails

Failure signals the verifier’s verdict misses

Better clip screening still leaves failures the verifier misses. On controlled synthetic clips, probes detected structural and deformation signals that did not always appear in the final verdict. In activation-patching tests, copying internal signals from a glitched clip into its clean twin changed the verifier’s score.

Targeted gains do not consistently improve overall detection

The category-specific tradeoffs support testing probes as an aid to human annotation, while evaluating the resulting verifier across a broad range of failures. These were small exploratory tests, with one training run per configuration and no independent replication.

Pinpointing when a video glitch occurs

A useful verifier needs to locate and explain a failure as well as flag the clip. A detection counted as correctly timed when it fell within roughly half a second of the human-labeled timestamp. Across about 1,150 labeled events in roughly 635 videos, the fine-tuned 8B model correctly timed 13% of events; the 32B model reached 16%. The examples below show both a matching explanation and a detected deformation whose final shape the model describes incorrectly.

Water

Model, Sora 2, 1.2 s: “The water suddenly starts swirling without any external force.”

Annotator: “The water forms a vortex without any external force at 1.2s.”

Nut

Model, Veo 3.1 Fast, 3.8 s: “The nut suddenly deforms without any external force, then is completely flattened.”

Annotator: “The nut undergoes an illogical deformation and becomes a cube at 4.1s.”

Held-out examples: the water explanation matches the human annotation. The verifier detects the nut’s deformation but describes its final shape incorrectly.

Testing generalization to new videos

Tests on additional generated videos showed that performance did not transfer consistently to new content. Reliable generalization needs evaluation on held-out scenes and prompts. Candidate detections in individual clips can guide human review, but they do not establish reliable detection across a new dataset.

Katana

The blade stretches.

Neon signs

The lettering morphs.

The katana blade stretches and the neon-sign lettering changes shape. These clips illustrate failure cases for review; they do not establish generalization performance.

Can verified failures improve our World-Action Model?

The workflow connects candidate detections, human-verified labels and agent-run training and evaluation. We’re now testing whether these verified failure examples improve our World-Action Model (WAM). Evaluation on held-out scenes and prompts will need to establish whether those updates improve performance on new videos.

The evaluation needs to track detection quality, explanation accuracy and the human effort required to verify labels. Improvements on one failure type must also be checked for regressions on others.

Build with Hedra Model Factory

Hedra Model Factory brings together forward deployed researchers, agents, GPU compute and optimized inference to help solve your problem with a multimodal model. Our team works with yours to prepare data, develop or source environments where needed, run post-training and evaluate results. We then help you deploy the model and use production data to inform its next version.

Bring us the problem you need to solve. We help you build, run and keep improving a model you own.