Find the defect.
Keep the evidence.

Video Failure Lab

A browser-based workbench for stress-testing generated video against explicit scene constraints. Load a local clip, sample frames, detect supported object classes, and inspect timestamped count anomalies before deciding whether the generator actually failed.

Includes a synthetic test, editable count thresholds, evidence frames, reviewer-controlled scene-context regions, and JSON/CSV exports. Video processing runs in the browser; model libraries and weights are fetched from external hosts.

Current limits: early-stage count-overage triage, not a validated general video benchmark. Identity swaps, object ownership, physics violations, and full prompt compliance are not automatically verified.

Task success and execution quality are different labels.

A robot can complete a task after a failed grasp, correction, or retry. Flattening that into a binary “success” label throws away useful performance information.

In manipulation video, that means recording meaningful task transitions without turning annotation into frame-by-frame noise. A failed grasp followed by a successful grasp is not “failure” overall — but it is still a retry, and retries are reliability data.

Do not convert inference into ground truth.

If an action happens outside the visible frame, the annotation should not quietly upgrade a plausible guess into a factual label.

This sounds obvious until the task pressures annotators to “complete the story.” Reliable datasets resist that pressure. A clean uncertainty boundary is more valuable than confident fiction.

A reviewer disagreement is not automatically a model defect.

When a model answer, evaluator judgment, and written rubric disagree, the fastest route to improvement is to identify which layer is actually inconsistent.

Changing the model to satisfy a broken rubric can make the system worse. Changing a rubric to accommodate a reviewer’s preference can destroy reproducibility. The standard has to survive independent inspection.

Reliability thinking did not start with AI.

Industrial systems teach a useful discipline: the alarming signal, the failed component, and the root cause are not always the same thing.

That mindset carries directly into model evaluation. Whether the signal is vibration, eddy-current data, a robot retry, or a hallucinated answer, the job is to separate observation from explanation and test the explanation against the system.