Specify the discrepancy.
Which stated condition failed? Define the intended result and the observed result without diagnosing the cause in advance.
Output: precise issue statement
Evaluation method / professional practice
Evidence first.
Verdict second.
An anomaly is an observation—not yet a diagnosis. I apply inspection and reliability discipline to AI model outputs, multimodal annotation, evaluator software, and the human workflows around them.
This is my personal working framework, not a formal industry standard or a client's scoring rubric. The examples below come from independently developed tools and synthetic test material.
The six-step working method
Wrong → Proof → Bucket → Badness → Count → Fix
Six questions that prevent a plausible finding from turning into an unsupported conclusion. Each decision leaves something another reviewer can inspect.
Locate the evidence.
Show the relevant source, frame, timestamp, image region, quote, or measurement. A model's claim is not independent ground truth.
Output: inspectable evidence
Find the failed layer.
Was it the model, input, crop, annotation, rubric, search provider, reviewer, or downstream workflow? Different causes need different fixes.
Output: defensible failure category
Assess the consequence.
Separate severity from certainty. A potentially serious issue with weak evidence might require review rather than an automatic failure.
Output: impact rationale + uncertainty
Define what the metric counts.
Count labeled objects, events, claims, or tasks using an explicit denominator. A synthetic regression case is not a population accuracy estimate.
Output: measured cases + denominator
Correct and rerun.
Preserve the original result, document the correction, and check both the failure case and previously passing cases for regressions.
Output: versioned correction + retest
Independent engineering example
The lamp was recognized.
The crop was wrong.
A synthetic red-shade lamp exposed a real development defect in Product Match Bench: object recognition and crop quality are separate measurements.
Case / synthetic fixture
Localization error with possible retrieval consequences
The model identified the product category but produced a box that cut off much of the wide lampshade. This is an independently developed test case, not a client evaluation or verified commercial identity.
Known fixture defect ≠ measured search accuracy
- Wrong
- The detected region did not cover the full visible lampshade.
- Proof
- Compare the synthetic source image, original detection box, and resulting search crop. The omitted side regions are visibly present in the source.
- Bucket
- Object localization and crop construction. Category recognition alone did not establish an adequate search image.
- Badness
- Potential loss of distinguishing product features. The effect on actual shopping rank or exact-product identification was not independently measured.
- Count
- One identified development fixture, used as a regression case. No representative sample size or empirical accuracy rate is implied.
- Fix
- Preserve the original box and give the visual search a more complete image region. Automated browser regression checks verify the synthetic shade remains visible in the actual test upload.
Second-layer lesson
Fixing one failure can expose another.
Sending a whole still photo can protect wide products, but can also overwhelm retrieval when a target is tiny within a cluttered scene. A later focused-search experiment used a newly created synthetic workshop to test a user-selected crop. The goal was to send only the chosen product region to AI recognition and optional visual search.
A correct detection, complete crop, useful retrieval candidate, and verified exact match are four different outcomes. The evaluator needs to keep them separate.
Evaluation boundaries
Do not collapse different judgments.
Some attractive metrics conceal the distinction that matters most. Reliable evaluation preserves it.
Detection ≠ diagnosis
An object-count anomaly or unusual camera color shift requires inspection before being called a generated-video defect.
Task success ≠ efficiency
A robot may finish a placement task after a failed grasp. Preserve the completed task and the unsuccessful attempt.
Severity ≠ certainty
An uncertain event may have serious implications without being confirmed. Use an explicit insufficient-evidence decision when necessary.
Similar ≠ identical
Visual resemblance to a commercial listing does not establish model number, exact product identity, or safe substitution.
Gold label ≠ infallible
Ambiguous instructions, weak reference labels, and reviewer inconsistencies can cause the evaluator itself to fail.
Passing tests ≠ measured accuracy
Software unit and browser tests verify known behavior. Real-world accuracy requires representative, independently labeled examples.
Design implications / The Test Bench
Build the review into the instrument.
The ultimate objective is an easy-to-use product that retains the evidence needed for a technically defensible review.
For the person doing the task
Choose the item or issue. See candidate results alongside the relevant source. Confirm, reject, correct, or mark insufficient evidence. Export a clear decision.
Useful software should not require the user to become a machine-learning engineer.
For the investigator
Keep source provenance, initial output, timestamps or regions, reviewer changes, evidence status, software/model versions, and regression results in a portable record.
Ground-truth labels must be independently established; model outputs must never silently become gold answers.
Current status: Video Failure Lab supports browser-side count-anomaly review; Product Match Bench retrieves candidates and its Evaluation Bench computes metrics from labeled cases; motion analysis remains research-stage. These are independently built tools with defined limits, not a single externally validated AI reliability platform.