Correct output ≠ correct process.
A system can land on the right answer for the wrong reason. Reliability work looks past the surface result and tests the path that produced it.
AI RELIABILITY / EVALUATION / TECHNICAL QA
Models don’t fail randomly.
I analyze why.
I work at the boundary between AI outputs and the systems that produce them — model behavior, prompts, rubrics, annotation, verification, and human workflow. My background in industrial QA and reliability means I care less about whether something looks impressive and more about whether the process survives contact with reality.
A correct answer does not prove a sound process.
Operating principles
Reliability before theater.
The useful question is rarely “Did the model say something plausible?” It is “What produced this result, what can break, and how do we know the system is still trustworthy when the easy example disappears?”
Generation is cheap. Verification is expensive.
AI can produce plausible material faster than humans can validate it. The bottleneck moves from creation to evidence, review, and confidence calibration.
Separate the failure source.
Model failure, prompt failure, rubric failure, annotation failure, and reviewer failure are different defects. Fixing the wrong layer creates fake improvement.
The workflow may be smarter than the hardware.
“AI-capable” hardware is only part of the system. The real intelligence often lives in sensing, orchestration, validation, fallback logic, and the human operating loop around it.
Failure decomposition
Stop blaming “the AI.”
An observed failure is the start of the investigation, not the diagnosis. Decompose the system before deciding what needs to change.
Observed outputWhat actually happened?
ModelCapability, hallucination, calibration.
PromptAmbiguity, missing constraints, context.
RubricBad criteria, contradictions, hidden assumptions.
AnnotationGround-truth quality, label drift, evidence gaps.
WorkflowHuman review, tooling, handoffs, downstream action.
Cross-domain experience
Physical systems taught me how to distrust easy answers.
The tools changed. The reliability habit did not: isolate the signal, understand the failure mode, test against evidence, and keep responsibility attached to the right layer.
Industrial reliability
Machines, signals, failure paths.
More than a decade of QA and reliability work across non-destructive testing, vibration analysis, HVAC/chiller systems, VFDs, technical reporting, and data-assisted troubleshooting.
NDTVIBRATIONHVACRROOT CAUSE
AI evaluation
Models, rubrics, evidence, human review.
Multimodal and robotics evaluation, image/video annotation QA, search and location relevance, rubric interpretation, error taxonomy, and evidence-first review.
MULTIMODALROBOTICSQARUBRICSVERIFICATION
Selected field notes
What I’m thinking about.
Short technical arguments from the same public voice I use on X: direct, testable, and more interested in failure modes than hype.
Process integrity
The output is right. The system can still be wrong.
Why evaluation has to inspect causal path, not just final-answer accuracy.
Verification economicsAI made generation cheap. It did not make truth cheap.
The cost moved downstream into checking, evidence, and human judgment.
Error taxonomyModel failure is only one bucket.
Prompt, rubric, annotation, and workflow defects deserve separate diagnoses.