Work

Everyone assumes structured output enforcement works. Nobody measures whether providers actually honour it.

87.6% single-call success collapses to 26.7% over ten chained calls

Every AI tool shows sources now. Almost none verify the sources actually say what the answer claims.

126× token cost difference between retrieval and long context

Recommenders retrain nightly. What happens if the model updates inside the request that reports the click?

Hit rate 0.045 → 0.390 per user, and still last at population level

Correlation dashboards get read as causal claims.

On the benchmark that matters most, the useful output was no number at all

Training-serving skew: features computed one way for training and another for serving. The failure is silent, it is common, and it makes offline metrics untrustworthy.

Sub-10ms online serving with point-in-time correct joins

MLflow tracks experiments, DVC versions datasets and git tracks code, but none of them talk to each other. When a prediction is wrong, reconstructing its origin means piecing together separate logs and human memory.

Any prediction traced back through five hops to the exact raw data row

A bad model reaches production and nobody notices until users do.

Degraded model detected and rolled back in under 60 seconds, no human involved

Anomalies in streaming data need catching in near real time, and simulated benchmarks lie.

Event to deduplicated alert in under two seconds