The problem with confidence
Most AI systems give you an answer and, at best, a confidence that nobody has checked. If a model says it is 90% sure, is it right nine times in ten? Often not. Large generative models are often overconfident, and the number they report has no fixed meaning.
That makes automation risky. You cannot say "act on everything above 0.9" if 0.9 means something different each week.
Calibrate first
Myles is trained and then calibrated so that its probabilities mean what they say: across many decisions where it gives an option 0.9, that option is right about 90% of the time. Calibration is measured on held-out data the model never trained on.
Then certify a threshold
With calibrated probabilities, you can choose an error rate first and find the threshold that achieves it. We set one threshold for each kind of input, because a routing decision and a safety check behave differently, and we set it on one held-out set and check it on another the model has never seen.
Choose the error rate you can accept. Myles keeps what it can decide inside it, and hands over the rest.
What it looks like in numbers
For the Myles-400M build behind our 91.8% result, on unseen test data:
| Target error | Decisions kept | Error on kept decisions |
|---|---|---|
| 5% | 73% | 2.7% |
| 2% | 40% | 1.4% |
| 1% | 16% | 0.5% |
Every result lands inside its target. The trade-off is explicit: the stricter the target, the more decisions go to a person. That is the point. A risk owner can see the trade-off before go-live and sign off the one that fits.
Why your own decisions matter
A threshold certified on our data is a strong start, not a guarantee on yours. Your messages, policies and edge cases differ. That is why every evaluation starts with a sample of your decisions: we measure Myles on them and show you the error rate, the threshold and what would go to a person, before you commit.