Concepts

Using the probabilities

Thresholds, certified error rates and handing over to a person.

Myles is calibrated: across many decisions where it gives an option 0.9, that option is right about 90% of the time. That is what lets you set a threshold and trust it.

Automate above a threshold

Pick a threshold per decision on your own held-out cases: automate above it and send the rest to a person. A higher threshold automates fewer cases with fewer errors.

refund = answers["refund"]["noul"]
if refund >= 0.9:
    approve()
elif refund <= 0.1:
    decline()
else:
    send_to_a_person(case, answers)   # pass the probabilities along

Yes/no answers are committed: probabilities for uncertain answers are reported as at least 0.80 towards the chosen side (see noul). Set yes/no thresholds above 0.801 and below 0.199, as in the example. A threshold between them, such as 0.7, would automate uncertain answers that a person should see.

Certified error rates

For a certified error rate, choose the error you can accept first, then find the threshold that keeps decisions inside it on one held-out set and check it on another. Set one threshold for each kind of input: a routing decision and a safety check behave differently.

For the Myles-400M build behind our 91.8% result, on unseen test data: at a 5% target it keeps 73% of decisions with 2.7% error; at 2%, 40% with 1.4%; at 1%, 16% with 0.5%. All results.

Your own data

A threshold set on our data is a starting point, not a guarantee on yours. We help pilot partners set and certify thresholds against their own decisions: request a self-hosted evaluation. Read more in Certified error rates, explained.