Examples

Hand over by cost

Ask a larger model or a person only when it is worth it.

A cheap, fast model can answer most cases on its own. The question is what to do with the rest. Here a refund decision is made on Myles-G 12B; when the answer is not settled, the same request goes to Myles-G 31B; when that is not settled either, a person decides. The choice is made by expected cost, so it moves with what mistakes cost you.

The request

{
  "model": "myles-g-12b",
  "state": {
    "policy": "Refunds within 30 days for unused items. Opened software is not refundable unless faulty.",
    "case": {"days_since_purchase": 21, "item": "design software, boxed", "opened": true,
             "customer_says": "It crashes every time I export a file."}
  },
  "questions": {
    "decision": {
      "type": "choice",
      "instructions": "What does the policy say should happen to this refund request?",
      "criteria": {
        "refund": "The policy allows a refund",
        "decline": "The policy does not allow a refund",
        "need_more": "The policy depends on something the case does not say"
      }
    }
  }
}

Reading the answer

"decision": {"type": "choice", "choice": "refund", "confidence": 0.36,
             "probabilities": {"refund": 0.58, "decline": 0.22, "need_more": 0.20}}

Weigh each action by what it costs when it turns out wrong, then pick the cheapest:

# request is the JSON above; decide(request) posts it to /v1/decide and returns the parsed response
answer = decide(request)["answers"]["decision"]

COSTS = {  # cost of each action if the right answer is refund / decline / need_more
    "refund":   {"refund": 0, "decline": 30, "need_more": 15},
    "decline":  {"refund": 25, "decline": 0, "need_more": 15},
    "escalate": {"refund": 2, "decline": 2, "need_more": 2},     # the larger model's price, plus the share it still sends to a person
}

def cheapest(probabilities, costs):
    expected = {action: sum(probabilities[label] * cost for label, cost in row.items())
                for action, row in costs.items()}
    return min(expected, key=expected.get)

action = cheapest(answer["probabilities"], COSTS)
if action == "escalate":
    answer = decide({**request, "model": "myles-g-31b"})["answers"]["decision"]
    action = cheapest(answer["probabilities"], {**COSTS, "escalate": {k: 4 for k in COSTS["refund"]}})
    if action == "escalate":
        send_to_a_person(request["state"]["case"], answer)

With the answer above, refunding costs about 9.6 on average and declining about 17.5, so escalating at 2 wins and the request goes to the 31B. Give escalation its full cost: the larger model's price plus the share of cases it will still pass to a person. On the 31B, escalating means a person, at 4. A confident 12B answer, such as 0.94 for refund, costs 1.8 to act on and is not escalated.

What to watch

• The share answered by each step: most requests should stop at the 12B.
• The spend on the second step, from the console's usage by model.
• Cases people overturn: they show which costs to adjust.