Concepts

Writing good questions

Questions and options that the model reads the way you mean them.

The question and its options are the cheapest place to gain accuracy. A clear question with options that cover every case is often worth more than a bigger model. This page sets out what works.

Ask one thing, plainly

• One decision per question. "Is this a refund request and is it within policy?" is two questions. Send both in the same request under different names; each comes back with its own probabilities.
• Ask it the way round you will act on it. "Is a refund permitted under the policy?" is easier to act on, and to set a threshold for, than "Should we refuse this refund?".
• Plain words, no answer hidden in it. Write instructions as you would ask a careful colleague. Leading wording ("Surely this is spam?") pulls the answer.

Options that cover every case

• Describe each option. In criteria, give each label a one-line description of when it applies. Labels alone are often ambiguous: "technical" and "account" overlap until you say which one a password reset belongs to.
• Make the options cover everything. If a case could fit none of them, add an explicit option for it, such as "other": "None of the teams above". Without one, the probability has nowhere honest to go.
• Keep options apart. Two options that both fit a case split the probability between them. Merge them, or say in the descriptions how to tell them apart.
• Keep the option names and order fixed once you have set thresholds, and check again on your own cases after any change (see Check on your own data first).

"route": {
  "type": "choice",
  "instructions": "Which team should handle this message?",
  "criteria": {
    "billing": "Payments, refunds, invoices and charges",
    "account": "Signing in, passwords and account settings",
    "technical": "The product not working as expected",
    "other": "None of the teams above"
  }
}

Keep arithmetic and dates in your code

Myles reads and judges; it is not a calculator. When a rule depends on a count, a sum or the number of days between two dates, work it out in your own code and put the result in the state, then ask about the result.

{
  "model": "myles-g-12b",
  "state": {
    "policy": "Refunds within 30 days with proof of purchase.",
    "case": {"days_since_purchase": 12, "receipt": true}
  },
  "questions": {
    "refund": {"type": "noul", "instructions": "Is a refund permitted under the policy?"}
  }
}

Here days_since_purchase was computed before the call. Sending the purchase date and today's date instead asks the model to do the subtraction, which is where avoidable errors come from.

Put the rule in the state

A refund policy, an eligibility rule or a routing guide belongs in the state, next to the case it applies to, not in the question. The question then stays short and the same rule can serve several questions. See Writing the state.

Scores: describe each level

For a score question, give each level a description, lowest first, that says what a case at that level looks like. Levels described only as numbers are read less consistently than levels described in words.

Start with a small labelled set

Before you rely on a question, write down the right answer for a few dozen real cases, run them, and read the ones where the model disagrees with you. Most disagreements point to a question or an option that can be read two ways. Fix the wording, then set thresholds on a larger set as described in Acting on the answer.