Concepts

Question types

Yes or no, one of several labels, or a level on a scale.

Every question has a type, the instructions (the question itself, in plain language) and, for choice and score, the criteria that list the options. Each question is read separately against the same state.

noul: yes or no

Optional criteria describe what each answer means, under true and false (or yes and no). Give both sides, or neither. A side may be null, and is then shown as "Yes" or "No". The response gives noul, the probability of yes.

"eligible": {
  "type": "noul",
  "instructions": "Does the applicant meet the residency rule?",
  "criteria": {
    "true": "Lived in the UK for at least 3 of the last 5 years",
    "false": "Residency not shown, or under 3 years"
  }
}

"eligible": { "type": "noul", "noul": 0.87 }

Yes/no answers are committed: probabilities for uncertain answers are reported as at least 0.80 towards the chosen side. When the model's own probability of yes falls between 0.2 and 0.8, the answer goes to its more likely side and is reported at one fixed probability, 0.801 today, so noul is never between 0.2 and 0.8. Answers outside that range are returned as read. A committed answer is less certain than its number: on our own held-out cases, committed answers are right about 70% of the time, not 80%. See Thresholds for where to set yes/no thresholds.

choice: one of several labels

criteria maps each label to a short description, or is a list of labels. A description may be null when the label says it all ({"calm": null, "upset": null}). The response gives the chosen label, its confidence and a probability for every label. Up to 26 labels per question: the options are read as letters A to Z in one pass.

"route": {
  "type": "choice",
  "instructions": "Which team should handle this message?",
  "criteria": {
    "billing": "Payments, invoices and refunds",
    "returns": "Returning or exchanging an item",
    "technical": "The product does not work",
    "other": "Anything else"
  }
}

"route": {
  "type": "choice",
  "choice": "returns",
  "confidence": 0.83,
  "probabilities": { "billing": 0.06, "returns": 0.89, "technical": 0.01, "other": 0.04 }
}

score: a level on an ordered scale

criteria is a list of level descriptions, lowest first; levels are numbered from 0. The response gives the most likely level, the probability-weighted score, a confidence, a legend (each level's description, by number) and a probability for every level.

"priority": {
  "type": "score",
  "instructions": "How urgent is this complaint?",
  "criteria": ["Routine", "Needs a reply this week", "Needs a reply today", "Safety risk, act now"]
}

"priority": {
  "type": "score",
  "score": 1.92,
  "level": 2,
  "confidence": 0.81,
  "legend": { "0": "Routine", "1": "Needs a reply this week", "2": "Needs a reply today", "3": "Safety risk, act now" },
  "probabilities": { "0": 0.03, "1": 0.12, "2": 0.75, "3": 0.10 }
}

Confidence

confidence runs from 0 to 1 and is our own measure of how settled an answer is. It is not a probability that the answer is right; for that, read probabilities.

• choice: the probability of the chosen label minus that of the runner-up. 0 is a tie at the top; 1 puts everything on one label.
• score: 1 minus the expected distance from the middle (median) level, divided by half the scale. Being unsure between neighbouring levels lowers it a little; being split between the two ends of the scale makes it 0.
• noul answers have no confidence: the probability of yes already says it.

Choosing a type

• Use noul for a single rule or check: is it allowed, is it present, is it supported.
• Use choice when exactly one of several labels applies: a team, a category, a tool.
• Use score when the options are ordered and being one level out matters less than being three out: urgency, risk, severity.