Examples

Retrieval relevance

Grade how well a passage answers a question.

A search or retrieval step returns passages for a question. Before passing them to a writer, a person or a larger model, grade how well each one answers the question, and keep only the useful ones.

The request

{
  "model": "myles-g-12b",
  "state": {
    "question": "How long do I have to return an unopened item?",
    "passage": "Unopened items can be returned within 30 days of delivery for a full refund. Opened items are accepted only if faulty."
  },
  "questions": {
    "relevance": {
      "type": "score",
      "instructions": "How well does the passage answer the question?",
      "criteria": [
        "Unrelated to the question",
        "On the topic but does not answer it",
        "Answers part of it",
        "Answers it fully"
      ]
    }
  }
}

Reading the answer

"relevance": {"type": "score", "score": 2.94, "level": 3, "confidence": 0.96,
              "legend": {"0": "Unrelated to the question", "1": "On the topic but does not answer it",
                         "2": "Answers part of it", "3": "Answers it fully"},
              "probabilities": {"0": 0.0, "1": 0.0, "2": 0.06, "3": 0.94}}

• Rank passages by score, the probability-weighted level, rather than by level alone: it separates a confident 3 from a borderline one.
• Drop passages whose probability of level 0 or 1 together is high.
• Send one request per passage. Requests are independent, so they can run in parallel, or as a batch for a large index.

What to watch

• Questions where no passage reaches level 2 or 3: they show gaps in what you have indexed.
• Keep the question and the passage as separate fields in the state, so the model never mistakes one for the other.