API reference

Images

Send photos, scans and screenshots with a decision on Myles-G 12B or Myles-G 31B.

Some decisions turn on a picture: the receipt behind a refund claim, the photo of a damaged parcel, the screenshot a customer attached. Both Myles-G models on the hosted API, Myles-G 12B and Myles-G 31B, read PNG and JPEG images in the same POST /v1/decide request as the text, so the image and the rule it is checked against are decided together, in one call, with a probability for every option as usual.

How it works

Send each image once, under a name of your choice, in assets. Then write that name wherever the image belongs: in the state, in a question's instructions or in a criteria description. The model sees the image at that point in the text.

{
  "model": "myles-g-12b",
  "state": {"receipt": "receipt_photo", "note": "Customer asks for a refund"},
  "questions": {
    "total_matches": {"type": "noul", "instructions": "Does the total printed on receipt_photo equal 23.40?"}
  },
  "assets": {
    "receipt_photo": {"type": "image", "format": "jpeg", "data_base64": "/9j/4AAQSkZJRgABAQ..."}
  },
  "image_budget": 280
}

Here receipt_photo appears twice, once in the state and once in the question, so the model sees the receipt twice. The answers come back exactly as they do for text; see Question types.

Images are read by Myles-G 12B (myles-g-12b, the hosted default) and by Myles-G 31B (myles-g-31b), at https://gw.launchables.ai (send "model": "myles-g-31b" there; see Hosted API). Both take the same assets, limits, image_budget and billing described on this page. A request with assets for any other model is refused with 422 images_not_supported.

The assets object

assets is optional. Each key is a reference: any non-empty text up to 1,024 bytes (UTF-8), with all references in a request together up to 64 KiB. Each value describes one image.

A request can also be sent as multipart/form-data, with the JSON body in the first part, named request, and one file part per image; an asset then names its file part with "part": "<part name>" instead of data_base64. Send one or the other for each asset, never both. Our SDKs send JSON. An image opt-in header that some existing decision-API clients send is accepted and not required: images need no header.

How references work

• Where they are found. In the state (a text state, or every text value anywhere inside a JSON state), in each question's instructions, and in criteria descriptions: what true and false mean for a noul question, the description of each choice label, and the text of each score level.
• Where they are not. Object keys in a JSON state, and option labels: neither the keys of a choice criteria object nor the items of a choice list are searched, because those are the labels that come back in the answer.
• Plain text, case-sensitive. A reference is matched exactly as written, anywhere in the text, and replaced by the image at that position. The text around it stays as you wrote it. Pick names that cannot turn up by accident, such as receipt_photo rather than photo.
• Every use shows the image. A reference written three times shows the image three times, and each one costs tokens.
• The longest reference wins. When one reference contains another (photo and photo_2), the longer one is matched.
• Put references early in a long state. The model reads the first 20,000 characters of the state (a JSON state as its indented JSON), and a reference beyond that point is refused with image_reference_truncated rather than dropped.
• Every asset must be used. An asset whose reference appears nowhere is refused with unreferenced_asset, and so is a multipart file part that no asset names.

In a JSON state, put the reference in as a value, for example {"photo": "photo_1"}. See Images in the state.

Formats and limits

Images are never resized or re-encoded. One that is outside these limits is refused with a 400 that names the asset, so resize before you send: a photo straight from a phone camera is usually larger than 1568 pixels a side. For example, with ImageMagick:

magick photo.jpg -resize '1568x1568>' -quality 85 receipt.jpg

That keeps the shape of the image, so check the shorter side afterwards: a long, thin receipt can end up under 280 pixels across. Cut it into parts, or pad its short side to 280 pixels (ImageMagick's -extent does this), before sending.

Image budget

image_budget is optional: 70, 140 or 280 (the default). It is the most input tokens one image may use. A lower budget is cheaper and faster but sees less detail, and small print is the first thing to suffer. Use 280 for receipts, documents and screenshots with small text; try 140 or 70 for photos where the question is about the scene as a whole, and keep the lower budget if the answers hold up on your own examples. Any other value is refused with 422 invalid_image_budget.

Billing

Images are input tokens, priced like text, and usage.input_tokens includes them. usage keeps its shape, {"input_tokens": n, "output_tokens": n}, and output stays free.

Very wide or very tall images use fewer. Each question is answered in its own pass over the state and that question:

• An image in the state is billed once per request, however many questions read it (once for each time its reference appears in the state). Every question starts from the same state, and the server reuses it.
• An image in a question's instructions or criteria is billed for that question, once for each use.

So when several questions are about the same picture, put its reference in the state rather than in each question.

Errors

Image errors use the usual body, {"error": "<message>", "type": "<type>"}, and the message names the asset. None is worth retrying without changing the request.

Every other status is in Errors and limits.

curl

Encode the file with base64 and build the body with jq, which reads the encoded image from a file, so large images do not hit the shell's argument limit. --data-binary @- sends the body unchanged, with its Content-Length.

jq -n --rawfile img <(base64 < receipt.jpg | tr -d '\n') '{
  state: {receipt: "receipt_photo", note: "Customer asks for a refund"},
  questions: {
    total_matches: {type: "noul", instructions: "Does the total printed on receipt_photo equal 23.40?"}
  },
  assets: {receipt_photo: {format: "jpeg", data_base64: $img}},
  image_budget: 280
}' | curl https://api.launchables.ai/v1/decide \
  -H "Authorization: Bearer $LAUNCHABLES_API_TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @-

It runs in bash or zsh, on Linux and macOS: tr -d '\n' removes the line breaks some versions of base64 add.

{
  "model": "myles-g-12b",
  "answers": { "total_matches": { "type": "noul", "noul": 0.94 } },
  "usage": { "input_tokens": 331, "output_tokens": 1 }
}

Illustrative probability and token counts.

TypeScript

With the JavaScript SDK, version 0.4.0 or later. imageFromPath reads the file as it is (Node only) and imageFromBytes takes a Uint8Array, Buffer or ArrayBuffer; both detect PNG or JPEG from the bytes and refuse anything else, or anything over 16 MiB, before sending.

import { Launchables, imageFromPath, noul } from "launchables";

const client = new Launchables(); // reads LAUNCHABLES_API_TOKEN

const resp = await client.decide(
  { receipt: "receipt_photo", note: "Customer asks for a refund" },
  { total_matches: noul("Does the total printed on receipt_photo equal 23.40?") },
  { assets: { receipt_photo: await imageFromPath("receipt.jpg") }, imageBudget: 280 },
);
console.log(resp.answers.total_matches.probability, resp.usage.input_tokens);

Python

With the Python SDK, version 0.4.0 or later. Image.from_path and Image.from_bytes check the format and size the same way.

from launchables import Image, Launchables, noul

client = Launchables()  # reads LAUNCHABLES_API_TOKEN

resp = client.decide(
    {"receipt": "receipt_photo", "note": "Customer asks for a refund"},
    {"total_matches": noul("Does the total printed on receipt_photo equal 23.40?")},
    assets={"receipt_photo": Image.from_path("receipt.jpg")},
    image_budget=280,
)
print(resp.answers["total_matches"].probability, resp.usage.input_tokens)

Without an SDK, read the file, encode it and send it as JSON:

import base64, os, requests

with open("receipt.jpg", "rb") as f:
    image = base64.b64encode(f.read()).decode("ascii")

response = requests.post(
    "https://api.launchables.ai/v1/decide",
    headers={"Authorization": f"Bearer {os.environ['LAUNCHABLES_API_TOKEN']}"},
    json={
        "state": {"receipt": "receipt_photo", "note": "Customer asks for a refund"},
        "questions": {"total_matches": {"type": "noul", "instructions": "Does the total printed on receipt_photo equal 23.40?"}},
        "assets": {"receipt_photo": {"format": "jpeg", "data_base64": image}},
    },
    timeout=60,
)
response.raise_for_status()
print(response.json()["answers"]["total_matches"]["noul"])

Privacy

Images are treated like the rest of the request: the endpoint does not store them, and its access log records only the method, the path, the status and the time taken.