# Images

> Send photos, scans and screenshots with a decision: Myles-G 12B and Myles-G 31B read PNG and JPEG images in the same request and return a probability for every option.

Section: API reference. Web page: https://www.launchables.ai/docs/api/images

Some decisions turn on a picture: the receipt behind a refund claim, the photo of a damaged parcel, the screenshot a customer attached. Both Myles-G models on the hosted API, Myles-G 12B and Myles-G 31B, read PNG and JPEG images in the same [`POST /v1/decide`](https://www.launchables.ai/docs/api/decide) request as the text, so the image and the rule it is checked against are decided together, in one call, with a probability for every option as usual.

## How it works

Send each image once, under a name of your choice, in `assets`. Then write that name wherever the image belongs: in the state, in a question's instructions or in a criteria description. The model sees the image at that point in the text.

```json
{
  "model": "myles-g-12b",
  "state": {"receipt": "receipt_photo", "note": "Customer asks for a refund"},
  "questions": {
    "total_matches": {"type": "noul", "instructions": "Does the total printed on receipt_photo equal 23.40?"}
  },
  "assets": {
    "receipt_photo": {"type": "image", "format": "jpeg", "data_base64": "/9j/4AAQSkZJRgABAQ..."}
  },
  "image_budget": 280
}
```

Here `receipt_photo` appears twice, once in the state and once in the question, so the model sees the receipt twice. The answers come back exactly as they do for text; see [Question types](https://www.launchables.ai/docs/question-types).

Images are read by Myles-G 12B (`myles-g-12b`, the hosted default) and by Myles-G 31B (`myles-g-31b`), at `https://gw.launchables.ai` (send `"model": "myles-g-31b"` there; see [Hosted API](https://www.launchables.ai/docs/deploy/hosted)). Both take the same `assets`, limits, `image_budget` and billing described on this page. A request with `assets` for any other model is refused with `422` `images_not_supported`.

## The assets object

`assets` is optional. Each key is a **reference**: any non-empty text up to 1,024 bytes (UTF-8), with all references in a request together up to 64 KiB. Each value describes one image.

| Field | Type | Meaning |
|---|---|---|
| `type` | text, optional | `image`. It can be left out; when it is sent it must be `image`. |
| `format` | text | `png`, `jpg` or `jpeg`. The bytes must really be in that format. |
| `data_base64` | text | The file's bytes in standard base64, with no `data:` prefix. |

A request can also be sent as `multipart/form-data`, with the JSON body in the first part, named `request`, and one file part per image; an asset then names its file part with `"part": "<part name>"` instead of `data_base64`. Send one or the other for each asset, never both. Our SDKs send JSON. An image opt-in header that some existing decision-API clients send is accepted and not required: images need no header.

## How references work

- **Where they are found.** In the state (a text state, or every text value anywhere inside a JSON state), in each question's `instructions`, and in criteria descriptions: what `true` and `false` mean for a `noul` question, the description of each `choice` label, and the text of each `score` level.
- **Where they are not.** Object keys in a JSON state, and option labels: neither the keys of a `choice` criteria object nor the items of a `choice` list are searched, because those are the labels that come back in the answer.
- **Plain text, case-sensitive.** A reference is matched exactly as written, anywhere in the text, and replaced by the image at that position. The text around it stays as you wrote it. Pick names that cannot turn up by accident, such as `receipt_photo` rather than `photo`.
- **Every use shows the image.** A reference written three times shows the image three times, and each one costs tokens.
- **The longest reference wins.** When one reference contains another (`photo` and `photo_2`), the longer one is matched.
- **Put references early in a long state.** The model reads the first 20,000 characters of the state (a JSON state as its indented JSON), and a reference beyond that point is refused with `image_reference_truncated` rather than dropped.
- **Every asset must be used.** An asset whose reference appears nowhere is refused with `unreferenced_asset`, and so is a multipart file part that no asset names.

In a JSON state, put the reference in as a value, for example `{"photo": "photo_1"}`. See [Images in the state](https://www.launchables.ai/docs/state#images-in-the-state).

## Formats and limits

| Limit | Value |
|---|---|
| Formats | PNG or JPEG, still images only (an animated PNG is refused) |
| Size of one image | 16 MiB, after base64 decoding |
| Width and height | Each from 280 to 1568 pixels |
| Assets per request | 8 |
| Images shown in one question | 8 (every use of a reference counts) |
| Request body | 32 MiB (larger bodies get `413`) |
| Reference in the state | Within the first 20,000 characters, the part of the state the model reads (a JSON state counts as its indented JSON) |

Images are never resized or re-encoded. One that is outside these limits is refused with a `400` that names the asset, so resize before you send: a photo straight from a phone camera is usually larger than 1568 pixels a side. For example, with ImageMagick:

```
magick photo.jpg -resize '1568x1568>' -quality 85 receipt.jpg
```

That keeps the shape of the image, so check the shorter side afterwards: a long, thin receipt can end up under 280 pixels across. Cut it into parts, or pad its short side to 280 pixels (ImageMagick's `-extent` does this), before sending.

## Image budget

`image_budget` is optional: `70`, `140` or `280` (the default). It is the most input tokens one image may use. A lower budget is cheaper and faster but sees less detail, and small print is the first thing to suffer. Use `280` for receipts, documents and screenshots with small text; try `140` or `70` for photos where the question is about the scene as a whole, and keep the lower budget if the answers hold up on your own examples. Any other value is refused with `422` `invalid_image_budget`.

## Billing

Images are input tokens, priced like text, and `usage.input_tokens` includes them. `usage` keeps its shape, `{"input_tokens": n, "output_tokens": n}`, and output stays free.

| `image_budget` | Tokens for one use of an image |
|---|---|
| `280` | Up to about 280 |
| `140` | Up to about 140 |
| `70` | Up to about 70 |

Very wide or very tall images use fewer. Each question is answered in its own pass over the state and that question:

- **An image in the state is billed once per request**, however many questions read it (once for each time its reference appears in the state). Every question starts from the same state, and the server reuses it.
- **An image in a question's instructions or criteria is billed for that question**, once for each use.

So when several questions are about the same picture, put its reference in the state rather than in each question.

## Errors

Image errors use the usual body, `{"error": "<message>", "type": "<type>"}`, and the message names the asset. None is worth retrying without changing the request.

| Status and type | When |
|---|---|
| `400` `invalid_assets` | `assets` is not an object, an entry is not an object, a reference is empty or too long, `type` is not `image`, or an entry has neither or both of `data_base64` and `part`. |
| `400` `unsupported_image_format` | `format` is not `png`, `jpg` or `jpeg`, the bytes are not in that format, or the PNG is animated. |
| `400` `invalid_image` | `data_base64` is not valid base64, or the image cannot be read. |
| `400` `image_too_large` | An image is over 16 MiB. |
| `400` `image_dimensions` | A side is under 280 or over 1568 pixels. The message gives the image's size. |
| `400` `too_many_images` | More than 8 assets, or a question would show more than 8 images. |
| `400` `unreferenced_asset` | An asset's reference appears nowhere in the request, or a multipart file part is not named by any asset. |
| `400` `image_reference_truncated` | A reference sits beyond the first 20,000 characters of the state, which the model would not read. The request is refused rather than answered without the image. |
| `400` `bad_multipart` | A `multipart/form-data` body is malformed. See [Request bodies](https://www.launchables.ai/docs/api/decide#request-bodies). |
| `413` `body_too_large` | The body is over 32 MiB. |
| `422` `invalid_image_budget` | `image_budget` is not 70, 140 or 280. |
| `422` `images_not_supported` | The model asked for does not read images (any model other than Myles-G 12B and Myles-G 31B). Send the request without `assets`, or to `myles-g-12b`. |

Every other status is in [Errors and limits](https://www.launchables.ai/docs/api/errors).

## curl

Encode the file with `base64` and build the body with [jq](https://jqlang.org), which reads the encoded image from a file, so large images do not hit the shell's argument limit. `--data-binary @-` sends the body unchanged, with its `Content-Length`.

```
jq -n --rawfile img <(base64 < receipt.jpg | tr -d '\n') '{
  state: {receipt: "receipt_photo", note: "Customer asks for a refund"},
  questions: {
    total_matches: {type: "noul", instructions: "Does the total printed on receipt_photo equal 23.40?"}
  },
  assets: {receipt_photo: {format: "jpeg", data_base64: $img}},
  image_budget: 280
}' | curl https://api.launchables.ai/v1/decide \
  -H "Authorization: Bearer $LAUNCHABLES_API_TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @-
```

It runs in bash or zsh, on Linux and macOS: `tr -d '\n'` removes the line breaks some versions of `base64` add.

```json
{
  "model": "myles-g-12b",
  "answers": { "total_matches": { "type": "noul", "noul": 0.94 } },
  "usage": { "input_tokens": 331, "output_tokens": 1 }
}
```

Illustrative probability and token counts.

## TypeScript

With the [JavaScript SDK](https://www.launchables.ai/docs/sdk-javascript), version 0.4.0 or later. `imageFromPath` reads the file as it is (Node only) and `imageFromBytes` takes a `Uint8Array`, `Buffer` or `ArrayBuffer`; both detect PNG or JPEG from the bytes and refuse anything else, or anything over 16 MiB, before sending.

```
import { Launchables, imageFromPath, noul } from "launchables";

const client = new Launchables(); // reads LAUNCHABLES_API_TOKEN

const resp = await client.decide(
  { receipt: "receipt_photo", note: "Customer asks for a refund" },
  { total_matches: noul("Does the total printed on receipt_photo equal 23.40?") },
  { assets: { receipt_photo: await imageFromPath("receipt.jpg") }, imageBudget: 280 },
);
console.log(resp.answers.total_matches.probability, resp.usage.input_tokens);
```

## Python

With the [Python SDK](https://github.com/launchables-ai/launchables-sdk/tree/main/python), version 0.4.0 or later. `Image.from_path` and `Image.from_bytes` check the format and size the same way.

```
from launchables import Image, Launchables, noul

client = Launchables()  # reads LAUNCHABLES_API_TOKEN

resp = client.decide(
    {"receipt": "receipt_photo", "note": "Customer asks for a refund"},
    {"total_matches": noul("Does the total printed on receipt_photo equal 23.40?")},
    assets={"receipt_photo": Image.from_path("receipt.jpg")},
    image_budget=280,
)
print(resp.answers["total_matches"].probability, resp.usage.input_tokens)
```

Without an SDK, read the file, encode it and send it as JSON:

```
import base64, os, requests

with open("receipt.jpg", "rb") as f:
    image = base64.b64encode(f.read()).decode("ascii")

response = requests.post(
    "https://api.launchables.ai/v1/decide",
    headers={"Authorization": f"Bearer {os.environ['LAUNCHABLES_API_TOKEN']}"},
    json={
        "state": {"receipt": "receipt_photo", "note": "Customer asks for a refund"},
        "questions": {"total_matches": {"type": "noul", "instructions": "Does the total printed on receipt_photo equal 23.40?"}},
        "assets": {"receipt_photo": {"format": "jpeg", "data_base64": image}},
    },
    timeout=60,
)
response.raise_for_status()
print(response.json()["answers"]["total_matches"]["noul"])
```

## Privacy

Images are treated like the rest of the request: the endpoint does not store them, and its access log records only the method, the path, the status and the time taken.