> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokenpath.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Offsets and encoding

> Choose the offset_encoding that matches how your language indexes strings.

Every `start` and `end` TokenPath returns is a character position. Languages disagree about what a "character" is, so you tell TokenPath which unit to count in with `offset_encoding`.

| `offset_encoding`    | Counts              | Use it from                                           |
| -------------------- | ------------------- | ----------------------------------------------------- |
| `utf-16` *(default)* | UTF-16 code units   | JavaScript, TypeScript, Java, C#, Kotlin, Swift, Dart |
| `utf-32`             | Unicode code points | Python, Ruby                                          |
| `utf-8`              | Bytes               | Go, Rust, C, C++, PHP, Elixir                         |

The names match the Language Server Protocol's `PositionEncodingKind`. The response always echoes the encoding it used, so a stored result stays readable later.

## Set it on every request

<CodeGroup>
  ```python Python theme={null}
  response = httpx.post(
      "https://api.tokenpath.ai/v1/attributions",
      headers={"Authorization": f"Bearer {os.environ['TOKENPATH_API_KEY']}"},
      json={
          "document": document,
          "question": question,
          "answer": answer,
          "offset_encoding": "utf-32",
      },
      timeout=90.0,
  ).json()

  assert response["offset_encoding"] == "utf-32"
  for item in response["spans"]:
      if item["source"]:
          source = item["source"]
          assert document[source["start"]:source["end"]] == source["text"]
  ```

  ```javascript JavaScript theme={null}
  // utf-16 is the default, so JavaScript can omit offset_encoding.
  const result = await response.json();

  for (const { source } of result.spans) {
    if (source) console.assert(doc.slice(source.start, source.end) === source.text);
  }
  ```

  ```go Go theme={null}
  // Go strings are byte-indexed: ask for utf-8.
  body, _ := json.Marshal(map[string]string{
  	"document":        document,
  	"question":        question,
  	"answer":          answer,
  	"offset_encoding": "utf-8",
  })
  // ...
  // document[source.Start:source.End] == source.Text
  ```
</CodeGroup>

The same encoding applies in both directions: to explicit `spans` you send and to every offset in the response. On `/v1/attributions/heatmap` it applies to `answer_offsets` and `document_offsets`.

## Why a mismatch is easy to miss

For plain ASCII text, all three encodings give the same numbers. A mismatch only appears after a character they count differently, and then every later offset drifts:

* **Emoji and other characters outside the Basic Multilingual Plane** are 2 units in UTF-16, 1 in UTF-32, and 4 in UTF-8.
* **Accented and non-Latin letters** are 1 unit in UTF-16 and UTF-32 but 2 to 3 bytes in UTF-8.
* **Extracted PDFs** often contain math symbols such as `𝑎` (U+1D44E) or `𝜋` (U+1D70B). They look like ordinary letters but sit outside the BMP, so a paper with equations can shift every highlight that comes after the first formula.

<Warning>
  If you omit `offset_encoding`, you get `utf-16`. Python code that slices a `utf-16` response will be off by one position for each astral character earlier in the string. Always send `"offset_encoding": "utf-32"` from Python.
</Warning>

## Skip offsets entirely

Every range also carries `text`, the exact substring it covers. If you only need to display the evidence, not locate it, use `text` and ignore the integers.