Offsets and encoding

Every start and end TokenPath returns is a character position. Languages disagree about what a “character” is, so you tell TokenPath which unit to count in with offset_encoding.

`offset_encoding`	Counts	Use it from
`utf-16` (default)	UTF-16 code units	JavaScript, TypeScript, Java, C#, Kotlin, Swift, Dart
`utf-32`	Unicode code points	Python, Ruby
`utf-8`	Bytes	Go, Rust, C, C++, PHP, Elixir

The names match the Language Server Protocol’s PositionEncodingKind. The response always echoes the encoding it used, so a stored result stays readable later.

Set it on every request

response = httpx.post(
    "https://api.tokenpath.ai/v1/attributions",
    headers={"Authorization": f"Bearer {os.environ['TOKENPATH_API_KEY']}"},
    json={
        "document": document,
        "question": question,
        "answer": answer,
        "offset_encoding": "utf-32",
    },
    timeout=90.0,
).json()

assert response["offset_encoding"] == "utf-32"
for item in response["spans"]:
    if item["source"]:
        source = item["source"]
        assert document[source["start"]:source["end"]] == source["text"]

// utf-16 is the default, so JavaScript can omit offset_encoding.
const result = await response.json();

for (const { source } of result.spans) {
  if (source) console.assert(doc.slice(source.start, source.end) === source.text);
}

// Go strings are byte-indexed: ask for utf-8.
body, _ := json.Marshal(map[string]string{
	"document":        document,
	"question":        question,
	"answer":          answer,
	"offset_encoding": "utf-8",
})
// ...
// document[source.Start:source.End] == source.Text

The same encoding applies in both directions: to explicit spans you send and to every offset in the response. On /v1/attributions/heatmap it applies to answer_offsets and document_offsets.

Why a mismatch is easy to miss

For plain ASCII text, all three encodings give the same numbers. A mismatch only appears after a character they count differently, and then every later offset drifts:

Emoji and other characters outside the Basic Multilingual Plane are 2 units in UTF-16, 1 in UTF-32, and 4 in UTF-8.
Accented and non-Latin letters are 1 unit in UTF-16 and UTF-32 but 2 to 3 bytes in UTF-8.
Extracted PDFs often contain math symbols such as 𝑎 (U+1D44E) or 𝜋 (U+1D70B). They look like ordinary letters but sit outside the BMP, so a paper with equations can shift every highlight that comes after the first formula.

If you omit offset_encoding, you get utf-16. Python code that slices a utf-16 response will be off by one position for each astral character earlier in the string. Always send "offset_encoding": "utf-32" from Python.

Skip offsets entirely

Every range also carries text, the exact substring it covers. If you only need to display the evidence, not locate it, use text and ignore the integers.

Get started

Core concepts

Advanced

Use cases

Cookbooks

Resources

Offsets and encoding

Set it on every request

Why a mismatch is easy to miss

Skip offsets entirely

​Set it on every request

​Why a mismatch is easy to miss

​Skip offsets entirely

Set it on every request

Why a mismatch is easy to miss

Skip offsets entirely