Explicit answer spans

By default, TokenPath chooses which phrases in the answer to attribute. Pass spans when your application needs to control that itself. For example, you may want to attribute one specific occurrence of a repeated number, or run a controlled experiment over fixed ranges. With spans, TokenPath attributes exactly the ranges you send and nothing else. The response has one entry per range, in the order you sent them.

{
  "document": "Adobe reported revenue of $5.87 billion in Q2, up 11% year over year.",
  "question": "How did Adobe do in Q2?",
  "answer": "Adobe earned $5.87 billion in revenue, up 11%.",
  "spans": [[13, 26], [42, 45]],
  "offset_encoding": "utf-32"
}

Here [13, 26] covers $5.87 billion and [42, 45] covers 11%. The response keeps your order:

{
  "spans": [
    {
      "answer": { "start": 13, "end": 26, "text": "$5.87 billion" },
      "source": { "start": 26, "end": 39, "text": "$5.87 billion", "confidence": 0.962135 }
    },
    {
      "answer": { "start": 42, "end": 45, "text": "11%" },
      "source": { "start": 50, "end": 53, "text": "11%", "confidence": 0.866541 }
    }
  ],
  "offset_encoding": "utf-32"
}

For a longer worked example, see the structured output notebook. It attributes every field of a JSON extraction and resolves two identical 15% values to the two different clauses they came from.

Rules

Each span is a half-open [start, end] range into answer, with 0 <= start < end <= len(answer).
Offsets use the request’s offset_encoding, and so does the len(answer) bound. See Offsets and encoding.
Send at least one span. "spans": [] and "spans": null fail with 422 validation_error. To let TokenPath choose, leave the field out.
A span whose phrase has no support comes back with "source": null.

Keep spans short

Attribution works best on fact-sized phrases, such as a number, a name, a date, an ID, or a short entity. Aim for 1 to 8 words. Numbers, dates, and names are usually best at 1 to 4. Good spans:

$5.87 billion
11%
March 2, 2026
Jordan Lee
remaining performance obligations
9,420 customers

Do not send full sentences, several facts joined by “and”, reasoning chains, paragraphs, or the whole answer as one span. A long span blends attention from several sources into one weak match.

Better	Avoid	Why
`Adobe reported [[revenue of $5.87 billion]], up [[11%]].`	`[[Adobe reported revenue of $5.87 billion, up 11% year over year.]]`	Revenue and growth are separate facts.
`Jordan starts on [[March 2, 2026]] and receives [[$20,000]].`	`[[Jordan starts on March 2, 2026 and receives a $20,000 signing bonus.]]`	The date and the bonus can come from different clauses.
`The company ended with [[9,420 customers]].`	`[[It also ended the quarter with 9,420 customers.]]`	The number is the part the source supports.

The [[...]] marks only show the selected ranges. Strip them before you send the answer.

Selecting spans during generation

One way to get ranges is to ask the model to mark them as it writes:

Wrap only short source-bearing phrases in [[double brackets]].
Use them for numbers, dates, names, entities, and brief factual phrases.
Do not wrap full sentences or multiple facts in one bracket.
Split compound claims into separate bracketed phrases.

Then strip the markers and record where each phrase landed in the clean answer:

import re

def strip_markers(marked: str) -> tuple[str, list[list[int]]]:
    """Remove [[...]] markers and return the clean answer with code-point spans."""
    clean, spans, cursor = [], [], 0
    for match in re.finditer(r"\[\[(.+?)\]\]", marked):
        clean.append(marked[cursor:match.start()])
        start = sum(len(part) for part in clean)
        clean.append(match.group(1))
        spans.append([start, start + len(match.group(1))])
        cursor = match.end()
    clean.append(marked[cursor:])
    return "".join(clean), spans

answer, spans = strip_markers("Adobe earned [[$5.87 billion]] in revenue, up [[11%]].")
# answer == "Adobe earned $5.87 billion in revenue, up 11%."
# spans  == [[13, 26], [42, 45]]  -> send with "offset_encoding": "utf-32"

Offset encoding

Explicit spans are read in the request’s offset_encoding, which defaults to utf-16. The helper above produces Python code-point offsets, so send "offset_encoding": "utf-32" with them. Offsets and encoding covers every language.

Get started

Core concepts

Advanced

Use cases

Cookbooks

Resources

Explicit answer spans

Rules

Keep spans short

Selecting spans during generation

Offset encoding

​Rules

​Keep spans short

​Selecting spans during generation

​Offset encoding

Rules

Keep spans short

Selecting spans during generation

Offset encoding