Skip to main content
TokenPath answers one question about a generated answer:
Which phrases in this answer came from the source, and where exactly in the source?

​
Inputs

Send the strings unchanged. TokenPath never rewrites them, and every offset it returns indexes into the exact strings you sent.

​
What happens

  1. TokenPath’s attribution model reads the document, question, and answer together, and records how much attention each answer token pays to each document token.
  2. It picks out the phrases in the answer that are worth citing, such as numbers, names, dates, and short factual claims. Filler words and connective text are left out.
  3. For each phrase, it finds the stretch of document text that draws the most attention, expands it to whole words, and snaps it to an exact copy of the phrase when the answer quotes the source.
Attribution is measured, not self-reported. TokenPath does not ask a model where its answer came from. It reads the attention of an open-weight attribution model over your text. Your generator is never involved, so this works with any provider.

​
Results

Results come back in answer order. An empty spans list means nothing in the answer could be tied to the document. That is a valid result, not an error. Ranges are half-open (start is inclusive, end is exclusive), so document[start:end] in Python or document.slice(start, end) in JavaScript gives the text. The unit depends on offset_encoding. See Offsets and encoding.

​
Confidence

confidence is the strength of one answer-to-source link, from 0 to 1. Each answer token spreads its attention across everything the model read, and that attention adds up to 1. confidence is the largest share of any one answer-token’s attention that lands inside the returned source range. A phrase that points squarely at one passage scores high. A phrase whose attention is scattered scores low. Because it uses the best-supported token rather than an average, a long phrase is not penalized for its filler words. Use confidence to rank citations, order sources, or style links by strength. It describes how strongly an answer phrase is tied to a source range. It is not a judgment of whether the answer is correct.

​
Threshold

Both attribution endpoints accept threshold (default 0.001, range 0 to 1). Attention below the threshold is dropped before phrases are resolved. The default suits almost every use. Raise it for a sparser raw heatmap.

​
Choosing phrases yourself

By default TokenPath chooses which answer phrases to attribute. To attribute exact ranges that you pick, pass spans. See Explicit answer spans.