<- all tokdocs

Extending RoPE Context From 4K to 32K: Why Position Interpolation Wins

By , founder of Mavric and Apso

View on TikTok ->

The question asks how to make a model trained on 4,000-token sequences use a full 32,000-token document with about a thousand fine-tuning steps. The option that matches the published research is to scale the positions down so 32K tokens fit inside the range the model saw in training, which is the Position Interpolation method. A TikTok from @contextwindows poses this as a "Meta interview question" and ends without giving an answer.

The interview question

The video's headline is "Trained on 4K. Needs 32K." It then asks "What would you change so the model can use all 32K tokens?" and shows four options:

  • A: "Fine-tune on long documents and leave the positions as they are"
  • B: "Scale the positions down so 32K tokens fit inside the range seen in training"
  • C: "Replace rotary embeddings with ALiBi and fine-tune"
  • D: "Limit attention to a sliding window of 4K tokens"

It closes with "Which would you choose, and how would you test that it really uses the far end of the document?" No answer appears in the video. The automatic transcript of the final seconds contains a line suggesting option D with "6,000 tokens," but that text is a transcription artifact. It runs past the end of the audio and contradicts the on-screen option, which says 4K.

I found no primary source showing that Meta asks this question in interviews. The technical content stands on its own without that label.

How rotary position embeddings work

The video says "Rotary embeddings rotate each pair of query and key dimensions by an angle proportional to the token's position." That matches the RoFormer paper, which describes Rotary Position Embedding (RoPE) as encoding absolute position with a rotation, so relative distance shows up inside self-attention.

It then says "Some pairs turn quickly, and the slowest never complete a full turn within 4,000 tokens." The animation shows this with four dials labelled from "fast" to "slowest." The RoFormer abstract I read does not list frequency values, so the video gives the idea without the numbers. The practical consequence is that the slowest-turning pairs reach angles beyond 4K that the model never saw during training.

Why does perplexity rise past the trained length?

The video says "perplexity looks normal for the first 4,000 tokens, then shoots up once positions go past anything the model saw in training." The Position Interpolation paper argues the same mechanism. Extrapolating past trained positions can produce extreme attention scores that break self-attention.

The video gives no perplexity numbers, so the "looks normal, then shoots up" curve is a description of the expected behavior and not a measured result.

What is Position Interpolation?

Position Interpolation is option B. Instead of extrapolating to positions the model has never seen, it compresses the input position indices so a longer sequence fits inside the original range.

According to the paper's abstract, the method extends LLaMA-family models from 7B to 65B parameters to a 32,768-token context window. It needs brief fine-tuning of under 1,000 steps and leaves the architecture unchanged. That fits the video's "about a thousand fine-tuning steps" budget.

The abstract also says performance within the original window stays fairly strong. The authors' theory gives an interpolation error bound at least about 600 times smaller than the extrapolation error. I read only the abstract page, so the training recipe and evaluation tables in the full paper are not checked here.

Why the other options fall short

These reasons follow from how each option is phrased. They are reasoning rather than results from a cited source.

Option A leaves the positions as they are. Angles beyond 4K stay out of distribution, and a thousand steps would have to teach the model to extrapolate on its own.

Option C swaps rotary embeddings for ALiBi. That throws away what pretraining learned about rotary positions and would need far more than a thousand steps to recover.

Option D limits each token to the previous 4K tokens. Perplexity would stay flat, but nothing at the far end of a 32K document could affect a token more than 4K away. That fails the requirement to use all 32,000 tokens.

How to test that the model uses the whole document

The video's closing question asks how you would test that the model really uses the far end of the document. A passkey or needle retrieval task fits: place a fact at several depths in a long input and check whether the model can retrieve it. The Position Interpolation abstract lists passkey retrieval among its evaluations, though I did not read the full protocol.

This test also separates option D from option B. A sliding window keeps perplexity low but would fail retrieval when the passkey sits more than 4K tokens back.

Bottom line

Option B, scaling positions down to fit the trained range, is the answer the research supports, and it matches the Position Interpolation paper's under-1,000-step result at 32,768 tokens. The video states the RoPE mechanism correctly but never reveals an answer. The "Meta interview question" framing has no source.

Sources

Published November 5, 2026. Based on a TikTok by @contextwindows. Full video notes (markdown)