---
title: "Extending RoPE Context From 4K to 32K: Why Position Interpolation Wins"
date: "2026-11-05"
video_id: "7704524238245334286"
tiktok_url: "https://www.tiktok.com/@contextwindows/video/7704524238245334286"
creator: "@contextwindows"
category: "AI"
tags: ["rope", "position interpolation", "context window", "llm interview", "rotary embeddings"]
excerpt: "A model trained on 4K tokens must use 32K. Why scaling positions down (Position Interpolation) beats fine-tuning, ALiBi or a sliding window, with sources."
---
The question asks how to make a model trained on 4,000-token sequences use a full 32,000-token document with about a thousand fine-tuning steps. The option that matches the published research is to scale the positions down so 32K tokens fit inside the range the model saw in training, which is the Position Interpolation method. A TikTok from @contextwindows poses this as a "Meta interview question" and ends without giving an answer.

## The interview question

The video's headline is "Trained on 4K. Needs 32K." It then asks "What would you change so the model can use all 32K tokens?" and shows four options:

- A: "Fine-tune on long documents and leave the positions as they are"
- B: "Scale the positions down so 32K tokens fit inside the range seen in training"
- C: "Replace rotary embeddings with ALiBi and fine-tune"
- D: "Limit attention to a sliding window of 4K tokens"

It closes with "Which would you choose, and how would you test that it really uses the far end of the document?" No answer appears in the video. The automatic transcript of the final seconds contains a line suggesting option D with "6,000 tokens," but that text is a transcription artifact. It runs past the end of the audio and contradicts the on-screen option, which says 4K.

I found no primary source showing that Meta asks this question in interviews. The technical content stands on its own without that label.

## How rotary position embeddings work

The video says "Rotary embeddings rotate each pair of query and key dimensions by an angle proportional to the token's position." That matches the RoFormer paper, which describes Rotary Position Embedding (RoPE) as encoding absolute position with a rotation, so relative distance shows up inside self-attention.

It then says "Some pairs turn quickly, and the slowest never complete a full turn within 4,000 tokens." The animation shows this with four dials labelled from "fast" to "slowest." The RoFormer abstract I read does not list frequency values, so the video gives the idea without the numbers. The practical consequence is that the slowest-turning pairs reach angles beyond 4K that the model never saw during training.

## Why does perplexity rise past the trained length?

The video says "perplexity looks normal for the first 4,000 tokens, then shoots up once positions go past anything the model saw in training." The Position Interpolation paper argues the same mechanism. Extrapolating past trained positions can produce extreme attention scores that break self-attention.

The video gives no perplexity numbers, so the "looks normal, then shoots up" curve is a description of the expected behavior and not a measured result.

## What is Position Interpolation?

Position Interpolation is option B. Instead of extrapolating to positions the model has never seen, it compresses the input position indices so a longer sequence fits inside the original range.

According to the paper's abstract, the method extends LLaMA-family models from 7B to 65B parameters to a 32,768-token context window. It needs brief fine-tuning of under 1,000 steps and leaves the architecture unchanged. That fits the video's "about a thousand fine-tuning steps" budget.

The abstract also says performance within the original window stays fairly strong. The authors' theory gives an interpolation error bound at least about 600 times smaller than the extrapolation error. I read only the abstract page, so the training recipe and evaluation tables in the full paper are not checked here.

## Why the other options fall short

These reasons follow from how each option is phrased. They are reasoning rather than results from a cited source.

**Option A** leaves the positions as they are. Angles beyond 4K stay out of distribution, and a thousand steps would have to teach the model to extrapolate on its own.

**Option C** swaps rotary embeddings for ALiBi. That throws away what pretraining learned about rotary positions and would need far more than a thousand steps to recover.

**Option D** limits each token to the previous 4K tokens. Perplexity would stay flat, but nothing at the far end of a 32K document could affect a token more than 4K away. That fails the requirement to use all 32,000 tokens.

## How to test that the model uses the whole document

The video's closing question asks how you would test that the model really uses the far end of the document. A passkey or needle retrieval task fits: place a fact at several depths in a long input and check whether the model can retrieve it. The Position Interpolation abstract lists passkey retrieval among its evaluations, though I did not read the full protocol.

This test also separates option D from option B. A sliding window keeps perplexity low but would fail retrieval when the passkey sits more than 4K tokens back.

## Bottom line

Option B, scaling positions down to fit the trained range, is the answer the research supports, and it matches the Position Interpolation paper's under-1,000-step result at 32,768 tokens. The video states the RoPE mechanism correctly but never reveals an answer. The "Meta interview question" framing has no source.

## Sources

- [Extending Context Window of Large Language Models via Positional Interpolation, arXiv 2306.15595](https://arxiv.org/abs/2306.15595): the position compression method, the 32,768-token extension, the under-1,000-step fine-tuning and the 600-times error bound.
- [RoFormer: Enhanced Transformer with Rotary Position Embedding, arXiv 2104.09864](https://arxiv.org/abs/2104.09864): the rotation-based description of RoPE.


## Video description and verification notes

**The file is a 61.65 s vertical clip, 1080x1920 (9:16), in an MP4/MOV container (isom). The video stream is HEVC Main, 30 fps, 118,369 bit/s, 1,849 frames, 61.63 s. The audio stream is HE-AACv2 stereo at 44.1 kHz, 48,159 bit/s, 61.65 s. The total file is 1,335,507 bytes at 173,293 bit/s, and the TikTok format id is bytevc1_1080p_173290-1. It was posted at epoch 1793849356, which is 2026-11-05 03:29:16 UTC. The handle is @contextwindows, nickname "Context Window", and the sound is credited as "original sound" by "Context Window". The description is 346 characters and ends in five hashtags. At archive capture (2026-11-05T13:01:15Z) the counts were 704 views, 15 likes, 3 comments, 1 repost and 14 saves. I read 4 of the extracted frames: 1 (about 0 s), 10 (about 18 s), 20 (about 38 s) and 30 (about 58 s). The transcript has 18 segments and about 190 words. Frame 1 shows a Meta logo, the caption "Meta interview question", and the headline "Trained on 4K. Needs 32K." above a 3D strip that enters an arch. Frame 10 shows four dial clocks labelled "fast" on the left and "slowest" on the right, a "32K" marker, a "4K" marker, and the caption "a full turn within 4,000 tokens." Frame 20 shows the title "What would you change so the model can use all 32K tokens?", the dials with red wedges, and option A in a dark card. Frame 30 shows all four options (A "Fine-tune on long documents and leave the positions as they are", B "Scale the positions down so 32K tokens fit inside the range seen in training", C "Replace rotary embeddings with ALiBi and fine-tune", D "Limit attention to a sliding window of 4K tokens") and the closing question "Which would you choose, and how would you test that it really uses the far end of the document?" No code, terminal, chart, benchmark or answer reveal appears in the frames I read. The spoken argument says rotary embeddings rotate query and key pairs by position-proportional angles, that the slowest pairs never complete a turn within 4,000 tokens, and that perplexity rises past the trained range. It then poses four options and asks which one to pick and how to test it. Segment 17 is timestamped 61.26 to 82.88 s, which is 21 s past the 61.65 s audio, so it is fabricated. Segments 15 and 16 (58.32 to 61.26 s) start after the last real sentence ends at 56.26 s, have a decoding temperature of 1.0 and an average log probability of -3.05, and contain garbled text. I treat all three as invented and quote none of them.**

## The transcript's closing lines are not a spoken answer

The reliable part of the transcript runs from 0.08 s to 56.26 s. At 52.06 to 56.26 s the video says "Which would you choose, and how would you test that it really uses the far end of the document?" That is the last coherent sentence. Whisper then emits "D. Limit attention 6,000 tokens." at 58.32 to 60.32 s. This contradicts the on-screen option D, which says a sliding window of 4K tokens, and the on-screen frame 30 near 58 s still shows the question card. The next segment is gibberish, and the final one claims to run 21 s beyond the audio. The segments carry temperature 1.0 and strongly negative log probabilities, which is how Whisper flags low-confidence decoding. The sensible reading is that the video ends on an open question and gives no answer. I found no evidence of a spoken verdict in any frame I read, but I read only 4 of roughly 31 frames, so a late on-screen reveal in the unread frames cannot be ruled out.

## Test of the premise: rotary embeddings and the "slowest pair" claim

At 10.44 to 15.98 s the video says "Rotary embeddings rotate each pair of query and key dimensions by an angle proportional to the token's position." This matches the RoFormer abstract, which says Rotary Position Embedding encodes absolute position with a rotation and captures relative distance inside self-attention. The abstract also lists flexibility with sequence length as a property. Source: [RoFormer, arXiv 2104.09864](https://arxiv.org/abs/2104.09864).

At 15.98 to 20.68 s the video says "Some pairs turn quickly, and the slowest never complete a full turn within 4,000 tokens." The page I fetched did not state frequency values. I checked the claim with my own arithmetic, assuming the common base of 10,000 and a head dimension of 128, neither of which the video specifies. The slowest pair then has a wavelength of roughly 2π times 10,000 raised to about 126/128, which is on the order of 58,000 tokens. That is longer than 4,096, so the claim holds under those assumptions. It also holds at 32K, where the slowest pair still would not finish a turn. The video's animation (frame 10, dials labelled "fast" and "slowest") shows the idea correctly but does not give those numbers. I mark the wavelength figure as my calculation, not a sourced result.

## Test of the premise: perplexity "shoots up" past the trained length

At 20.68 to 28.32 s the video says "perplexity looks normal for the first 4,000 tokens, then shoots up once positions go past anything the model saw in training." The Position Interpolation paper argues the same mechanism: extrapolating past trained positions can produce extreme attention scores that break self-attention. The paper's abstract page says that, instead of extrapolating, the method compresses input position indices to fit the original range. Source: [Extending Context Window of Large Language Models via Positional Interpolation, arXiv 2306.15595](https://arxiv.org/abs/2306.15595). The video gives no perplexity numbers, and I could not source any figure from it, so the specific "looks normal, then shoots up" curve is a description and not a measured result in this clip.

## Which option the sources support: B, with a caveat on "scale down"

Option B, "Scale the positions down so 32,000 tokens fit inside the range seen in training", describes Position Interpolation. The fetched abstract page says the method extends LLaMA-family models from 7B to 65B parameters to a 32,768-token window, with brief fine-tuning of under 1,000 steps, and that the architecture is unchanged. That fits the video's "about a thousand fine-tuning steps" budget at 28.32 to 31.98 s. The same page says performance within the original window stays fairly strong, and that the authors' theory gives an interpolation error bound at least about 600 times smaller than extrapolation's. I read only the abstract page, so I did not check the training recipe, learning rate or evaluation tables in the full paper.

The other options fail for reasons that follow from how they are phrased, which is my reasoning and not a fetched source. Option A leaves the positions unchanged, so the angles beyond 4K are still out-of-distribution and a thousand steps must teach the model to extrapolate. Option C swaps the position scheme, which discards what pretraining learned about rotary positions and needs far more than a thousand steps to recover. Option D limits each token to the previous 4K tokens, so by construction nothing at the far end of a 32K document can influence a token that is more than 4K away. D would keep perplexity flat but would not satisfy "use all 32,000 tokens." The video's own test question points at this: "how would you test that it really uses the far end of the document?" A plausible test is a passkey or needle retrieval task with the answer placed at several depths. The PI abstract page lists passkey retrieval among its evaluations, though I did not read the protocol.

## The "Meta interview question" label is unverified

Frame 1 shows a Meta logo and the caption "Meta interview question", and the title and description repeat "Meta interview question". I found no primary source (a Meta posting, interview guide or company statement) that this question was asked in a Meta interview. The fetched PI page also did not name the paper's affiliations in what I read. The label is an unverified framing, and the technical content stands independently of it.

## Key Takeaways

- **Verified:** The rotary embedding description at 10.44 to 15.98 s (pairs of query and key dimensions rotated by an angle that grows with position) matches the RoFormer abstract.
- **Verified:** Compressing positions to fit the trained range and fine-tuning for under 1,000 steps to reach 32,768 tokens is what the Position Interpolation abstract describes, so option B is the sourced match.
- **Correction:** The transcript lines "D. Limit attention 6,000 tokens." and "B. Giveisp guys a round like this.", plus the final "h." segment stamped to 82.88 s, are Whisper artifacts. The audio is 61.65 s, the on-screen option D says 4K tokens, and the last real sentence ends at 56.26 s.
- **Clarification:** The video gives no answer and no test result. Its closing question about testing the far end of the document is left open, and I read only 4 of the extracted frames.
- **Unverified:** The "Meta interview question" label has no primary source in what I fetched, and the slowest-pair wavelength of about 58,000 tokens is my own calculation under an assumed base of 10,000 and head dimension of 128.

## Resources

- [Extending Context Window of Large Language Models via Positional Interpolation, arXiv 2306.15595](https://arxiv.org/abs/2306.15595) establishes the position-compression method, the 32,768-token extension, the under-1,000-step fine-tuning claim and the 600-times error-bound claim I cite.
- [RoFormer: Enhanced Transformer with Rotary Position Embedding, arXiv 2104.09864](https://arxiv.org/abs/2104.09864) establishes the rotation-based description of RoPE I cite.
