← All explorers
Loading explorer…
Read the theory: transformers
Read the theory: transformers
Hold content vectors fixed and move their positions. The RoPE dot product depends on their gap: shifting both positions equally preserves this demo’s score.
RoPE rotates query/key coordinate pairs at several frequencies. For these fixed content vectors, the dot product depends on the relative offset m−n, so shifting both positions equally preserves it. In a trained model the content vectors also depend on context; this identity alone does not guarantee whole-model shift invariance or long-context quality.
Illustrative model for building intuition, not a live system.