RePo Repositions LLM Context, Outperforming Baselines on Long-Context Tasks
Sakana AI researchers introduced Context Re-Positioning (RePo), a technique allowing LLMs to dynamically re-organize their internal view of input data for long-context tasks. RePo uses a lightweight neural network to dynamically re-organize token positions by content, instead of fixed positional embeddings. This helps LLMs cluster related information, outperforming baselines on complex tasks like retrieval-augmented generation and question answering over extended documents. RePo outperformed standard models by over 11 points on the RULER benchmark and maintained high accuracy when extrapolating to 4x longer contexts (up to 16,000 tokens).
Read full story→