Large language models (LLMs) have demonstrated impressive capabilities across a wide range of text and multimodal tasks. Yet, numerous applications, including document and video understanding, in-context learning, and inference-time scaling, require the processing and reasoning over long sequences of tokens. The restricted context window of LLMs presents a substantial challenge in these cases, as key information dispersed across lengthy documents may be missed. This constraint necessitates models that can effectively manage ultra-long contexts without compromising performance on routine tasks.
Current strategies for extending the context of long-context language models can be categorized into three types: exact attention methods, approximate attention methods, and approaches utilizing additional modules. Techniques like Position Interpolation, NTK-aware, Dynamic NTK, YaRN, and CLEX improve attention mechanisms with redesigned position embeddings. Recent developments include models like GPT-4o, Gemini, and Claude, capable of handling context windows encompassing hundreds of thousands of tokens. However, as they are closed-source, their reproducibility is limited. Open-source projects such as ProLong apply NTK-aware scaling but demand significant computational resources, whereas Gradient achieves prolonged context capabilities through continued pretraining without compromising standard task performance.
Researchers from UIUC and NVIDIA have developed an efficient training strategy for constructing ultra-long context LLMs from aligned instruct models, extending context lengths from 128K to 1M, 2M, and 4M tokens. The methodology employs efficient continued pretraining techniques to expand the context window while utilizing instruction tuning to maintain instruction-following and reasoning skills. Furthermore, their UltraLong-8B model sets the benchmark with outstanding performance on various long-context evaluations, showing balanced enhancements for both long and short context assignments. The research deeply examines pivotal design decisions, illustrating the effects of scaling strategies and data composition.
The proposed approach involves two primary stages: continued pretraining and instruction tuning. Together, these steps facilitate the efficient processing of ultra-long inputs while sustaining robust task performance. A YaRN-based scaling method is utilized for context extension with constant hyperparameters set as α = 1 and β = 4, differing from NTK-aware scaling techniques. Scaling factors are calculated based on the target context length and employ larger scaling factors for RoPE embeddings to support extended sequences and reduce performance loss at maximum lengths. Researchers select high-quality SFT datasets across general, mathematics, and coding domains for training the models and further refine responses using GPT-4o and GPT-4o-mini, ensuring meticulous data cleaning.
The proposed models exhibit superior long-context retrieval skills in the Needle in a Haystack passkey retrieval test. While baseline models such as Llama-3-8B-Instruct-Gradient-1048k pass the test, models like Llama3.1-8B-Instruct and Llama-3-8B-ProLong-512k-Instruct encounter errors. Conversely, the UltraLong models achieve 100% accuracy across all input lengths and depths, demonstrating strong retrieval competency. UltraLong scores the highest on RULER for inputs up to 512K and 1M tokens, achieves top F1 scores on LV-Eval within 128K and 256K token lengths, and delivers the best performance on InfiniteBench. Moreover, the models exhibit strong performance across general, math, and code domains with average scores exceeding the base model’s scores.
NVIDIA Impulsa la IA en EE.UU. con Producción Millonaria Durante Conflicto Arancelario
Discover Fomantis’ Shiny Form Potential in Pokémon Go
Related Posts
Students Create the Ultimate Speedy Robot for…
junio 1, 2025
Dad’s Day Delights: The Ultimate 2025 Gift Guide
junio 1, 2025
