<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Long Context | Web IR NLP Group @ NUS</title><link>https://wing.comp.nus.edu.sg/tag/long-context/</link><atom:link href="https://wing.comp.nus.edu.sg/tag/long-context/index.xml" rel="self" type="application/rss+xml"/><description>Long Context</description><generator>Hugo Blox Builder (https://hugoblox.com)</generator><language>en-us</language><lastBuildDate>Mon, 01 Dec 2025 00:00:00 +0000</lastBuildDate><image><url>https://wing.comp.nus.edu.sg/media/icon_hu_ce731d16d7c8719.png</url><title>Long Context</title><link>https://wing.comp.nus.edu.sg/tag/long-context/</link></image><item><title>Context Window Scheduling for Efficient LLM Pretraining</title><link>https://wing.comp.nus.edu.sg/project/context-window-scheduling/</link><pubDate>Mon, 01 Dec 2025 00:00:00 +0000</pubDate><guid>https://wing.comp.nus.edu.sg/project/context-window-scheduling/</guid><description>&lt;p&gt;Recent advances in LLM pretraining have featured ever-expanding context windows to process longer sequences. Long context windows are costly to train with, however, and it is not obvious that every token should be trained at the full window length. This project, carried out with collaborators at Sea AI Lab, studies how the context window itself should be scheduled during pretraining.&lt;/p&gt;
&lt;p&gt;In &lt;a href="https://wing.comp.nus.edu.sg/publication/skyladder-2025/"&gt;SkyLadder: Better and Faster Pretraining via Context Window Scheduling&lt;/a&gt; (NeurIPS 2025), a pilot study shows that under a fixed token budget, models pretrained with shorter context windows consistently outperform their long-context counterparts. Motivated by this, &lt;strong&gt;SkyLadder&lt;/strong&gt; implements a simple short-to-long context window transition during pretraining. Pretraining 1B-parameter models (up to 32K context) and 3B-parameter models (8K context) on 100B tokens, SkyLadder yields consistent gains of up to 3.7% on common benchmarks and matches or exceeds baseline results on long-context tasks, while training up to 22% faster than the baselines. The &lt;a href="https://github.com/sail-sg/SkyLadder" target="_blank" rel="noopener"&gt;SkyLadder repository&lt;/a&gt; provides the pretraining code.&lt;/p&gt;</description></item></channel></rss>