Context length (context window)

The context length determines how much information a language model retains in its "short-term memory" while processing text. It is measured in tokens, which are sub-word units of text. A larger context length allows the model to access and utilize more information from the input sequence, enabling it to better understand the relationships between different parts of the text, resolve ambiguities, and generate more coherent and relevant outputs. Increasing context length is computationally expensive, requiring more memory and processing power during both training and inference. Some techniques, such as sparse attention mechanisms and memory compression methods, are actively being developed to extend context lengths efficiently. The size of the context window significantly impacts a model's performance on tasks requiring long-range dependencies, such as summarization, question answering over long documents, and code generation.