Click here to download project base paper of train shorts.
Abstract:
Since the introduction of the transformer model by Vaswani et al. (2017), a fundamental question has yet to be answered: how does a model achieve extrapolation at inference time for sequences that are longer than it saw during training? We first show that extrapolation can be enabled by simply changing the position representation method, though we find that current methods do not allow for efficient extrapolation. We therefore introduce a simpler and more efficient position method, Attention with Linear Biases (ALiBi). ALiBi does not add positional embeddings to word embeddings; instead, it biases query-key attention scores with a penalty that is proportional to their distance. We show that this method train shorts a 1.3 billion parameter model on input sequences of length 1024 that extrapolates to input sequences of length 2048
achieving the same perplexity as a sinusoidal position embedding model trained on inputs of length 2048 but training 11% faster and using 11% less memory. ALiBi’s inductive bias towards recency also leads it to outperform multiple strong position methods on the WikiText-103 benchmark., Advanced Deep Learning.
INTRODUCTION: When constructing a transformer-based language model, a major design decision is the length of training sequences, denoted L here, which has to date been equivalent to the length of inference sequences. More context, achieved by a larger L, improves predictions at inference time. But longer sequences are more expensive to train on.2 Beforetransformers, RNN language models were trained on shorter-L sequences and assumed to generalize to longer contexts inference time (Mikolovet al., 2010; Mikolov & Zweig, 2012; Zarembaetal.,2014). Vaswanietal. (2017), introducing the transformer, speculated that it“may […] extra plate to sequence lengths longer than the one sen counter enduring training.”We define extrapolation as a model’s ability to continue performing well as the number of input tokens during validation increases beyond the number of tokens on which the model was trained.
