Rotatory-Positional-Embeddings Original Paper:- https://arxiv.org/abs/2104.09864 https://towardsdatascience.com/understanding-positional-embeddings-in-transformers-from-absolute-to-rotary-31c082e16b26/ https://aiexpjourney.substack.com/p/an-in-depth-exploration-of-rotary-position-embedding-rope-ac351a45c794 https://medium.com/@DataDry/decoding-rotary-positional-embeddings-rope-the-secret-sauce-for-smarter-transformers-193cbc01e4ed When we were discussing about Positional Embeddings mentioned in foundational paper “Attention is All You Need” , we got to know the importance of positional encodings , also we got to know the two different approaches Learned & Fixed though the paper preferred fixed positional embeddings over learned embeddings, the later results and metrics showed that fixed positional embeddings could not accurately process the relation ship between different words or tokens , especially when the sequence whose length is more than sequence encoutered during training. ...