Self-Attention Mechanism

Self-attention allows each token in a sequence to attend to every other token, computing relevance scores that determine how much context each position contributes. This mechanism replaces the sequential processing of RNNs with parallel computation over all positions.

The key innovation is that attention weights are dynamic — they change based on the input, unlike fixed convolution filters. This enables the model to flexibly route information based on content rather than position.