What is attention in AI?
Also called: self-attention, attention mechanism
Attention is the mechanism in transformer models that lets each word look at every other word in the text and decide which ones matter most for understanding it. This is how a model keeps track of context, like what 'it' refers to in a sentence.
How does attention work?
For each token, the model scores how relevant every other token is, then builds a new representation that blends in the most relevant ones. Many attention 'heads' do this in parallel, each catching different relationships.
Why does attention matter?
Attention is what lets language models handle long, context-heavy text. It's also why longer context windows cost more to process.
An example
Example: In 'Riya gave Aman her notes because she was absent', attention helps a model work out who 'she' is from context.
Related terms
Learn it properly
Attention is covered in AI Engineer, with short lessons, practice questions and instant feedback.