Self-Attention<\/strong> is a mechanism that allows a model to look at other parts of the same input<\/strong> when understanding a specific part.<\/p>\n\n\n\n \ud83d\udc40 Basically:<\/p>\n\n\n\n Each word (or patch, in images) decides how much to pay attention<\/strong> to every other word in the same input \u2014 including itself!<\/p>\n<\/blockquote>\n\n\n\n \ud83d\udcd6 Think of it like this: Input sentence:<\/strong><\/p>\n\n\n\n \u201cThe cat sat on the mat.\u201d<\/p>\n<\/blockquote>\n\n\n\n Let\u2019s focus on “cat<\/strong>” \ud83d\udc31<\/p>\n\n\n\n With self-attention<\/strong>, the model asks:<\/p>\n\n\n\n It might give attention scores<\/strong> like:<\/p>\n\n\n\n So, “cat” mostly pays attention to itself and “sat” (because they\u2019re closely related). This helps the model understand relationships<\/strong> better \ud83e\udde9<\/p>\n\n\n\n Because the model is attending to itself<\/strong> \u2014 each word (or input token) looks at all other tokens in the same sequence<\/strong>, including itself \ud83d\udd01<\/p>\n\n\n\n It\u2019s like each word is thinking:<\/p>\n\n\n\n \u201cHey, what do the rest of us mean together?\u201d \ud83e\udde0\ud83d\udcad<\/p>\n<\/blockquote>\n\n\n\n \u2705 Transformers (BERT, GPT, etc.) \ud83e\udd16 Each word is turned into three vectors<\/strong>:<\/p>\n\n\n\n The model computes attention like this:<\/p>\n\n\n\n \ud83d\udcca This math helps decide how much focus<\/strong> (weight) to give each word in the sentence.<\/p>\n\n\n\n Self-Attention<\/strong> lets each word or token in a sequence pay attention<\/strong> to all others \u2014 to understand context, relationships, and meaning better. \ud83e\udde0\u2728<\/p>\n<\/blockquote>\n\n\n\n <\/p>\n","protected":false},"excerpt":{"rendered":"\n
When understanding the meaning of the word “bank<\/strong>“, the model looks at surrounding words<\/strong> to decide:<\/p>\n\n\n\n\n
\n\n\n\n\ud83e\udde0 Simple Example:<\/h2>\n\n\n\n
\n
\n
Word<\/th> Attention Score (to “cat”)<\/th><\/tr><\/thead> The<\/td> 0.1<\/td><\/tr> cat<\/td> 0.4 \u2705 (itself)<\/td><\/tr> sat<\/td> 0.3 \ud83e\ude91<\/td><\/tr> on<\/td> 0.05<\/td><\/tr> the<\/td> 0.05<\/td><\/tr> mat<\/td> 0.1 \ud83e\uddfa<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n\ud83d\udd0d Why Is It Called Self<\/em>-Attention?<\/h2>\n\n\n\n
\n
\n\n\n\n\ud83e\uddea Where Is Self-Attention Used?<\/h2>\n\n\n\n
\u2705 Vision Transformers (ViT) \ud83d\uddbc\ufe0f
\u2705 Text translation \ud83c\udf0d
\u2705 Chatbots & summarization \u270d\ufe0f<\/p>\n\n\n\n
\n\n\n\n\u2699\ufe0f How It Works (Quick Look)<\/h2>\n\n\n\n
\n
Attention(Q, K, V) = softmax(Q \u00d7 K\u1d40 \/ \u221ad) \u00d7 V\n<\/code><\/pre>\n\n\n\n
\n\n\n\n\ud83d\udff0 Summary Table<\/h2>\n\n\n\n
Feature<\/th> Self-Attention \ud83d\udd01<\/th><\/tr><\/thead> \ud83e\udde0 Focuses on<\/td> All other tokens in the same input<\/td><\/tr> \ud83d\udc41\ufe0f Learns<\/td> Word relationships & context<\/td><\/tr> \ud83d\udccd Used in<\/td> Transformers (text & vision)<\/td><\/tr> \ud83d\udca1 Helps with<\/td> Meaning, context, dependencies<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n\u2705 TL;DR:<\/h3>\n\n\n\n
\n
\n\n\n\n