From the course: Introduction to Attention-Based Neural Networks

Unlock this course with a free trial

Join today to access over 24,700 courses taught by industry experts.

Implementing Bahdanau attention

Implementing Bahdanau attention

From the course: Introduction to Attention-Based Neural Networks

Start my 1-month free trial Buy for my team

Implementing Bahdanau attention

“

- Next up, we define the attention module that allows our decoder RNN to pay attention to important parts of the input image. We are using Bahdanau's Attention, that is additive attention, and you'll see that the operations that we perform on the features are the operations that we discussed for Bahdanau's Attention. Initialize the attention module by specifying the size of the encoder hidden state, the size of the decoder hidden state, and the size of the attention. These are the parameters of your attention module, initialize the member variable attention size to the attention size. In Bahdanau's Attention, we pass the encoder and decoder hidden states through a weights layer, and this is where we initialize the weights layer. full_A here is the trainable weight vector that generates the alignment scores. Here is what a forward pass through the attention model looks like. As an input, we accept the feature map…

Contents