From the course: Introduction to Attention-Based Neural Networks
Join today to access over 24,700 courses taught by industry experts.
The need for long memory cells
From the course: Introduction to Attention-Based Neural Networks
The need for long memory cells
- RNN memory cells have a hidden state that allow the cell to remember the past. However, the memory capacity of such a cell is not very big which is why, in order to get the best results from our recurrent neural networks, we need to use long memory cells. Now, the regular recurrent cell suffers from what we call short-term memory. We've discussed in the earlier video that the memory of a recurrent neuron is because of the hidden state that it holds. It is this memory that allows recurrent cells to remember what happened in the past, what went by in the sequence, and that's what allows recurrent neurons to predict the next bit of information in the sequence. However, the memory capacity of a regular recurrent neurons is not very large. The simple hidden state is not enough to store all of the information from the past. Now, if you're working with very long input sequences your recurrent neurons may not process these well…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.
