Recurrent Neural Networks(RNN)

ke, 23 July 2018

[ deep_learning  ]
  • Recurrent Neural Networks

what are deep convnets learning?



  • RNN types

  • Language model

you are using speech recognition and get the following two sentence:
The apple and pair salad.
The apple and pear salad.

whick one is the prefer one?

using language model to get the prossibility of these sentence, choose the better one:
p(The apple and pair salad) = 3.2 * 10^-13 p(The apple and pear salad) = 5.7 * 10^-10

The apple and pear salad is the better one sentence


P(y<1>, y<2>, y<3>) = p(y<1>) * p(y<2>|y<1>) * p(y<3>|y<1>,y<2>)

  • Sampling novel sequences

Sampling a sequence from a trained RNN

  • Vanishing gradients with RNNs

exploding gradient can use gradient clipping to slove the problem—-look at your vectors, and if it is bigger than some threshold, rescale some of your gradient vector so that is not too big.

vanishing gradients see the follow

  • Gated Recurrent Unit(GRU)



  • LSTM(long short term memory) unit


  • Bidirectional RNN

  • Deep RNNs