##ymptotically linear in the number of time steps used by the turing machine and asymptotically linear in the length of the input ( siegelmann and sontag 1991 siegelmann 1995 siegelmann and sontag 1995, ;, ;, ; hyotyniemi 1996, ). the functions computable by a turing machine are discrete, so these results regard exact implementation of the function, not approximations. the rnn, when used as a turing machine, takes a binary sequence as input and its outputs must be discretized to provide a binary output. it is possible to compute all functions in this setting using a single speciﬁc rnn of ﬁnite size ( siegelmann and sontag 1995 ( ) use 886 units ). the “ input ” of the turing machine is a speciﬁcation of the function to be computed, so the same network that simulates this turing machine is [UNK] for all problems. the theoretical rnn used for the proof can simulate an unbounded stack by representing its activations and weights with rational numbers of unbounded precision. we now develop the forward propagation equations for the rnn depicted in ﬁgure. the ﬁgure does not specify the choice of activation function for the 10


##ed stack by representing its activations and weights with rational numbers of unbounded precision. we now develop the forward propagation equations for the rnn depicted in ﬁgure. the ﬁgure does not specify the choice of activation function for the 10. 3 hidden units. here we assume the hyperbolic tangent activation function. also, the ﬁgure does not specify exactly what form the output and loss function take. here we assume that the output is discrete, as if the rnn is used to predict words or characters. a natural way to represent discrete variables is to regard the output o as giving the unnormalized log probabilities of each possible value of the discrete variable. we can then apply the softmax operation as a post - processing step to obtain a vector [UNK] of normalized probabilities over the output. forward propagation begins with a speciﬁcation of the initial state h ( 0 ). then, for each time step from 379

/home/ricoiban/GEMMA/mnlp_chatsplaining/RAG/Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville (z-lib.org).pdf


chapter 10. sequence modeling : recurrent and recursive nets u v w o ( t−1 ) o ( t−1 ) h o y l x o ( ) t o ( ) t o ( + 1 ) t o ( + 1 ) t l ( t−1 ) l ( t−1 ) l ( ) t l ( ) t l ( + 1 ) t l ( + 1 ) t y ( t−1 ) y ( t−1 ) y ( ) t y ( ) t y ( + 1 ) t y ( + 1 ) t h ( t−1 ) h ( t−1 ) h ( ) t h ( ) t h ( + 1 ) t h ( + 1 ) t x ( t−1 ) x ( t−1 ) x ( ) t x ( ) t x ( + 1 ) t x ( + 1 ) t w w w w o ( )... o ( )... h ( )... h ( )... v v v u u u unfold figure 10. 4 : an rnn whose only recurrence is the feedback connection from the output to the hidden layer. at each time step t, the input is xt, the hidden layer activations are h ( ) t, the


##fold figure 10. 4 : an rnn whose only recurrence is the feedback connection from the output to the hidden layer. at each time step t, the input is xt, the hidden layer activations are h ( ) t, the outputs are o ( ) t, the targets are y ( ) t and the loss is l ( ) t. ( left ) circuit diagram. ( right ) unfolded computational graph. such an rnn is less powerful ( can express a smaller set of functions ) than those in the family represented by ﬁgure. the rnn 10. 3 in ﬁgure can choose to put any information it wants about the past into its hidden 10. 3 representation h and transmit h to the future. the rnn in this ﬁgure is trained to put a speciﬁc output value into o, and o is the only information it is allowed to send to the future. there are no direct connections from h going forward. the previous h is connected to the present only indirectly, via the predictions it was used to produce. unless o is very high - dimensional and rich, it will usually lack important information from the past. this makes the rnn in this ﬁgure less powerful, but it may be easier to train


, via the predictions it was used to produce. unless o is very high - dimensional and rich, it will usually lack important information from the past. this makes the rnn in this ﬁgure less powerful, but it may be easier to train because each time step can be trained in isolation from the others, allowing greater parallelization during training, as described in section. 10. 2. 1 380


chapter 10. sequence modeling : recurrent and recursive nets t t τ = 1 to =, we apply the following update equations : a ( ) t = + b w h ( 1 ) t− + ux ( ) t ( 10. 8 ) h ( ) t = tanh ( a ( ) t ) ( 10. 9 ) o ( ) t = + c v h ( ) t ( 10. 10 ) [UNK] ( ) t = softmax ( o ( ) t ) ( 10. 11 ) where the parameters are the bias vectors b and c along with the weight matrices u, v and w, respectively for input - to - hidden, hidden - to - output and hidden - to - hidden connections. this is an example of a recurrent network that maps an input sequence to an output sequence of the same length. the total loss for a given sequence of values paired with a sequence of values would then be just x y the sum of the losses over all the time steps. for example, if l ( ) t is the negative log - likelihood of y ( ) t given x ( 1 ),..., x ( ) t, then l { x ( 1 ),..., x ( ) τ } {,


if l ( ) t is the negative log - likelihood of y ( ) t given x ( 1 ),..., x ( ) t, then l { x ( 1 ),..., x ( ) τ } {, y ( 1 ),..., y ( ) τ } ( 10. 12 ) = t l ( ) t ( 10. 13 ) = − t log pmodel y ( ) t | { x ( 1 ),..., x ( ) t }, ( 10. 14 ) where pmodel y ( ) t | { x ( 1 ),..., x ( ) t } is given by reading the entry for y ( ) t from the model ’ s output vector [UNK] ( ) t. computing the gradient of this loss function with respect to the parameters is an expensive operation. the gradient computation involves performing a forward propagation pass moving left to right through our illustration of the unrolled graph in ﬁgure, followed by a backward propagation pass 10. 3 moving right to left through the graph. the runtime is o ( τ ) and cannot be reduced by parallelization because the forward propagation graph is inherently sequential ; each time step may only be computed after the previous one


a backward propagation pass 10. 3 moving right to left through the graph. the runtime is o ( τ ) and cannot be reduced by parallelization because the forward propagation graph is inherently sequential ; each time step may only be computed after the previous one. states computed in the forward pass must be stored until they are reused during the backward pass, so the memory cost is also o ( τ ). the back - propagation algorithm applied to the unrolled graph with o ( τ ) cost is called back - propagation through time or bptt and is discussed further in section. the network with recurrence 10. 2. 2 between hidden units is thus very powerful but also expensive to train. is there an alternative? 10. 2. 1 teacher forcing and networks with output recurrence the network with recurrent connections only from the output at one time step to the hidden units at the next time step ( shown in ﬁgure ) is strictly less powerful 10. 4 381


chapter 10. sequence modeling : recurrent and recursive nets because it lacks hidden - to - hidden recurrent connections. for example, it cannot simulate a universal turing machine. because this network lacks hidden - to - hidden recurrence, it requires that the output units capture all of the information about the past that the network will use to predict the future. because the output units are explicitly trained to match the training set targets, they are unlikely to capture the necessary information about the past history of the input, unless the user knows how to describe the full state of the system and provides it as part of the training set targets. the advantage of eliminating hidden - to - hidden recurrence is that, for any loss function based on comparing the prediction at time t to the training target at time t, all the time steps are decoupled. training can thus be parallelized, with the gradient for each step t computed in isolation. there is no need to compute the output for the previous time step ﬁrst, because the training set provides the ideal value of that output. h ( t−1 ) h ( t−1 ) w h ( ) t h ( ) t...... x ( t−1 ) x ( t−1 ) x ( ) t


the training set provides the ideal value of that output. h ( t−1 ) h ( t−1 ) w h ( ) t h ( ) t...... x ( t−1 ) x ( t−1 ) x ( ) t x ( ) t x ( )... x ( )... w w u u u h ( ) τ h ( ) τ x ( ) τ x ( ) τ w u o ( ) τ o ( ) τ y ( ) τ y ( ) τ l ( ) τ l ( ) τ v...... figure 10. 5 : time - unfolded recurrent neural network with a single output at the end of the sequence. such a network can be used to summarize a sequence and produce a ﬁxed - size representation used as input for further processing. there might be a target right at the end ( as depicted here ) or the gradient on the output o ( ) t can be obtained by back - propagating from further downstream modules. models that have recurrent connections from their outputs leading back into the model may be trained with teacher forcing. teacher forcing is a procedure that emerges from the maximum likelihood criterion, in which during training the model receives the ground truth output y ( )


modules. models that have recurrent connections from their outputs leading back into the model may be trained with teacher forcing. teacher forcing is a procedure that emerges from the maximum likelihood criterion, in which during training the model receives the ground truth output y ( ) t as input at time t + 1. we can see this by examining a sequence with two time steps. the conditional maximum 382


chapter 10. sequence modeling : recurrent and recursive nets o ( t−1 ) o ( t−1 ) o ( ) t o ( ) t h ( t−1 ) h ( t−1 ) h ( ) t h ( ) t x ( t−1 ) x ( t−1 ) x ( ) t x ( ) t w v v u u o ( t−1 ) o ( t−1 ) o ( ) t o ( ) t l ( t−1 ) l ( t−1 ) l ( ) t l ( ) t y ( t−1 ) y ( t−1 ) y ( ) t y ( ) t h ( t−1 ) h ( t−1 ) h ( ) t h ( ) t x ( t−1 ) x ( t−1 ) x ( ) t x ( ) t w v v u u train time test time figure 10. 6 : illustration of teacher forcing. teacher forcing is a training technique that is applicable to rnns that have connections from their output to their hidden states at the next time step. ( left ) at train time, we feed the correct outputy ( ) t drawn from the train set as input to h ( + 1 ) t. when the model is deployed, the true output is generally ( right ) not known


time step. ( left ) at train time, we feed the correct outputy ( ) t drawn from the train set as input to h ( + 1 ) t. when the model is deployed, the true output is generally ( right ) not known. in this case, we approximate the correct output y ( ) t with the model ’ s output o ( ) t, and feed the output back into the model. 383


chapter 10. sequence modeling : recurrent and recursive nets likelihood criterion is log p y ( 1 ), y ( 2 ) | x ( 1 ), x ( 2 ) ( 10. 15 ) = log p y ( 2 ) | y ( 1 ), x ( 1 ), x ( 2 ) + log p y ( 1 ) | x ( 1 ), x ( 2 ) ( 10. 16 ) in this example, we see that at time t = 2, the model is trained to maximize the conditional probability of y ( 2 ) given both the x sequence so far and the previous y value from the training set. maximum likelihood thus speciﬁes that during training, rather than feeding the model ’ s own output back into itself, these connections should be fed with the target values specifying what the correct output should be. this is illustrated in ﬁgure. 10. 6 we originally motivated teacher forcing as allowing us to avoid back - propagation through time in models that lack hidden - to - hidden connections. teacher forcing may still be applied to models that have hidden - to - hidden connections so long as they have connections from the output at one time step to values computed in the next time step. however, as soon as the hidden units become


hidden connections. teacher forcing may still be applied to models that have hidden - to - hidden connections so long as they have connections from the output at one time step to values computed in the next time step. however, as soon as the hidden units become a function of earlier time steps, the bptt algorithm is necessary. some models may thus be trained with both teacher forcing and bptt. the disadvantage of strict teacher forcing arises if the network is going to be later used in an open - loop mode, with the network outputs ( or samples from the output distribution ) fed back as input. in this case, the kind of inputs that the network sees during training could be quite [UNK] from the kind of inputs that it will see at test time. one way to mitigate this problem is to train with both teacher - forced inputs and with free - running inputs, for example by predicting the correct target a number of steps in the future through the unfolded recurrent output - to - input paths. in this way, the network can learn to take into account input conditions ( such as those it generates itself in the free - running mode ) not seen during training and how to map the state back towards one that will make the network generate proper outputs after a few steps. another approach ( ben


to take into account input conditions ( such as those it generates itself in the free - running mode ) not seen during training and how to map the state back towards one that will make the network generate proper outputs after a few steps. another approach ( bengio et al., ) to mitigate the gap between the inputs seen at train time and the 2015b inputs seen at test time randomly chooses to use generated values or actual data values as input. this approach exploits a curriculum learning strategy to gradually use more of the generated values as input. 10. 2. 2 computing the gradient in a recurrent neural network computing the gradient through a recurrent neural network is straightforward. one simply applies the generalized back - propagation algorithm of section 6. 5. 6 384


chapter 10. sequence modeling : recurrent and recursive nets to the unrolled computational graph. no specialized algorithms are necessary. gradients obtained by back - propagation may then be used with any general - purpose gradient - based techniques to train an rnn. to gain some intuition for how the bptt algorithm behaves, we provide an example of how to compute gradients by bptt for the rnn equations above ( equation and equation ). the nodes of our computational graph include 10. 8 10. 12 the parameters u, v, w, b and c as well as the sequence of nodes indexed by t for x ( ) t, h ( ) t, o ( ) t and l ( ) t. for each node n we need to compute the gradient ∇nl recursively, based on the gradient computed at nodes that follow it in the graph. we start the recursion with the nodes immediately preceding the ﬁnal loss ∂l ∂l ( ) t = 1. ( 10. 17 ) in this derivation we assume that the outputs o ( ) t are used as the argument to the softmax function to obtain the vector [UNK] of probabilities over the output. we also assume that the loss is the negative log - likelihood of the true


in this derivation we assume that the outputs o ( ) t are used as the argument to the softmax function to obtain the vector [UNK] of probabilities over the output. we also assume that the loss is the negative log - likelihood of the true target y ( ) t given the input so far. the gradient ∇o ( ) t l on the outputs at time step t, for all i, t, is as follows : ( ∇o ( ) t l ) i = ∂l ∂o ( ) t i = ∂l ∂l ( ) t ∂l ( ) t ∂o ( ) t i = [UNK] ( ) t i −1i, y ( ) t. ( 10. 18 ) we work our way backwards, starting from the end of the sequence. at the ﬁnal time step, τ h ( ) τ only has o ( ) τ as a descendent, so its gradient is simple : ∇h ( ) τ l = v ∇o ( ) τ l. ( 10. 19 ) we can then iterate backwards in time to back - propagate gradients through time, from t = τ −1 down to t = 1, noting that h ( ) t ( for t < τ ) has as descendents both o


) we can then iterate backwards in time to back - propagate gradients through time, from t = τ −1 down to t = 1, noting that h ( ) t ( for t < τ ) has as descendents both o ( ) t and h ( + 1 ) t. its gradient is thus given by ∇h ( ) t l = ∂h ( + 1 ) t ∂h ( ) t ( ∇h ( + 1 ) t l ) + ∂o ( ) t ∂h ( ) t ( ∇o ( ) t l ) ( 10. 20 ) = w ( ∇h ( + 1 ) t l ) diag 1 − h ( + 1 ) t 2 + v ( ∇o ( ) t l ) ( 10. 21 ) where diag 1 − h ( + 1 ) t 2 indicates the diagonal matrix containing the elements 1 − ( h ( + 1 ) t i ) 2. this is the jacobian of the hyperbolic tangent associated with the hidden unit at time. i t + 1 385


chapter 10. sequence modeling : recurrent and recursive nets once the gradients on the internal nodes of the computational graph are obtained, we can obtain the gradients on the parameter nodes. because the parameters are shared across many time steps, we must take some care when denoting calculus operations involving these variables. the equations we wish to implement use the bprop method of section, that computes the contribution 6. 5. 6 of a single edge in the computational graph to the gradient. however, the ∇wf operator used in calculus takes into account the contribution of w to the value of f due to edges in the computational graph. to resolve this ambiguity, we all introduce dummy variables w ( ) t that are deﬁned to be copies of w but with each w ( ) t used only at time step t. we may then use ∇w ( ) t to denote the contribution of the weights at time step to the gradient. t using this notation, the gradient on the remaining parameters is given by : ∇cl = t ∂o ( ) t ∂c ∇o ( ) t l = t ∇o ( ) t l ( 10. 22 ) ∇bl = t ∂h ( ) t ∂b ( ) t ∇h ( ) t l =


##cl = t ∂o ( ) t ∂c ∇o ( ) t l = t ∇o ( ) t l ( 10. 22 ) ∇bl = t ∂h ( ) t ∂b ( ) t ∇h ( ) t l = t diag 1 − h ( ) t 2 ∇h ( ) t l ( 10. 23 ) ∇v l = t i ∂l ∂o ( ) t i ∇v o ( ) t i = t ( ∇o ( ) t l ) h ( ) t ( 10. 24 ) ∇wl = t i ∂l ∂h ( ) t i ∇w ( ) t h ( ) t i ( 10. 25 ) = t diag 1 − h ( ) t 2 ( ∇h ( ) t l ) h ( 1 ) t− ( 10. 26 ) ∇ul = t i ∂l ∂h ( ) t i ∇u ( ) t h ( ) t i ( 10. 27 ) = t diag 1 − h ( ) t 2 ( ∇h ( ) t l ) x ( ) t ( 10. 28 ) we do not need to compute the gradient with respect to x ( ) t for training because it does not have any parameters as ancestors in the computational graph de


t 2 ( ∇h ( ) t l ) x ( ) t ( 10. 28 ) we do not need to compute the gradient with respect to x ( ) t for training because it does not have any parameters as ancestors in the computational graph deﬁning the loss. 386


chapter 10. sequence modeling : recurrent and recursive nets 10. 2. 3 recurrent networks as directed graphical models in the example recurrent network we have developed so far, the losses l ( ) t were cross - entropies between training targetsy ( ) t and outputs o ( ) t. as with a feedforward network, it is in principle possible to use almost any loss with a recurrent network. the loss should be chosen based on the task. as with a feedforward network, we usually wish to interpret the output of the rnn as a probability distribution, and we usually use the cross - entropy associated with that distribution to deﬁne the loss. mean squared error is the cross - entropy loss associated with an output distribution that is a unit gaussian, for example, just as with a feedforward network. when we use a predictive log - likelihood training objective, such as equa - tion, we train the rnn to estimate the conditional distribution of the next 10. 12 sequence element y ( ) t given the past inputs. this may mean that we maximize the log - likelihood log ( p y ( ) t | x ( 1 ),..., x ( ) t ), ( 10


next 10. 12 sequence element y ( ) t given the past inputs. this may mean that we maximize the log - likelihood log ( p y ( ) t | x ( 1 ),..., x ( ) t ), ( 10. 29 ) or, if the model includes connections from the output at one time step to the next time step, log ( p y ( ) t | x ( 1 ),..., x ( ) t, y ( 1 ),..., y ( 1 ) t− ). ( 10. 30 ) decomposing the joint probability over the sequence of y values as a series of one - step probabilistic predictions is one way to capture the full joint distribution across the whole sequence. when we do not feed past y values as inputs that condition the next step prediction, the directed graphical model contains no edges from any y ( ) i in the past to the current y ( ) t. in this case, the outputs y are conditionally independent given the sequence of x values. when we do feed the actual y values ( not their prediction, but the actual observed or generated values ) back into the network, the directed graphical model contains edges from all y ( ) i values in the past


independent given the sequence of x values. when we do feed the actual y values ( not their prediction, but the actual observed or generated values ) back into the network, the directed graphical model contains edges from all y ( ) i values in the past to the current y ( ) t value. as a simple example, let us consider the case where the rnn models only a sequence of scalar random variables y = { y ( 1 ),..., y ( ) τ }, with no additional inputs x. the input at time step t is simply the output at time step t −1. the rnn then deﬁnes a directed graphical model over the y variables. we parametrize the joint distribution of these observations using the chain rule ( equation ) for conditional 3. 6 probabilities : p p ( ) = y ( y ( 1 ),..., y ( ) τ ) = τ t = 1 p ( y ( ) t | y ( 1 ) t−, y ( 2 ) t−,..., y ( 1 ) ) ( 10. 31 ) where the right - hand side of the bar is empty for t = 1, of course. hence the negative log - likelihood of a set of values


( 2 ) t−,..., y ( 1 ) ) ( 10. 31 ) where the right - hand side of the bar is empty for t = 1, of course. hence the negative log - likelihood of a set of values { y ( 1 ),..., y ( ) τ } according to such a model 387


chapter 10. sequence modeling : recurrent and recursive nets y ( 1 ) y ( 1 ) y ( 2 ) y ( 2 ) y ( 3 ) y ( 3 ) y ( 4 ) y ( 4 ) y ( 5 ) y ( 5 ) y ( )... y ( )... figure 10. 7 : fully connected graphical model for a sequencey ( 1 ), y ( 2 ),..., y ( ) t,... : every past observation y ( ) i may inﬂuence the conditional distribution of some y ( ) t ( for t > i ), given the previous values. parametrizing the graphical model directly according to this graph ( as in equation ) might be very [UNK], with an ever growing number of 10. 6 inputs and parameters for each element of the sequence. rnns obtain the same full connectivity but [UNK] parametrization, as illustrated in ﬁgure. 10. 8 is l = t l ( ) t ( 10. 32 ) where l ( ) t = log ( − p y ( ) t = y ( ) t | y ( 1 ) t−, y ( 2 ) t−,..., y ( 1 ) ). ( 10


10. 32 ) where l ( ) t = log ( − p y ( ) t = y ( ) t | y ( 1 ) t−, y ( 2 ) t−,..., y ( 1 ) ). ( 10. 33 ) y ( 1 ) y ( 1 ) y ( 2 ) y ( 2 ) y ( 3 ) y ( 3 ) y ( 4 ) y ( 4 ) y ( 5 ) y ( 5 ) y ( )... y ( )... h ( 1 ) h ( 1 ) h ( 2 ) h ( 2 ) h ( 3 ) h ( 3 ) h ( 4 ) h ( 4 ) h ( 5 ) h ( 5 ) h ( )... h ( )... figure 10. 8 : introducing the state variable in the graphical model of the rnn, even though it is a deterministic function of its inputs, helps to see how we can obtain a very [UNK] parametrization, based on equation. every stage in the sequence ( for 10. 5 h ( ) t and y ( ) t ) involves the same structure ( the same number of inputs for each node ) and can share the same parameters with the other stages. the edges in a graphical


every stage in the sequence ( for 10. 5 h ( ) t and y ( ) t ) involves the same structure ( the same number of inputs for each node ) and can share the same parameters with the other stages. the edges in a graphical model indicate which variables depend directly on other variables. many graphical models aim to achieve statistical and computational [UNK] by omitting edges that do not correspond to strong interactions. for 388


chapter 10. sequence modeling : recurrent and recursive nets example, it is common to make the markov assumption that the graphical model should only contain edges from { y ( ) t k −,..., y ( 1 ) t− } to y ( ) t, rather than containing edges from the entire past history. however, in some cases, we believe that all past inputs should have an inﬂuence on the next element of the sequence. rnns are useful when we believe that the distribution over y ( ) t may depend on a value of y ( ) i from the distant past in a way that is not captured by the [UNK] of y ( ) i on y ( 1 ) t−. one way to interpret an rnn as a graphical model is to view the rnn as deﬁning a graphical model whose structure is the complete graph, able to represent direct dependencies between any pair of y values. the graphical model over the y values with the complete graph structure is shown in ﬁgure. the complete 10. 7 graph interpretation of the rnn is based on ignoring the hidden units h ( ) t by marginalizing them out of the model. it is more interesting to consider the graphical model structure of rnns that results


ﬁgure. the complete 10. 7 graph interpretation of the rnn is based on ignoring the hidden units h ( ) t by marginalizing them out of the model. it is more interesting to consider the graphical model structure of rnns that results from regarding the hidden units h ( ) t as random variables. 1 including the hidden units in the graphical model reveals that the rnn provides a very [UNK] parametrization of the joint distribution over the observations. suppose that we represented an arbitrary joint distribution over discrete values with a tabular representation — an array containing a separate entry for each possible assignment of values, with the value of that entry giving the probability of that assignment occurring. if y can take on k [UNK] values, the tabular representation would have o ( kτ ) parameters. by comparison, due to parameter sharing, the number of parameters in the rnn is o ( 1 ) as a function of sequence length. the number of parameters in the rnn may be adjusted to control model capacity but is not forced to scale with sequence length. equation shows that the rnn parametrizes 10. 5 long - term relationships between variables [UNK], using recurrent applications of the same function f and same parameters θ at each time step. figure 10. 8 illustrates the


scale with sequence length. equation shows that the rnn parametrizes 10. 5 long - term relationships between variables [UNK], using recurrent applications of the same function f and same parameters θ at each time step. figure 10. 8 illustrates the graphical model interpretation. incorporating the h ( ) t nodes in the graphical model decouples the past and the future, acting as an intermediate quantity between them. a variable y ( ) i in the distant past may inﬂuence a variable y ( ) t via its [UNK] on h. the structure of this graph shows that the model can be [UNK] parametrized by using the same conditional probability distributions at each time step, and that when the variables are all observed, the probability of the joint assignment of all variables can be evaluated [UNK]. even with the [UNK] parametrization of the graphical model, some operations remain computationally challenging. for example, it is [UNK] to predict missing 1the conditional distribution over these variables given their parents is deterministic. this is perfectly legitimate, though it is somewhat rare to design a graphical model with such deterministic hidden units. 389


chapter 10. sequence modeling : recurrent and recursive nets values in the middle of the sequence. the price recurrent networks pay for their reduced number of parameters is that the parameters may be [UNK]. optimizing the parameter sharing used in recurrent networks relies on the assumption that the same parameters can be used for [UNK] time steps. equivalently, the assumption is that the conditional probability distribution over the variables at time t + 1 given the variables at time t is stationary, meaning that the relationship between the previous time step and the next time step does not depend on t. in principle, it would be possible to use t as an extra input at each time step and let the learner discover any time - dependence while sharing as much as it can between [UNK] time steps. this would already be much better than using a [UNK] conditional probability distribution for each t, but the network would then have to extrapolate when faced with new values of. t to complete our view of an rnn as a graphical model, we must describe how to draw samples from the model. the main operation that we need to perform is simply to sample from the conditional distribution at each time step. however, there is one additional complication. the rnn must have some mechanism for determining the length


describe how to draw samples from the model. the main operation that we need to perform is simply to sample from the conditional distribution at each time step. however, there is one additional complication. the rnn must have some mechanism for determining the length of the sequence. this can be achieved in various ways. in the case when the output is a symbol taken from a vocabulary, one can add a special symbol corresponding to the end of a sequence ( schmidhuber 2012, ). when that symbol is generated, the sampling process stops. in the training set, we insert this symbol as an extra member of the sequence, immediately after x ( ) τ in each training example. another option is to introduce an extra bernoulli output to the model that represents the decision to either continue generation or halt generation at each time step. this approach is more general than the approach of adding an extra symbol to the vocabulary, because it may be applied to any rnn, rather than only rnns that output a sequence of symbols. for example, it may be applied to an rnn that emits a sequence of real numbers. the new output unit is usually a sigmoid unit trained with the cross - entropy loss. in this approach the sigmoid is trained to maximize


for example, it may be applied to an rnn that emits a sequence of real numbers. the new output unit is usually a sigmoid unit trained with the cross - entropy loss. in this approach the sigmoid is trained to maximize the log - probability of the correct prediction as to whether the sequence ends or continues at each time step. another way to determine the sequence length τ is to add an extra output to the model that predicts the integer τ itself. the model can sample a value of τ and then sample τ steps worth of data. this approach requires adding an extra input to the recurrent update at each time step so that the recurrent update is aware of whether it is near the end of the generated sequence. this extra input can either consist of the value of τ or can consist of τ t −, the number of remaining 390


chapter 10. sequence modeling : recurrent and recursive nets time steps. without this extra input, the rnn might generate sequences that end abruptly, such as a sentence that ends before it is complete. this approach is based on the decomposition p ( x ( 1 ),..., x ( ) τ ) = ( ) ( p τ p x ( 1 ),..., x ( ) τ | τ. ) ( 10. 34 ) the strategy of predicting τ directly is used for example by goodfellow et al. ( ). 2014d 10. 2. 4 modeling sequences conditioned on context with rnns in the previous section we described how an rnn could correspond to a directed graphical model over a sequence of random variables y ( ) t with no inputs x. of course, our development of rnns as in equation included a sequence of 10. 8 inputs x ( 1 ), x ( 2 ),..., x ( ) τ. in general, rnns allow the extension of the graphical model view to represent not only a joint distribution over the y variables but also a conditional distribution over y given x. as discussed in the context of feedforward networks in section, any model representing a variable 6. 2. 1


extension of the graphical model view to represent not only a joint distribution over the y variables but also a conditional distribution over y given x. as discussed in the context of feedforward networks in section, any model representing a variable 6. 2. 1. 1 p ( y ; θ ) can be reinterpreted as a model representing a conditional distribution p ( y ω | ) with ω = θ. we can extend such a model to represent a distribution p ( y x | ) by using the same p ( y ω | ) as before, but making ω a function of x. in the case of an rnn, this can be achieved in [UNK] ways. we review here the most common and obvious choices. previously, we have discussed rnns that take a sequence of vectors x ( ) t for t = 1,..., τ as input. another option is to take only a single vector x as input. when x is a ﬁxed - size vector, we can simply make it an extra input of the rnn that generates the y sequence. some common ways of providing an extra input to an rnn are : 1. as an extra input at each time step, or 2. as the initial state h ( 0 ), or 3.


of the rnn that generates the y sequence. some common ways of providing an extra input to an rnn are : 1. as an extra input at each time step, or 2. as the initial state h ( 0 ), or 3. both. the ﬁrst and most common approach is illustrated in ﬁgure. the interaction 10. 9 between the input x and each hidden unit vector h ( ) t is parametrized by a newly introduced weight matrix r that was absent from the model of only the sequence of y values. the same product xr is added as additional input to the hidden units at every time step. we can think of the choice of x as determining the value 391


chapter 10. sequence modeling : recurrent and recursive nets of xr that is [UNK] a new bias parameter used for each of the hidden units. the weights remain independent of the input. we can think of this model as taking the parameters θ of the non - conditional model and turning them into ω, where the bias parameters within are now a function of the input. ω o ( t−1 ) o ( t−1 ) o ( ) t o ( ) t o ( + 1 ) t o ( + 1 ) t l ( t−1 ) l ( t−1 ) l ( ) t l ( ) t l ( + 1 ) t l ( + 1 ) t y ( t−1 ) y ( t−1 ) y ( ) t y ( ) t y ( + 1 ) t y ( + 1 ) t h ( t−1 ) h ( t−1 ) h ( ) t h ( ) t h ( + 1 ) t h ( + 1 ) t w w w w s ( )... s ( )... h ( )... h ( )... v v v u u u x y ( )... y ( )... r r r r r figure 10. 9 : an rnn


s ( )... h ( )... h ( )... v v v u u u x y ( )... y ( )... r r r r r figure 10. 9 : an rnn that maps a ﬁxed - length vectorx into a distribution over sequences y. this rnn is appropriate for tasks such as image captioning, where a single image is used as input to a model that then produces a sequence of words describing the image. each element y ( ) t of the observed output sequence serves both as input ( for the current time step ) and, during training, as target ( for the previous time step ). rather than receiving only a single vector x as input, the rnn may receive a sequence of vectors x ( ) t as input. the rnn described in equation corre - 10. 8 sponds to a conditional distribution p ( y ( 1 ),..., y ( ) τ | x ( 1 ),..., x ( ) τ ) that makes a conditional independence assumption that this distribution factorizes as t p ( y ( ) t | x ( 1 ),..., x ( ) t ). ( 10. 35 ) to


..., x ( ) τ ) that makes a conditional independence assumption that this distribution factorizes as t p ( y ( ) t | x ( 1 ),..., x ( ) t ). ( 10. 35 ) to remove the conditional independence assumption, we can add connections from the output at time t to the hidden unit at time t + 1, as shown in ﬁgure. the 10. 10 model can then represent arbitrary probability distributions over the y sequence. this kind of model representing a distribution over a sequence given another 392


chapter 10. sequence modeling : recurrent and recursive nets o ( t−1 ) o ( t−1 ) o ( ) t o ( ) t o ( + 1 ) t o ( + 1 ) t l ( t−1 ) l ( t−1 ) l ( ) t l ( ) t l ( + 1 ) t l ( + 1 ) t y ( t−1 ) y ( t−1 ) y ( ) t y ( ) t y ( + 1 ) t y ( + 1 ) t h ( t−1 ) h ( t−1 ) h ( ) t h ( ) t h ( + 1 ) t h ( + 1 ) t w w w w h ( )... h ( )... h ( )... h ( )... v v v u u u x ( t−1 ) x ( t−1 ) r x ( ) t x ( ) t x ( + 1 ) t x ( + 1 ) t r r figure 10. 10 : a conditional recurrent neural network mapping a variable - length sequence of x values into a distribution over sequences of y values of the same length. compared to ﬁgure, this rnn contains connections from the previous output to the current state. 10. 3


conditional recurrent neural network mapping a variable - length sequence of x values into a distribution over sequences of y values of the same length. compared to ﬁgure, this rnn contains connections from the previous output to the current state. 10. 3 these connections allow this rnn to model an arbitrary distribution over sequences ofy given sequences of x of the same length. the rnn of ﬁgure is only able to represent 10. 3 distributions in which the y values are conditionally independent from each other given the values. x 393


chapter 10. sequence modeling : recurrent and recursive nets sequence still has one restriction, which is that the length of both sequences must be the same. we describe how to remove this restriction in section. 10. 4 o ( t−1 ) o ( t−1 ) o ( ) t o ( ) t o ( + 1 ) t o ( + 1 ) t l ( t−1 ) l ( t−1 ) l ( ) t l ( ) t l ( + 1 ) t l ( + 1 ) t y ( t−1 ) y ( t−1 ) y ( ) t y ( ) t y ( + 1 ) t y ( + 1 ) t h ( t−1 ) h ( t−1 ) h ( ) t h ( ) t h ( + 1 ) t h ( + 1 ) t x ( t−1 ) x ( t−1 ) x ( ) t x ( ) t x ( + 1 ) t x ( + 1 ) t g ( t−1 ) g ( t−1 ) g ( ) t g ( ) t g ( + 1 ) t g ( + 1 ) t figure 10. 11 : computation of a typical bidirectional recurrent neural network, meant to learn to map input sequences x to target sequences y,


( ) t g ( ) t g ( + 1 ) t g ( + 1 ) t figure 10. 11 : computation of a typical bidirectional recurrent neural network, meant to learn to map input sequences x to target sequences y, with loss l ( ) t at each step t. the h recurrence propagates information forward in time ( towards the right ) while the g recurrence propagates information backward in time ( towards the left ). thus at each point t, the output units o ( ) t can beneﬁt from a relevant summary of the past in itsh ( ) t input and from a relevant summary of the future in its g ( ) t input. 10. 3 bidirectional rnns all of the recurrent networks we have considered up to now have a “ causal ” struc - ture, meaning that the state at time t only captures information from the past, x ( 1 ),..., x ( 1 ) t−, and the present input x ( ) t. some of the models we have discussed also allow information from past y values to [UNK] the current state when the y values are available. however, in many applications we want to output a prediction of y


##−, and the present input x ( ) t. some of the models we have discussed also allow information from past y values to [UNK] the current state when the y values are available. however, in many applications we want to output a prediction of y ( ) t which may 394


chapter 10. sequence modeling : recurrent and recursive nets depend on the whole input sequence. for example, in speech recognition, the correct interpretation of the current sound as a phoneme may depend on the next few phonemes because of co - articulation and potentially may even depend on the next few words because of the linguistic dependencies between nearby words : if there are two interpretations of the current word that are both acoustically plausible, we may have to look far into the future ( and the past ) to disambiguate them. this is also true of handwriting recognition and many other sequence - to - sequence learning tasks, described in the next section. bidirectional recurrent neural networks ( or bidirectional rnns ) were invented to address that need ( schuster and paliwal 1997, ). they have been extremely suc - cessful ( graves 2012, ) in applications where that need arises, such as handwriting recognition ( graves 2008 graves and schmidhuber 2009 et al., ;, ), speech recogni - tion ( graves and schmidhuber 2005 graves 2013 baldi, ; et al., ) and bioinformatics ( et al., ). 1999 as the name suggests, bid


, ;, ), speech recogni - tion ( graves and schmidhuber 2005 graves 2013 baldi, ; et al., ) and bioinformatics ( et al., ). 1999 as the name suggests, bidirectional rnns combine an rnn that moves forward through time beginning from the start of the sequence with another rnn that moves backward through time beginning from the end of the sequence. figure 10. 11 illustrates the typical bidirectional rnn, with h ( ) t standing for the state of the sub - rnn that moves forward through time and g ( ) t standing for the state of the sub - rnn that moves backward through time. this allows the output units o ( ) t to compute a representation that depends on both the past and the future but is most sensitive to the input values around time t, without having to specify a ﬁxed - size window around t ( as one would have to do with a feedforward network, a convolutional network, or a regular rnn with a ﬁxed - size look - ahead [UNK] ). this idea can be naturally extended to 2 - dimensional input, such as images, by having rnns, each one going in one of the


##lutional network, or a regular rnn with a ﬁxed - size look - ahead [UNK] ). this idea can be naturally extended to 2 - dimensional input, such as images, by having rnns, each one going in one of the four directions : up, down, left, four right. at each point ( i, j ) of a 2 - d grid, an output oi, j could then compute a representation that would capture mostly local information but could also depend on long - range inputs, if the rnn is able to learn to carry that information. compared to a convolutional network, rnns applied to images are typically more expensive but allow for long - range lateral interactions between features in the same feature map (, ; visin et al. 2015 kalchbrenner 2015 et al., ). indeed, the forward propagation equations for such rnns may be written in a form that shows they use a convolution that computes the bottom - up input to each layer, prior to the recurrent propagation across the feature map that incorporates the lateral interactions. 395


chapter 10. sequence modeling : recurrent and recursive nets 10. 4 encoder - decoder sequence - to - sequence architec - tures we have seen in ﬁgure how an rnn can map an input sequence to a ﬁxed - size 10. 5 vector. we have seen in ﬁgure how an rnn can map a ﬁxed - size vector to a 10. 9 sequence. we have seen in ﬁgures,, and how an rnn can 10. 3 10. 4 10. 10 10. 11 map an input sequence to an output sequence of the same length. encoder … x ( 1 ) x ( 1 ) x ( 2 ) x ( 2 ) x ( )... x ( )... x ( nx ) x ( nx ) decoder … y ( 1 ) y ( 1 ) y ( 2 ) y ( 2 ) y ( )... y ( )... y ( ny ) y ( ny ) c figure 10. 12 : example of an encoder - decoder or sequence - to - sequence rnn architecture, for learning to generate an output sequence ( y ( 1 ),..., y ( n y ) ) given


) c figure 10. 12 : example of an encoder - decoder or sequence - to - sequence rnn architecture, for learning to generate an output sequence ( y ( 1 ),..., y ( n y ) ) given an input sequence ( x ( 1 ), x ( 2 ),..., x ( nx ) ). it is composed of an encoder rnn that reads the input sequence and a decoder rnn that generates the output sequence ( or computes the probability of a given output sequence ). the ﬁnal hidden state of the encoder rnn is used to compute a generally ﬁxed - size context variablec which represents a semantic summary of the input sequence and is given as input to the decoder rnn. here we discuss how an rnn can be trained to map an input sequence to an output sequence which is not necessarily of the same length. this comes up in many applications, such as speech recognition, machine translation or question 396


chapter 10. sequence modeling : recurrent and recursive nets answering, where the input and output sequences in the training set are generally not of the same length ( although their lengths might be related ). we often call the input to the rnn the “ context. ” we want to produce a representation of this context, c. the context c might be a vector or sequence of vectors that summarize the input sequence x x = ( ( 1 ),..., x ( nx ) ). the simplest rnn architecture for mapping a variable - length sequence to another variable - length sequence was ﬁrst proposed by ( ) and cho et al. 2014a shortly after by sutskever 2014 et al. ( ), who independently developed that archi - tecture and were the ﬁrst to obtain state - of - the - art translation using this approach. the former system is based on scoring proposals generated by another machine translation system, while the latter uses a standalone recurrent network to generate the translations. these authors respectively called this architecture, illustrated in ﬁgure, the encoder - decoder or sequence - to - sequence architecture. the 10. 12 idea is very simple : ( 1 ) an encoder or reader or input rnn


these authors respectively called this architecture, illustrated in ﬁgure, the encoder - decoder or sequence - to - sequence architecture. the 10. 12 idea is very simple : ( 1 ) an encoder or reader or input rnn processes the input sequence. the encoder emits the context c, usually as a simple function of its ﬁnal hidden state. ( 2 ) a decoder or writer or output rnn is conditioned on that ﬁxed - length vector ( just like in ﬁgure ) to generate the output sequence 10. 9 y = ( y ( 1 ),..., y ( ny ) ). the innovation of this kind of architecture over those presented in earlier sections of this chapter is that the lengths nx and ny can vary from each other, while previous architectures constrained nx = ny = τ. in a sequence - to - sequence architecture, the two rnns are trained jointly to maximize the average of log p ( y ( 1 ),..., y ( ny ) | x ( 1 ),..., x ( nx ) ) over all the pairs of x and y sequences in the training set. the last state hnx of the encoder rnn is typically used


, y ( ny ) | x ( 1 ),..., x ( nx ) ) over all the pairs of x and y sequences in the training set. the last state hnx of the encoder rnn is typically used as a representation c of the input sequence that is provided as input to the decoder rnn. if the context c is a vector, then the decoder rnn is simply a vector - to - sequence rnn as described in section. as we have seen, there are at least 10. 2. 4 two ways for a vector - to - sequence rnn to receive input. the input can be provided as the initial state of the rnn, or the input can be connected to the hidden units at each time step. these two ways can also be combined. there is no constraint that the encoder must have the same size of hidden layer as the decoder. one clear limitation of this architecture is when the context c output by the encoder rnn has a dimension that is too small to properly summarize a long sequence. this phenomenon was observed by ( ) in the context bahdanau et al. 2015 of machine translation. they proposed to make c a variable - length sequence rather than


has a dimension that is too small to properly summarize a long sequence. this phenomenon was observed by ( ) in the context bahdanau et al. 2015 of machine translation. they proposed to make c a variable - length sequence rather than a ﬁxed - size vector. additionally, they introduced an attention mechanism that learns to associate elements of the sequence c to elements of the output 397


chapter 10. sequence modeling : recurrent and recursive nets sequence. see section for more details. 12. 4. 5. 1 10. 5 deep recurrent networks the computation in most rnns can be decomposed into three blocks of parameters and associated transformations : 1. from the input to the hidden state, 2. from the previous hidden state to the next hidden state, and 3. from the hidden state to the output. with the rnn architecture of ﬁgure, each of these three blocks is associated 10. 3 with a single weight matrix. in other words, when the network is unfolded, each of these corresponds to a shallow transformation. by a shallow transformation, we mean a transformation that would be represented by a single layer within a deep mlp. typically this is a transformation represented by a learned [UNK] transformation followed by a ﬁxed nonlinearity. would it be advantageous to introduce depth in each of these operations? experimental evidence ( graves 2013 pascanu 2014a et al., ; et al., ) strongly suggests so. the experimental evidence is in agreement with the idea that we need enough depth in order to perform the required mappings. see also schmidhuber 1992 ( ), el hihi and bengio 1996 jaeger


al., ) strongly suggests so. the experimental evidence is in agreement with the idea that we need enough depth in order to perform the required mappings. see also schmidhuber 1992 ( ), el hihi and bengio 1996 jaeger 2007a ( ), or ( ) for earlier work on deep rnns. graves 2013 et al. ( ) were the ﬁrst to show a signiﬁcant beneﬁt of decomposing the state of an rnn into multiple layers as in ﬁgure ( left ). we can think 10. 13 of the lower layers in the hierarchy depicted in ﬁgure a as playing a role 10. 13 in transforming the raw input into a representation that is more appropriate, at the higher levels of the hidden state. pascanu 2014a et al. ( ) go a step further and propose to have a separate mlp ( possibly deep ) for each of the three blocks enumerated above, as illustrated in ﬁgure b. considerations of representational 10. 13 capacity suggest to allocate enough capacity in each of these three steps, but doing so by adding depth may hurt learning by making optimization [UNK]. in general, it is easier to optimize shallower architectures, and


representational 10. 13 capacity suggest to allocate enough capacity in each of these three steps, but doing so by adding depth may hurt learning by making optimization [UNK]. in general, it is easier to optimize shallower architectures, and adding the extra depth of ﬁgure b makes the shortest path from a variable in time step 10. 13 t to a variable in time step t + 1 become longer. for example, if an mlp with a single hidden layer is used for the state - to - state transition, we have doubled the length of the shortest path between variables in any two [UNK] time steps, compared with the ordinary rnn of ﬁgure. however, as argued by 10. 3 pascanu 2014a et al. ( ), this 398


chapter 10. sequence modeling : recurrent and recursive nets h y x z ( a ) ( b ) ( c ) x h y x h y figure 10. 13 : a recurrent neural network can be made deep in many ways ( pascanu et al., ). the hidden recurrent state can be broken down into groups organized 2014a ( a ) hierarchically. deeper computation ( e. g., an mlp ) can be introduced in the input - to - ( b ) hidden, hidden - to - hidden and hidden - to - output parts. this may lengthen the shortest path linking [UNK] time steps. the path - lengthening [UNK] can be mitigated by ( c ) introducing skip connections. 399


chapter 10. sequence modeling : recurrent and recursive nets can be mitigated by introducing skip connections in the hidden - to - hidden path, as illustrated in ﬁgure c. 10. 13 10. 6 recursive neural networks x ( 1 ) x ( 1 ) x ( 2 ) x ( 2 ) x ( 3 ) x ( 3 ) v v v y l x ( 4 ) x ( 4 ) v o u w u w u w figure 10. 14 : a recursive network has a computational graph that generalizes that of the recurrent network from a chain to a tree. a variable - size sequencex ( 1 ), x ( 2 ),..., x ( ) t can be mapped to a ﬁxed - size representation ( the outputo ), with a ﬁxed set of parameters ( the weight matrices u, v, w ). the ﬁgure illustrates a supervised learning case in which some target is provided which is associated with the whole sequence. y recursive neural networks2 represent yet another generalization of recurrent networks, with a [UNK] kind of computational graph, which is structured as a deep tree, rather than the chain - like structure of rnns. the typical computational graph for a rec


recursive neural networks2 represent yet another generalization of recurrent networks, with a [UNK] kind of computational graph, which is structured as a deep tree, rather than the chain - like structure of rnns. the typical computational graph for a recursive network is illustrated in ﬁgure. recursive neural 10. 14 2we suggest to not abbreviate “ recursive neural network ” as “ rnn ” to avoid confusion with “ recurrent neural network. ” 400


chapter 10. sequence modeling : recurrent and recursive nets networks were introduced by pollack 1990 ( ) and their potential use for learning to reason was described by ( ). recursive networks have been successfully bottou 2011 applied to processing data structures as input to neural nets ( frasconi 1997 et al.,, 1998 socher 2011a c 2013a ), in natural language processing ( et al.,,, ) as well as in computer vision (, ). socher et al. 2011b one clear advantage of recursive nets over recurrent nets is that for a sequence of the same length τ, the depth ( measured as the number of compositions of nonlinear operations ) can be drastically reduced from τ to o ( log τ ), which might help deal with long - term dependencies. an open question is how to best structure the tree. one option is to have a tree structure which does not depend on the data, such as a balanced binary tree. in some application domains, external methods can suggest the appropriate tree structure. for example, when processing natural language sentences, the tree structure for the recursive network can be ﬁxed to the structure of the parse tree of the sentence provided by a natural language parser (,,


can suggest the appropriate tree structure. for example, when processing natural language sentences, the tree structure for the recursive network can be ﬁxed to the structure of the parse tree of the sentence provided by a natural language parser (,, ). ideally, one would like the learner itself to socher et al. 2011a 2013a discover and infer the tree structure that is appropriate for any given input, as suggested by ( ). bottou 2011 many variants of the recursive net idea are possible. for example, frasconi et al. ( ) and 1997 frasconi 1998 et al. ( ) associate the data with a tree structure, and associate the inputs and targets with individual nodes of the tree. the computation performed by each node does not have to be the traditional artiﬁcial neuron computation ( [UNK] transformation of all inputs followed by a monotone nonlinearity ). for example, ( ) propose using tensor operations socher et al. 2013a and bilinear forms, which have previously been found useful to model relationships between concepts ( weston 2010 bordes 2012 et al., ; et al., ) when the concepts are represented by continuous vectors ( embeddings ). 10. 7 the challenge of long


, which have previously been found useful to model relationships between concepts ( weston 2010 bordes 2012 et al., ; et al., ) when the concepts are represented by continuous vectors ( embeddings ). 10. 7 the challenge of long - term dependencies the mathematical challenge of learning long - term dependencies in recurrent net - works was introduced in section. the basic problem is that gradients prop - 8. 2. 5 agated over many stages tend to either vanish ( most of the time ) or explode ( rarely, but with much damage to the optimization ). even if we assume that the parameters are such that the recurrent network is stable ( can store memories, with gradients not exploding ), the [UNK] with long - term dependencies arises from the exponentially smaller weights given to long - term interactions ( involving the multiplication of many jacobians ) compared to short - term ones. many other sources provide a deeper treatment (, ; hochreiter 1991 doya 1993 bengio, ; et al., 401


chapter 10. sequence modeling : recurrent and recursive nets − − − 60 40 20 0 20 40 60 input coordinate −4 −3 −2 −1 0 1 2 3 4 projection of output 0 1 2 3 4 5 figure 10. 15 : when composing many nonlinear functions ( like the linear - tanh layer shown here ), the result is highly nonlinear, typically with most of the values associated with a tiny derivative, some values with a large derivative, and many alternations between increasing and decreasing. in this plot, we plot a linear projection of a 100 - dimensional hidden state down to a single dimension, plotted on the y - axis. the x - axis is the coordinate of the initial state along a random direction in the 100 - dimensional space. we can thus view this plot as a linear cross - section of a high - dimensional function. the plots show the function after each time step, or equivalently, after each number of times the transition function has been composed. 1994 pascanu 2013 ; et al., ). in this section, we describe the problem in more detail. the remaining sections describe approaches to overcoming the problem. recurrent networks involve the composition of the same function multiple times, once per time step. these compositions can result


et al., ). in this section, we describe the problem in more detail. the remaining sections describe approaches to overcoming the problem. recurrent networks involve the composition of the same function multiple times, once per time step. these compositions can result in extremely nonlinear behavior, as illustrated in ﬁgure. 10. 15 in particular, the function composition employed by recurrent neural networks somewhat resembles matrix multiplication. we can think of the recurrence relation h ( ) t = w h ( 1 ) t− ( 10. 36 ) as a very simple recurrent neural network lacking a nonlinear activation function, and lacking inputs x. as described in section, this recurrence relation 8. 2. 5 essentially describes the power method. it may be simpliﬁed to h ( ) t = w th ( 0 ), ( 10. 37 ) and if admits an eigendecomposition of the form w w q q = λ, ( 10. 38 ) 402


chapter 10. sequence modeling : recurrent and recursive nets with orthogonal, the recurrence may be simpliﬁed further to q h ( ) t = qλtqh ( 0 ). ( 10. 39 ) the eigenvalues are raised to the power of t causing eigenvalues with magnitude less than one to decay to zero and eigenvalues with magnitude greater than one to explode. any component of h ( 0 ) that is not aligned with the largest eigenvector will eventually be discarded. this problem is particular to recurrent networks. in the scalar case, imagine multiplying a weight w by itself many times. the product wt will either vanish or explode depending on the magnitude of w. however, if we make a non - recurrent network that has a [UNK] weightw ( ) t at each time step, the situation is [UNK]. if the initial state is given by, then the state at time 1 t is given by tw ( ) t. suppose that the w ( ) t values are generated randomly, independently from one another, with zero mean and variance v. the variance of the product is o ( vn ). to obtain some desired variance v∗we may choose the individual


t. suppose that the w ( ) t values are generated randomly, independently from one another, with zero mean and variance v. the variance of the product is o ( vn ). to obtain some desired variance v∗we may choose the individual weights with variance v = n√ v∗. very deep feedforward networks with carefully chosen scaling can thus avoid the vanishing and exploding gradient problem, as argued by ( ). sussillo 2014 the vanishing and exploding gradient problem for rnns was independently discovered by separate researchers (, ;,, ). hochreiter 1991 bengio et al. 1993 1994 one may hope that the problem can be avoided simply by staying in a region of parameter space where the gradients do not vanish or explode. unfortunately, in order to store memories in a way that is robust to small perturbations, the rnn must enter a region of parameter space where gradients vanish (,, bengio et al. 1993 1994 ). speciﬁcally, whenever the model is able to represent long term dependencies, the gradient of a long term interaction has exponentially smaller magnitude than the gradient of a short term interaction. it does not mean that it is impossible to learn, but that it might take a very long time


is able to represent long term dependencies, the gradient of a long term interaction has exponentially smaller magnitude than the gradient of a short term interaction. it does not mean that it is impossible to learn, but that it might take a very long time to learn long - term dependencies, because the signal about these dependencies will tend to be hidden by the smallest ﬂuctuations arising from short - term dependencies. in practice, the experiments in ( ) show that as we increase the span of the dependencies that bengio et al. 1994 need to be captured, gradient - based optimization becomes increasingly [UNK], with the probability of successful training of a traditional rnn via sgd rapidly reaching 0 for sequences of only length 10 or 20. for a deeper treatment of recurrent networks as dynamical systems, see doya ( ), ( ) and ( ), with a review 1993 bengio et al. 1994 siegelmann and sontag 1995 in pascanu 2013 et al. ( ). the remaining sections of this chapter discuss various approaches that have been proposed to reduce the [UNK] of learning long - term dependencies ( in some cases allowing an rnn to learn dependencies across 403


chapter 10. sequence modeling : recurrent and recursive nets hundreds of steps ), but the problem of learning long - term dependencies remains one of the main challenges in deep learning. 10. 8 echo state networks the recurrent weights mapping from h ( 1 ) t− to h ( ) t and the input weights mapping from x ( ) t to h ( ) t are some of the most [UNK] parameters to learn in a recurrent network. one proposed (, ;, ;, ; jaeger 2003 maass et al. 2002 jaeger and haas 2004 jaeger 2007b, ) approach to avoiding this [UNK] is to set the recurrent weights such that the recurrent hidden units do a good job of capturing the history of past inputs, and learn only the output weights. this is the idea that was independently proposed for echo state networks or esns (, ;, ) jaeger and haas 2004 jaeger 2007b and liquid state machines (, ). the latter is similar, except maass et al. 2002 that it uses spiking neurons ( with binary outputs ) instead of the continuous - valued hidden units used for esns. both esns and liquid state machines are termed reservoir computing ( lukosevicius and jaeger 2009, ) to denote


2002 that it uses spiking neurons ( with binary outputs ) instead of the continuous - valued hidden units used for esns. both esns and liquid state machines are termed reservoir computing ( lukosevicius and jaeger 2009, ) to denote the fact that the hidden units form of reservoir of temporal features which may capture [UNK] aspects of the history of inputs. one way to think about these reservoir computing recurrent networks is that they are similar to kernel machines : they map an arbitrary length sequence ( the history of inputs up to time t ) into a ﬁxed - length vector ( the recurrent state h ( ) t ), on which a linear predictor ( typically a linear regression ) can be applied to solve the problem of interest. the training criterion may then be easily designed to be convex as a function of the output weights. for example, if the output consists of linear regression from the hidden units to the output targets, and the training criterion is mean squared error, then it is convex and may be solved reliably with simple learning algorithms (, ). jaeger 2003 the important question is therefore : how do we set the input and recurrent weights so that a rich set of histories can be represented in the recurrent neural network state? the answer proposed in


##bly with simple learning algorithms (, ). jaeger 2003 the important question is therefore : how do we set the input and recurrent weights so that a rich set of histories can be represented in the recurrent neural network state? the answer proposed in the reservoir computing literature is to view the recurrent net as a dynamical system, and set the input and recurrent weights such that the dynamical system is near the edge of stability. the original idea was to make the eigenvalues of the jacobian of the state - to - state transition function be close to. as explained in section, an important 1 8. 2. 5 characteristic of a recurrent network is the eigenvalue spectrum of the jacobians j ( ) t = ∂s ( ) t ∂s ( 1 ) t−. of particular importance is the spectral radius of j ( ) t, deﬁned to be the maximum of the absolute values of its eigenvalues. 404


chapter 10. sequence modeling : recurrent and recursive nets to understand the [UNK] of the spectral radius, consider the simple case of back - propagation with a jacobian matrix j that does not change with t. this case happens, for example, when the network is purely linear. suppose that j has an eigenvector v with corresponding eigenvalue λ. consider what happens as we propagate a gradient vector backwards through time. if we begin with a gradient vector g, then after one step of back - propagation, we will have jg, and after n steps we will have jng. now consider what happens if we instead back - propagate a perturbed version of g. if we begin with g + δv, then after one step, we will have j ( g + δv ). after n steps, we will have j n ( g + δv ). from this we can see that back - propagation starting from g and back - propagation starting from g + δv diverge by δj nv after n steps of back - propagation. if v is chosen to be a unit eigenvector of j with eigenvalue λ, then multiplication by the jacobian simply scales the [UNK] at each step


##v diverge by δj nv after n steps of back - propagation. if v is chosen to be a unit eigenvector of j with eigenvalue λ, then multiplication by the jacobian simply scales the [UNK] at each step. the two executions of back - propagation are separated by a distance of δ λ | | n. when v corresponds to the largest value of | | λ, this perturbation achieves the widest possible separation of an initial perturbation of size. δ when | | λ > 1, the deviation size δ λ | | n grows exponentially large. when | | λ < 1, the deviation size becomes exponentially small. of course, this example assumed that the jacobian was the same at every time step, corresponding to a recurrent network with no nonlinearity. when a nonlinearity is present, the derivative of the nonlinearity will approach zero on many time steps, and help to prevent the explosion resulting from a large spectral radius. indeed, the most recent work on echo state networks advocates using a spectral radius much larger than unity (, ;, ). yildiz et al. 2012 jaeger 2012 everything we have said about back - propagation via repeated matrix multipli - cation applies equally to


work on echo state networks advocates using a spectral radius much larger than unity (, ;, ). yildiz et al. 2012 jaeger 2012 everything we have said about back - propagation via repeated matrix multipli - cation applies equally to forward propagation in a network with no nonlinearity, where the state h ( + 1 ) t = h ( ) t w. when a linear map w always shrinks h as measured by the l2 norm, then we say that the map is contractive. when the spectral radius is less than one, the mapping from h ( ) t to h ( + 1 ) t is contractive, so a small change becomes smaller after each time step. this necessarily makes the network forget information about the past when we use a ﬁnite level of precision ( such as 32 bit integers ) to store the state vector. the jacobian matrix tells us how a small change of h ( ) t propagates one step forward, or equivalently, how the gradient on h ( + 1 ) t propagates one step backward, during back - propagation. note that neither w nor j need to be symmetric ( al - though they are square and real ), so they can have complex - valued eigenvalues and e


+ 1 ) t propagates one step backward, during back - propagation. note that neither w nor j need to be symmetric ( al - though they are square and real ), so they can have complex - valued eigenvalues and eigenvectors, with imaginary components corresponding to potentially oscillatory 405


chapter 10. sequence modeling : recurrent and recursive nets behavior ( if the same jacobian was applied iteratively ). even though h ( ) t or a small variation of h ( ) t of interest in back - propagation are real - valued, they can be expressed in such a complex - valued basis. what matters is what happens to the magnitude ( complex absolute value ) of these possibly complex - valued basis [UNK], when we multiply the matrix by the vector. an eigenvalue with magnitude greater than one corresponds to magniﬁcation ( exponential growth, if applied iteratively ) or shrinking ( exponential decay, if applied iteratively ). with a nonlinear map, the jacobian is free to change at each step. the dynamics therefore become more complicated. however, it remains true that a small initial variation can turn into a large variation after several steps. one [UNK] between the purely linear case and the nonlinear case is that the use of a squashing nonlinearity such as tanh can cause the recurrent dynamics to become bounded. note that it is possible for back - propagation to retain unbounded dynamics even when forward propagation has bounded dynamics, for example, when a sequence of tanh units are all in the middle of their linear regime


the recurrent dynamics to become bounded. note that it is possible for back - propagation to retain unbounded dynamics even when forward propagation has bounded dynamics, for example, when a sequence of tanh units are all in the middle of their linear regime and are connected by weight matrices with spectral radius greater than. however, it is 1 rare for all of the units to simultaneously lie at their linear activation point. tanh the strategy of echo state networks is simply to ﬁx the weights to have some spectral radius such as, where information is carried forward through time but 3 does not explode due to the stabilizing [UNK] of saturating nonlinearities like tanh. more recently, it has been shown that the techniques used to set the weights in esns could be used to the weights in a fully trainable recurrent net - initialize work ( with the hidden - to - hidden recurrent weights trained using back - propagation through time ), helping to learn long - term dependencies ( sutskever 2012 sutskever, ; et al., ). in this setting, an initial spectral radius of 1. 2 performs well, combined 2013 with the sparse initialization scheme described in section. 8. 4 10. 9 leaky units and other strategies for multiple time


, ; et al., ). in this setting, an initial spectral radius of 1. 2 performs well, combined 2013 with the sparse initialization scheme described in section. 8. 4 10. 9 leaky units and other strategies for multiple time scales one way to deal with long - term dependencies is to design a model that operates at multiple time scales, so that some parts of the model operate at ﬁne - grained time scales and can handle small details, while other parts operate at coarse time scales and transfer information from the distant past to the present more [UNK]. various strategies for building both ﬁne and coarse time scales are possible. these include the addition of skip connections across time, “ leaky units ” that integrate signals with [UNK] time constants, and the removal of some of the connections 406


chapter 10. sequence modeling : recurrent and recursive nets used to model ﬁne - grained time scales. 10. 9. 1 adding skip connections through time one way to obtain coarse time scales is to add direct connections from variables in the distant past to variables in the present. the idea of using such skip connections dates back to ( ) and follows from the idea of incorporating delays in lin et al. 1996 feedforward neural networks (, ). in an ordinary recurrent lang and hinton 1988 network, a recurrent connection goes from a unit at time t to a unit at time t + 1. it is possible to construct recurrent networks with longer delays (, ). bengio 1991 as we have seen in section, gradients may vanish or explode exponentially 8. 2. 5 with respect to the number of time steps. ( ) introduced recurrent lin et al. 1996 connections with a time - delay of d to mitigate this problem. gradients now diminish exponentially as a function of τ d rather than τ. since there are both delayed and single step connections, gradients may still explode exponentially in τ. this allows the learning algorithm to capture longer dependencies although not all long - term dependencies may be represented well


function of τ d rather than τ. since there are both delayed and single step connections, gradients may still explode exponentially in τ. this allows the learning algorithm to capture longer dependencies although not all long - term dependencies may be represented well in this way. 10. 9. 2 leaky units and a spectrum of [UNK] time scales another way to obtain paths on which the product of derivatives is close to one is to have units with linear self - connections and a weight near one on these connections. when we accumulate a running average µ ( ) t of some value v ( ) t by applying the update µ ( ) t ←αµ ( 1 ) t− + ( 1 −α ) v ( ) t the α parameter is an example of a linear self - connection from µ ( 1 ) t− to µ ( ) t. when α is near one, the running average remembers information about the past for a long time, and when α is near zero, information about the past is rapidly discarded. hidden units with linear self - connections can behave similarly to such running averages. such hidden units are called leaky units. skip connections through d time steps are a way of ensuring that a unit can always learn to be inﬂuenced by a value from d


with linear self - connections can behave similarly to such running averages. such hidden units are called leaky units. skip connections through d time steps are a way of ensuring that a unit can always learn to be inﬂuenced by a value from d time steps earlier. the use of a linear self - connection with a weight near one is a [UNK] way of ensuring that the unit can access values from the past. the linear self - connection approach allows this [UNK] to be adapted more smoothly and ﬂexibly by adjusting the real - valued α rather than by adjusting the integer - valued skip length. these ideas were proposed by ( ) and by ( ). mozer 1992 el hihi and bengio 1996 leaky units were also found to be useful in the context of echo state networks (, ). jaeger et al. 2007 407


chapter 10. sequence modeling : recurrent and recursive nets there are two basic strategies for setting the time constants used by leaky units. one strategy is to manually ﬁx them to values that remain constant, for example by sampling their values from some distribution once at initialization time. another strategy is to make the time constants free parameters and learn them. having such leaky units at [UNK] time scales appears to help with long - term dependencies (, ; mozer 1992 pascanu 2013 et al., ). 10. 9. 3 removing connections another approach to handle long - term dependencies is the idea of organizing the state of the rnn at multiple time - scales (, ), with el hihi and bengio 1996 information ﬂowing more easily through long distances at the slower time scales. this idea [UNK] from the skip connections through time discussed earlier because it involves actively removing length - one connections and replacing them with longer connections. units modiﬁed in such a way are forced to operate on a long time scale. skip connections through time edges. units receiving such add new connections may learn to operate on a long time scale but may also choose to focus on their other short - term connections. there are [UNK] ways in which a group of


on a long time scale. skip connections through time edges. units receiving such add new connections may learn to operate on a long time scale but may also choose to focus on their other short - term connections. there are [UNK] ways in which a group of recurrent units can be forced to operate at [UNK] time scales. one option is to make the recurrent units leaky, but to have [UNK] groups of units associated with [UNK] ﬁxed time scales. this was the proposal in ( ) and has been successfully used in mozer 1992 pascanu et al. ( ). another option is to have explicit and discrete updates taking place 2013 at [UNK] times, with a [UNK] frequency for [UNK] groups of units. this is the approach of ( ) and el hihi and bengio 1996 koutnik 2014 et al. ( ). it worked well on a number of benchmark datasets. 10. 10 the long short - term memory and other gated rnns as of this writing, the most [UNK] sequence models used in practical applications are called gated rnns. these include the long short - term memory and networks based on the. gated recurrent unit like leaky units, gated rnns are based on the idea of creating paths through time that have derivatives


practical applications are called gated rnns. these include the long short - term memory and networks based on the. gated recurrent unit like leaky units, gated rnns are based on the idea of creating paths through time that have derivatives that neither vanish nor explode. leaky units did this with connection weights that were either manually chosen constants or were parameters. gated rnns generalize this to connection weights that may change 408


chapter 10. sequence modeling : recurrent and recursive nets at each time step. × input input gate forget gate output gate output state self - loop × + × figure 10. 16 : block diagram of the lstm recurrent network “ cell. ” cells are connected recurrently to each other, replacing the usual hidden units of ordinary recurrent networks. an input feature is computed with a regular artiﬁcial neuron unit. its value can be accumulated into the state if the sigmoidal input gate allows it. the state unit has a linear self - loop whose weight is controlled by the forget gate. the output of the cell can be shut [UNK] the output gate. all the gating units have a sigmoid nonlinearity, while the input unit can have any squashing nonlinearity. the state unit can also be used as an extra input to the gating units. the black square indicates a delay of a single time step. leaky units allow the network to accumulate information ( such as evidence for a particular feature or category ) over a long duration. however, once that information has been used, it might be useful for the neural network to forget the old state. for example, if a sequence is made of sub - sequences and we


evidence for a particular feature or category ) over a long duration. however, once that information has been used, it might be useful for the neural network to forget the old state. for example, if a sequence is made of sub - sequences and we want a leaky unit to accumulate evidence inside each sub - subsequence, we need a mechanism to forget the old state by setting it to zero. instead of manually deciding when to clear the state, we want the neural network to learn to decide when to do it. this 409


chapter 10. sequence modeling : recurrent and recursive nets is what gated rnns do. 10. 10. 1 lstm the clever idea of introducing self - loops to produce paths where the gradient can ﬂow for long durations is a core contribution of the initial long short - term memory ( lstm ) model ( hochreiter and schmidhuber 1997, ). a crucial addition has been to make the weight on this self - loop conditioned on the context, rather than ﬁxed (, ). by making the weight of this self - loop gated ( controlled gers et al. 2000 by another hidden unit ), the time scale of integration can be changed dynamically. in this case, we mean that even for an lstm with ﬁxed parameters, the time scale of integration can change based on the input sequence, because the time constants are output by the model itself. the lstm has been found extremely successful in many applications, such as unconstrained handwriting recognition ( graves et al., ), speech recognition ( 2009 graves 2013 graves and jaitly 2014 et al., ;, ), handwriting generation ( graves 2013, ), machine translation ( sutskever 2014 et al., ),


handwriting recognition ( graves et al., ), speech recognition ( 2009 graves 2013 graves and jaitly 2014 et al., ;, ), handwriting generation ( graves 2013, ), machine translation ( sutskever 2014 et al., ), image captioning (, ; kiros et al. 2014b vinyals 2014b xu 2015 et al., ; et al., ) and parsing ( vinyals 2014a et al., ). the lstm block diagram is illustrated in ﬁgure. the corresponding 10. 16 forward propagation equations are given below, in the case of a shallow recurrent network architecture. deeper architectures have also been successfully used ( graves et al., ; 2013 pascanu 2014a et al., ). instead of a unit that simply applies an element - wise nonlinearity to the [UNK] transformation of inputs and recurrent units, lstm recurrent networks have “ lstm cells ” that have an internal recurrence ( a self - loop ), in addition to the outer recurrence of the rnn. each cell has the same inputs and outputs as an ordinary recurrent network, but has more parameters and a system of gating units that controls the ﬂow of information


- loop ), in addition to the outer recurrence of the rnn. each cell has the same inputs and outputs as an ordinary recurrent network, but has more parameters and a system of gating units that controls the ﬂow of information. the most important component is the state unit s ( ) t i that has a linear self - loop similar to the leaky units described in the previous section. however, here, the self - loop weight ( or the associated time constant ) is controlled by a forget gate unit f ( ) t i ( for time step t and cell ), that sets this weight to a value between 0 and 1 via a sigmoid unit : i f ( ) t i = σ bf i + j u f i, j x ( ) t j + j wf i, j h ( 1 ) t− j, ( 10. 40 ) where x ( ) t is the current input vector and h ( ) t is the current hidden layer vector, containing the outputs of all the lstm cells, and bf, uf, w f are respectively biases, input weights and recurrent weights for the forget gates. the lstm cell 410


chapter 10. sequence modeling : recurrent and recursive nets internal state is thus updated as follows, but with a conditional self - loop weight f ( ) t i : s ( ) t i = f ( ) t i s ( 1 ) t− i + g ( ) t i σ b i + j ui, jx ( ) t j + j wi, jh ( 1 ) t− j, ( 10. 41 ) where b, u and w respectively denote the biases, input weights and recurrent weights into the lstm cell. the external input gate unit g ( ) t i is computed similarly to the forget gate ( with a sigmoid unit to obtain a gating value between 0 and 1 ), but with its own parameters : g ( ) t i = σ bg i + j ug i, jx ( ) t j + j wg i, jh ( 1 ) t− j. ( 10. 42 ) the output h ( ) t i of the lstm cell can also be shut [UNK], via the output gate q ( ) t i, which also uses a sigmoid unit for gating : h ( ) t i = tanh s ( ) t i q ( )


of the lstm cell can also be shut [UNK], via the output gate q ( ) t i, which also uses a sigmoid unit for gating : h ( ) t i = tanh s ( ) t i q ( ) t i ( 10. 43 ) q ( ) t i = σ bo i + j uo i, jx ( ) t j + j w o i, j h ( 1 ) t− j ( 10. 44 ) which has parameters bo, u o, w o for its biases, input weights and recurrent weights, respectively. among the variants, one can choose to use the cell state s ( ) t i as an extra input ( with its weight ) into the three gates of the i - th unit, as shown in ﬁgure. this would require three additional parameters. 10. 16 lstm networks have been shown to learn long - term dependencies more easily than the simple recurrent architectures, ﬁrst on artiﬁcial data sets designed for testing the ability to learn long - term dependencies (, ; bengio et al. 1994 hochreiter and schmidhuber 1997 hochreiter 2001, ; et al., ), then on challenging sequence processing tasks


data sets designed for testing the ability to learn long - term dependencies (, ; bengio et al. 1994 hochreiter and schmidhuber 1997 hochreiter 2001, ; et al., ), then on challenging sequence processing tasks where state - of - the - art performance was obtained ( graves 2012, ; graves 2013 sutskever 2014 et al., ; et al., ). variants and alternatives to the lstm have been studied and used and are discussed next. 10. 10. 2 other gated rnns which pieces of the lstm architecture are actually necessary? what other successful architectures could be designed that allow the network to dynamically control the time scale and forgetting behavior of [UNK] units? 411


chapter 10. sequence modeling : recurrent and recursive nets some answers to these questions are given with the recent work on gated rnns, whose units are also known as gated recurrent units or grus (, ; cho et al. 2014b chung 2014 2015a jozefowicz 2015 chrupala 2015 et al.,, ; et al., ; et al., ). the main [UNK] with the lstm is that a single gating unit simultaneously controls the forgetting factor and the decision to update the state unit. the update equations are the following : h ( ) t i = u ( 1 ) t− i h ( 1 ) t− i + ( 1 −u ( 1 ) t− i ) σ bi + j ui, jx ( 1 ) t− j + j wi, j r ( 1 ) t− j h ( 1 ) t− j, ( 10. 45 ) where u stands for “ update ” gate and r for “ reset ” gate. their value is deﬁned as usual : u ( ) t i = σ bu i + j uu i, jx ( ) t j + j wu i, jh ( ) t j ( 10. 46 ) and r ( ) t


value is deﬁned as usual : u ( ) t i = σ bu i + j uu i, jx ( ) t j + j wu i, jh ( ) t j ( 10. 46 ) and r ( ) t i = σ br i + j u r i, jx ( ) t j + j wr i, jh ( ) t j. ( 10. 47 ) the reset and updates gates can individually “ ignore ” parts of the state vector. the update gates act like conditional leaky integrators that can linearly gate any dimension, thus choosing to copy it ( at one extreme of the sigmoid ) or completely ignore it ( at the other extreme ) by replacing it by the new “ target state ” value ( towards which the leaky integrator wants to converge ). the reset gates control which parts of the state get used to compute the next target state, introducing an additional nonlinear [UNK] in the relationship between past state and future state. many more variants around this theme can be designed. for example the reset gate ( or forget gate ) output could be shared across multiple hidden units. alternately, the product of a global gate ( covering a whole group of units, such as an entire layer ) and a


around this theme can be designed. for example the reset gate ( or forget gate ) output could be shared across multiple hidden units. alternately, the product of a global gate ( covering a whole group of units, such as an entire layer ) and a local gate ( per unit ) could be used to combine global control and local control. however, several investigations over architectural variations of the lstm and gru found no variant that would clearly beat both of these across a wide range of tasks (, ; [UNK] al. 2015 jozefowicz 2015 [UNK] et al., ). et al. ( ) found that a crucial ingredient is the forget gate, while 2015 jozefowicz et al. ( ) found that adding a bias of 1 to the lstm forget gate, a practice 2015 advocated by ( ), makes the lstm as strong as the best of the gers et al. 2000 explored architectural variants. 412


chapter 10. sequence modeling : recurrent and recursive nets 10. 11 optimization for long - term dependencies section and section have described the vanishing and exploding gradient 8. 2. 5 10. 7 problems that occur when optimizing rnns over many time steps. an interesting idea proposed by martens and sutskever 2011 ( ) is that second derivatives may vanish at the same time that ﬁrst derivatives vanish. second - order optimization algorithms may roughly be understood as dividing the ﬁrst derivative by the second derivative ( in higher dimension, multiplying the gradient by the inverse hessian ). if the second derivative shrinks at a similar rate to the ﬁrst derivative, then the ratio of ﬁrst and second derivatives may remain relatively constant. unfortunately, second - order methods have many drawbacks, including high computational cost, the need for a large minibatch, and a tendency to be attracted to saddle points. martens and sutskever 2011 ( ) found promising results using second - order methods. later, sutskever 2013 et al. ( ) found that simpler methods such as nesterov momentum with careful initialization could achieve similar results. see sutskever 2012 ( ) for more detail. both of these approaches have largely been replaced by simply using


sutskever 2013 et al. ( ) found that simpler methods such as nesterov momentum with careful initialization could achieve similar results. see sutskever 2012 ( ) for more detail. both of these approaches have largely been replaced by simply using sgd ( even without momentum ) applied to lstms. this is part of a continuing theme in machine learning that it is often much easier to design a model that is easy to optimize than it is to design a more powerful optimization algorithm. 10. 11. 1 clipping gradients as discussed in section, strongly nonlinear functions such as those computed 8. 2. 4 by a recurrent net over many time steps tend to have derivatives that can be either very large or very small in magnitude. this is illustrated in ﬁgure and 8. 3 ﬁgure, in which we see that the objective function ( as a function of the 10. 17 parameters ) has a “ landscape ” in which one ﬁnds “ [UNK] ” : wide and rather ﬂat regions separated by tiny regions where the objective function changes quickly, forming a kind of [UNK]. the [UNK] that arises is that when the parameter gradient is very large, a gradient descent parameter update could throw the parameters very far, into a region where the objective function


separated by tiny regions where the objective function changes quickly, forming a kind of [UNK]. the [UNK] that arises is that when the parameter gradient is very large, a gradient descent parameter update could throw the parameters very far, into a region where the objective function is larger, undoing much of the work that had been done to reach the current solution. the gradient tells us the direction that corresponds to the steepest descent within an inﬁnitesimal region surrounding the current parameters. outside of this inﬁnitesimal region, the cost function may begin to curve back upwards. the update must be chosen to be small enough to avoid traversing too much upward curvature. we typically use learning rates that 413


chapter 10. sequence modeling : recurrent and recursive nets decay slowly enough that consecutive steps have approximately the same learning rate. a step size that is appropriate for a relatively linear part of the landscape is often inappropriate and causes uphill motion if we enter a more curved part of the landscape on the next step. figure 10. 17 : example of the [UNK] of gradient clipping in a recurrent network with two parameters w and b. gradient clipping can make gradient descent perform more reasonably in the vicinity of extremely steep [UNK]. these steep [UNK] commonly occur in recurrent networks near where a recurrent network behaves approximately linearly. the [UNK] exponentially steep in the number of time steps because the weight matrix is multiplied by itself once for each time step. ( left ) gradient descent without gradient clipping overshoots the bottom of this small ravine, then receives a very large gradient from the [UNK]. the large gradient catastrophically propels the parameters outside the axes of the plot. gradient descent with gradient clipping has a more moderate ( right ) reaction to the [UNK]. while it does ascend the [UNK], the step size is restricted so that it cannot be propelled away from steep region near the solution. figure adapted with permission from pascanu 2013 et al. ( )


moderate ( right ) reaction to the [UNK]. while it does ascend the [UNK], the step size is restricted so that it cannot be propelled away from steep region near the solution. figure adapted with permission from pascanu 2013 et al. ( ). a simple type of solution has been in use by practitioners for many years : clipping the gradient. there are [UNK] instances of this idea ( mikolov 2012, ; pascanu 2013 et al., ). one option is to clip the parameter gradient from a minibatch element - wise ( mikolov 2012, ) just before the parameter update. another is to clip the norm | | | | g of the gradient g ( pascanu 2013 et al., ) just before the parameter update : if | | | | g > v ( 10. 48 ) g ←gv | | | | g ( 10. 49 ) 414


chapter 10. sequence modeling : recurrent and recursive nets where v is the norm threshold and g is used to update parameters. because the gradient of all the parameters ( including [UNK] groups of parameters, such as weights and biases ) is renormalized jointly with a single scaling factor, the latter method has the advantage that it guarantees that each step is still in the gradient direction, but experiments suggest that both forms work similarly. although the parameter update has the same direction as the true gradient, with gradient norm clipping, the parameter update vector norm is now bounded. this bounded gradient avoids performing a detrimental step when the gradient explodes. in fact, even simply taking a random step when the gradient magnitude is above a threshold tends to work almost as well. if the explosion is so severe that the gradient is numerically inf or nan ( considered inﬁnite or not - a - number ), then a random step of size v can be taken and will typically move away from the numerically unstable conﬁguration. clipping the gradient norm per - minibatch will not change the direction of the gradient for an individual minibatch. however, taking the average of the norm - clipped gradient from many minibatches is not equivalent to clipping the norm


. clipping the gradient norm per - minibatch will not change the direction of the gradient for an individual minibatch. however, taking the average of the norm - clipped gradient from many minibatches is not equivalent to clipping the norm of the true gradient ( the gradient formed from using all examples ). examples that have large gradient norm, as well as examples that appear in the same minibatch as such examples, will have their contribution to the ﬁnal direction diminished. this stands in contrast to traditional minibatch gradient descent, where the true gradient direction is equal to the average over all minibatch gradients. put another way, traditional stochastic gradient descent uses an unbiased estimate of the gradient, while gradient descent with norm clipping introduces a heuristic bias that we know empirically to be useful. with element - wise clipping, the direction of the update is not aligned with the true gradient or the minibatch gradient, but it is still a descent direction. it has also been proposed ( graves 2013, ) to clip the back - propagated gradient ( with respect to hidden units ) but no comparison has been published between these variants ; we conjecture that all these methods behave similarly. 10. 11. 2 regularizing


also been proposed ( graves 2013, ) to clip the back - propagated gradient ( with respect to hidden units ) but no comparison has been published between these variants ; we conjecture that all these methods behave similarly. 10. 11. 2 regularizing to encourage information flow gradient clipping helps to deal with exploding gradients, but it does not help with vanishing gradients. to address vanishing gradients and better capture long - term dependencies, we discussed the idea of creating paths in the computational graph of the unfolded recurrent architecture along which the product of gradients associated with arcs is near 1. one approach to achieve this is with lstms and other self - loops and gating mechanisms, described above in section. another idea is 10. 10 to regularize or constrain the parameters so as to encourage “ information ﬂow. ” in particular, we would like the gradient vector ∇h ( ) t l being back - propagated to 415


chapter 10. sequence modeling : recurrent and recursive nets maintain its magnitude, even if the loss function only penalizes the output at the end of the sequence. formally, we want ( ∇h ( ) t l ) ∂h ( ) t ∂h ( 1 ) t− ( 10. 50 ) to be as large as ∇h ( ) t l. ( 10. 51 ) with this objective, pascanu 2013 et al. ( ) propose the following regularizer : ω = t | ∇ ( h ( ) t l ) ∂h ( ) t ∂h ( 1 ) t− | | | ∇h ( ) t l | | −1 2. ( 10. 52 ) computing the gradient of this regularizer may appear [UNK], but pascanu et al. ( ) propose an approximation in which we consider the back - propagated 2013 vectors ∇h ( ) t l as if they were constants ( for the purpose of this regularizer, so that there is no need to back - propagate through them ). the experiments with this regularizer suggest that, if combined with the norm clipping heuristic ( which handles gradient explosion ), the regularizer can considerably increase the span of the dependencies that an rn


back - propagate through them ). the experiments with this regularizer suggest that, if combined with the norm clipping heuristic ( which handles gradient explosion ), the regularizer can considerably increase the span of the dependencies that an rnn can learn. because it keeps the rnn dynamics on the edge of explosive gradients, the gradient clipping is particularly important. without gradient clipping, gradient explosion prevents learning from succeeding. a key weakness of this approach is that it is not as [UNK] as the lstm for tasks where data is abundant, such as language modeling. 10. 12 explicit memory intelligence requires knowledge and acquiring knowledge can be done via learning, which has motivated the development of large - scale deep architectures. however, there are [UNK] kinds of knowledge. some knowledge can be implicit, sub - conscious, and [UNK] to verbalize — such as how to walk, or how a dog looks [UNK] from a cat. other knowledge can be explicit, declarative, and relatively straightforward to put into words — every day commonsense knowledge, like “ a cat is a kind of animal, ” or very speciﬁc facts that you need to know to accomplish your current goals, like “ the meeting with the sales team is at 3 : 00


words — every day commonsense knowledge, like “ a cat is a kind of animal, ” or very speciﬁc facts that you need to know to accomplish your current goals, like “ the meeting with the sales team is at 3 : 00 pm in room 141. ” neural networks excel at storing implicit knowledge. however, they struggle to memorize facts. stochastic gradient descent requires many presentations of the 416


chapter 10. sequence modeling : recurrent and recursive nets task network, controlling the memory memory cells writing mechanism reading mechanism figure 10. 18 : a schematic of an example of a network with an explicit memory, capturing some of the key design elements of the neural turing machine. in this diagram we distinguish the “ representation ” part of the model ( the “ task network, ” here a recurrent net in the bottom ) from the “ memory ” part of the model ( the set of cells ), which can store facts. the task network learns to “ control ” the memory, deciding where to read from and where to write to within the memory ( through the reading and writing mechanisms, indicated by bold arrows pointing at the reading and writing addresses ). 417


chapter 10. sequence modeling : recurrent and recursive nets same input before it can be stored in a neural network parameters, and even then, that input will not be stored especially precisely. graves 2014b et al. ( ) hypothesized that this is because neural networks lack the equivalent of the working memory system that allows human beings to explicitly hold and manipulate pieces of information that are relevant to achieving some goal. such explicit memory components would allow our systems not only to rapidly and “ intentionally ” store and retrieve speciﬁc facts but also to sequentially reason with them. the need for neural networks that can process information in a sequence of steps, changing the way the input is fed into the network at each step, has long been recognized as important for the ability to reason rather than to make automatic, intuitive responses to the input (, ). hinton 1990 to resolve this [UNK], weston 2014 et al. ( ) introduced memory networks that include a set of memory cells that can be accessed via an addressing mecha - nism. memory networks originally required a supervision signal instructing them how to use their memory cells. graves 2014b et al. ( ) introduced the neural turing machine, which is able to learn to read from and write arbitrary content to