a carefully chosen scaling or gain factor g that accounts for the nonlinearity applied at each layer. they derive speciﬁc values of the scaling factor for [UNK] types of nonlinear activation functions. this initialization scheme is also motivated by a model of a deep network as a sequence of matrix multiplies without nonlinearities. under such a model, this initialization scheme guarantees that the total number of training iterations required to reach convergence is independent of depth. increasing the scaling factor g pushes the network toward the regime where activations increase in norm as they propagate forward through the network and gradients increase in norm as they propagate backward. ( ) showed sussillo 2014 that setting the gain factor correctly is [UNK] to train networks as deep as 303


chapter 8. optimization for training deep models 1, 000 layers, without needing to use orthogonal initializations. a key insight of this approach is that in feedforward networks, activations and gradients can grow or shrink on each step of forward or back - propagation, following a random walk behavior. this is because feedforward networks use a [UNK] weight matrix at each layer. if this random walk is tuned to preserve norms, then feedforward networks can mostly avoid the vanishing and exploding gradients problem that arises when the same weight matrix is used at each step, described in section. 8. 2. 5 unfortunately, these optimal criteria for initial weights often do not lead to optimal performance. this may be for three [UNK] reasons. first, we may be using the wrong criteria — it may not actually be beneﬁcial to preserve the norm of a signal throughout the entire network. second, the properties imposed at initialization may not persist after learning has begun to proceed. third, the criteria might succeed at improving the speed of optimization but inadvertently increase generalization error. in practice, we usually need to treat the scale of the weights as a hyperparameter whose optimal value lies somewhere roughly near but not exactly equal to the theoretical predictions. one drawback to scaling rules that

/home/ricoiban/GEMMA/mnlp_chatsplaining/RAG/Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville (z-lib.org).pdf


of optimization but inadvertently increase generalization error. in practice, we usually need to treat the scale of the weights as a hyperparameter whose optimal value lies somewhere roughly near but not exactly equal to the theoretical predictions. one drawback to scaling rules that set all of the initial weights to have the same standard deviation, such as 1 √m, is that every individual weight becomes extremely small when the layers become large. ( ) introduced an martens 2010 alternative initialization scheme called sparse initialization in which each unit is initialized to have exactly k non - zero weights. the idea is to keep the total amount of input to the unit independent from the number of inputs m without making the magnitude of individual weight elements shrink with m. sparse initialization helps to achieve more diversity among the units at initialization time. however, it also imposes a very strong prior on the weights that are chosen to have large gaussian values. because it takes a long time for gradient descent to shrink “ incorrect ” large values, this initialization scheme can cause problems for units such as maxout units that have several ﬁlters that must be carefully coordinated with each other. when computational resources allow it, it is usually a good idea to treat the initial scale of the weights for each layer as


scheme can cause problems for units such as maxout units that have several ﬁlters that must be carefully coordinated with each other. when computational resources allow it, it is usually a good idea to treat the initial scale of the weights for each layer as a hyperparameter, and to choose these scales using a hyperparameter search algorithm described in section, such 11. 4. 2 as random search. the choice of whether to use dense or sparse initialization can also be made a hyperparameter. alternately, one can manually search for the best initial scales. a good rule of thumb for choosing the initial scales is to look at the range or standard deviation of activations or gradients on a single minibatch of data. if the weights are too small, the range of activations across the minibatch will shrink as the activations propagate forward through the network. by repeatedly identifying the ﬁrst layer with unacceptably small activations and 304


chapter 8. optimization for training deep models increasing its weights, it is possible to eventually obtain a network with reasonable initial activations throughout. if learning is still too slow at this point, it can be useful to look at the range or standard deviation of the gradients as well as the activations. this procedure can in principle be automated and is generally less computationally costly than hyperparameter optimization based on validation set error because it is based on feedback from the behavior of the initial model on a single batch of data, rather than on feedback from a trained model on the validation set. while long used heuristically, this protocol has recently been speciﬁed more formally and studied by ( ). mishkin and matas 2015 so far we have focused on the initialization of the weights. fortunately, initialization of other parameters is typically easier. the approach for setting the biases must be coordinated with the approach for settings the weights. setting the biases to zero is compatible with most weight initialization schemes. there are a few situations where we may set some biases to non - zero values : • if a bias is for an output unit, then it is often beneﬁcial to initialize the bias to obtain the right marginal statistics of the output


there are a few situations where we may set some biases to non - zero values : • if a bias is for an output unit, then it is often beneﬁcial to initialize the bias to obtain the right marginal statistics of the output. to do this, we assume that the initial weights are small enough that the output of the unit is determined only by the bias. this justiﬁes setting the bias to the inverse of the activation function applied to the marginal statistics of the output in the training set. for example, if the output is a distribution over classes and this distribution is a highly skewed distribution with the marginal probability of class i given by element ci of some vector c, then we can set the bias vector b by solving the equation softmax ( b ) = c. this applies not only to classiﬁers but also to models we will encounter in part, such as autoencoders and boltzmann iii machines. these models have layers whose output should resemble the input data x, and it can be very helpful to initialize the biases of such layers to match the marginal distribution over. x • sometimes we may want to choose the bias to avoid causing too much saturation at initialization. for example, we may set


, and it can be very helpful to initialize the biases of such layers to match the marginal distribution over. x • sometimes we may want to choose the bias to avoid causing too much saturation at initialization. for example, we may set the bias of a relu hidden unit to 0. 1 rather than 0 to avoid saturating the relu at initialization. this approach is not compatible with weight initialization schemes that do not expect strong input from the biases though. for example, it is not recommended for use with random walk initialization (, ). sussillo 2014 • sometimes a unit controls whether other units are able to participate in a function. in such situations, we have a unit with output u and another unit h ∈ [ 0, 1 ], and they are multiplied together to produce an output uh. we 305


chapter 8. optimization for training deep models can view h as a gate that determines whether uh u ≈ or uh ≈0. in these situations, we want to set the bias for h so that h ≈1 most of the time at initialization. otherwise u does not have a chance to learn. for example, jozefowicz 2015 et al. ( ) advocate setting the bias to for the forget gate of 1 the lstm model, described in section. 10. 10 another common type of parameter is a variance or precision parameter. for example, we can perform linear regression with a conditional variance estimate using the model p y y ( | n x ) = ( | wt x + 1 ) b, / β ( 8. 24 ) where β is a precision parameter. we can usually initialize variance or precision parameters to 1 safely. another approach is to assume the initial weights are close enough to zero that the biases may be set while ignoring the [UNK] of the weights, then set the biases to produce the correct marginal mean of the output, and set the variance parameters to the marginal variance of the output in the training set. besides these simple constant or random methods of initializing model parame - ters, it is possible to initialize model parameters using


correct marginal mean of the output, and set the variance parameters to the marginal variance of the output in the training set. besides these simple constant or random methods of initializing model parame - ters, it is possible to initialize model parameters using machine learning. a common strategy discussed in part of this book is to initialize a supervised model with iii the parameters learned by an unsupervised model trained on the same inputs. one can also perform supervised training on a related task. even performing supervised training on an unrelated task can sometimes yield an initialization that [UNK] faster convergence than a random initialization. some of these initialization strategies may yield faster convergence and better generalization because they encode information about the distribution in the initial parameters of the model. others apparently perform well primarily because they set the parameters to have the right scale or set [UNK] units to compute [UNK] functions from each other. 8. 5 algorithms with adaptive learning rates neural network researchers have long realized that the learning rate was reliably one of the hyperparameters that is the most [UNK] to set because it has a signiﬁcant impact on model performance. as we have discussed in sections and, the 4. 3 8. 2 cost is often highly sensitive to some directions in parameter space


##parameters that is the most [UNK] to set because it has a signiﬁcant impact on model performance. as we have discussed in sections and, the 4. 3 8. 2 cost is often highly sensitive to some directions in parameter space and insensitive to others. the momentum algorithm can mitigate these issues somewhat, but does so at the expense of introducing another hyperparameter. in the face of this, it is natural to ask if there is another way. if we believe that the directions of sensitivity are somewhat axis - aligned, it can make sense to use a separate learning 306


chapter 8. optimization for training deep models rate for each parameter, and automatically adapt these learning rates throughout the course of learning. the algorithm (, ) is an early heuristic approach delta - bar - delta jacobs 1988 to adapting individual learning rates for model parameters during training. the approach is based on a simple idea : if the partial derivative of the loss, with respect to a given model parameter, remains the same sign, then the learning rate should increase. if the partial derivative with respect to that parameter changes sign, then the learning rate should decrease. of course, this kind of rule can only be applied to full batch optimization. more recently, a number of incremental ( or mini - batch - based ) methods have been introduced that adapt the learning rates of model parameters. this section will brieﬂy review a few of these algorithms. 8. 5. 1 adagrad the adagrad algorithm, shown in algorithm, individually adapts the learning 8. 4 rates of all model parameters by scaling them inversely proportional to the square root of the sum of all of their historical squared values (, ). the duchi et al. 2011 parameters with the largest partial derivative of the loss have a correspondingly rapid decrease in their learning rate, while parameters with small partial


to the square root of the sum of all of their historical squared values (, ). the duchi et al. 2011 parameters with the largest partial derivative of the loss have a correspondingly rapid decrease in their learning rate, while parameters with small partial derivatives have a relatively small decrease in their learning rate. the net [UNK] is greater progress in the more gently sloped directions of parameter space. in the context of convex optimization, the adagrad algorithm enjoys some desirable theoretical properties. however, empirically it has been found that — for training deep neural network models — the accumulation of squared gradients from the beginning of training can result in a premature and excessive decrease in the [UNK] learning rate. adagrad performs well for some but not all deep learning models. 8. 5. 2 rmsprop the rmsprop algorithm (, ) modiﬁes adagrad to perform better in hinton 2012 the non - convex setting by changing the gradient accumulation into an exponentially weighted moving average. adagrad is designed to converge rapidly when applied to a convex function. when applied to a non - convex function to train a neural network, the learning trajectory may pass through many [UNK] structures and eventually arrive at a region that is a locally convex bowl. adagrad shrinks the learning rate according to


to a convex function. when applied to a non - convex function to train a neural network, the learning trajectory may pass through many [UNK] structures and eventually arrive at a region that is a locally convex bowl. adagrad shrinks the learning rate according to the entire history of the squared gradient and may 307


chapter 8. optimization for training deep models algorithm 8. 4 the adagrad algorithm require : global learning rate require : initial parameter θ require : small constant, perhaps δ 10−7, for numerical stability initialize gradient accumulation variable r = 0 while do stopping criterion not met sample a minibatch of m examples from the training set { x ( 1 ),..., x ( ) m } with corresponding targets y ( ) i. compute gradient : g ←1 m∇θ i l f ( ( x ( ) i ; ) θ, y ( ) i ) accumulate squared gradient : r r g g ← + compute update : ∆θ ←− δ + √r g. ( division and square root applied element - wise ) apply update : θ θ θ ← + ∆ end while have made the learning rate too small before arriving at such a convex structure. rmsprop uses an exponentially decaying average to discard history from the extreme past so that it can converge rapidly after ﬁnding a convex bowl, as if it were an instance of the adagrad algorithm initialized within that bowl. rmsprop is shown in its standard form in algorithm and combined with 8. 5 nesterov momentum in algorithm. compared to adagrad, the use of the


as if it were an instance of the adagrad algorithm initialized within that bowl. rmsprop is shown in its standard form in algorithm and combined with 8. 5 nesterov momentum in algorithm. compared to adagrad, the use of the 8. 6 moving average introduces a new hyperparameter, ρ, that controls the length scale of the moving average. empirically, rmsprop has been shown to be an [UNK] and practical op - timization algorithm for deep neural networks. it is currently one of the go - to optimization methods being employed routinely by deep learning practitioners. 8. 5. 3 adam adam (, ) is yet another adaptive learning rate optimization kingma and ba 2014 algorithm and is presented in algorithm. the name “ adam ” derives from 8. 7 the phrase “ adaptive moments. ” in the context of the earlier algorithms, it is perhaps best seen as a variant on the combination of rmsprop and momentum with a few important distinctions. first, in adam, momentum is incorporated directly as an estimate of the ﬁrst order moment ( with exponential weighting ) of the gradient. the most straightforward way to add momentum to rmsprop is to apply momentum to the rescaled gradients. the use of momentum in combination with rescaling


estimate of the ﬁrst order moment ( with exponential weighting ) of the gradient. the most straightforward way to add momentum to rmsprop is to apply momentum to the rescaled gradients. the use of momentum in combination with rescaling does not have a clear theoretical motivation. second, adam includes 308


chapter 8. optimization for training deep models algorithm 8. 5 the rmsprop algorithm require : global learning rate, decay rate. ρ require : initial parameter θ require : small constant δ, usually 10−6, used to stabilize division by small numbers. initialize accumulation variables r = 0 while do stopping criterion not met sample a minibatch of m examples from the training set { x ( 1 ),..., x ( ) m } with corresponding targets y ( ) i. compute gradient : g ←1 m∇θ i l f ( ( x ( ) i ; ) θ, y ( ) i ) accumulate squared gradient : r r g g ←ρ + ( 1 ) −ρ compute parameter update : ∆θ = − √ δ + r g. ( 1 √ δ + r applied element - wise ) apply update : θ θ θ ← + ∆ end while bias corrections to the estimates of both the ﬁrst - order moments ( the momentum term ) and the ( uncentered ) second - order moments to account for their initialization at the origin ( see algorithm ). rmsprop also incorporates an estimate of the 8. 7 ( uncentered ) second - order moment, however it lacks the correction factor. thus, unlike in adam,


moments to account for their initialization at the origin ( see algorithm ). rmsprop also incorporates an estimate of the 8. 7 ( uncentered ) second - order moment, however it lacks the correction factor. thus, unlike in adam, the rmsprop second - order moment estimate may have high bias early in training. adam is generally regarded as being fairly robust to the choice of hyperparameters, though the learning rate sometimes needs to be changed from the suggested default. 8. 5. 4 choosing the right optimization algorithm in this section, we discussed a series of related algorithms that each seek to address the challenge of optimizing deep models by adapting the learning rate for each model parameter. at this point, a natural question is : which algorithm should one choose? unfortunately, there is currently no consensus on this point. ( ) schaul et al. 2014 presented a valuable comparison of a large number of optimization algorithms across a wide range of learning tasks. while the results suggest that the family of algorithms with adaptive learning rates ( represented by rmsprop and adadelta ) performed fairly robustly, no single best algorithm has emerged. currently, the most popular optimization algorithms actively in use include sgd, sgd with momentum, rmsprop, rms


adaptive learning rates ( represented by rmsprop and adadelta ) performed fairly robustly, no single best algorithm has emerged. currently, the most popular optimization algorithms actively in use include sgd, sgd with momentum, rmsprop, rmsprop with momentum, adadelta and adam. the choice of which algorithm to use, at this point, seems to depend 309


chapter 8. optimization for training deep models algorithm 8. 6 rmsprop algorithm with nesterov momentum require : global learning rate, decay rate, momentum [UNK]. ρ α require : initial parameter, initial velocity. θ v initialize accumulation variable r = 0 while do stopping criterion not met sample a minibatch of m examples from the training set { x ( 1 ),..., x ( ) m } with corresponding targets y ( ) i. compute interim update : [UNK] θ v ← + α compute gradient : g ←1 [UNK] i l f ( ( x ( ) i ; [UNK] y ), ( ) i ) accumulate gradient : r r g g ←ρ + ( 1 ) −ρ compute velocity update : v v ←α − √r g. ( 1 √r applied element - wise ) apply update : θ θ v ← + end while largely on the user ’ s familiarity with the algorithm ( for ease of hyperparameter tuning ). 8. 6 approximate second - order methods in this section we discuss the application of second - order methods to the training of deep networks. see ( ) for an earlier treatment of this subject. lecun et al. 1998a for simplicity of exposition, the only objective function we examine is the empirical risk


we discuss the application of second - order methods to the training of deep networks. see ( ) for an earlier treatment of this subject. lecun et al. 1998a for simplicity of exposition, the only objective function we examine is the empirical risk : j ( ) = θ ex, [UNK] ( ) x, y [ ( ( ; ) ) ] = l f x θ, y 1 m m i = 1 l f ( ( x ( ) i ; ) θ, y ( ) i ). ( 8. 25 ) however the methods we discuss here extend readily to more general objective functions that, for instance, include parameter regularization terms such as those discussed in chapter. 7 8. 6. 1 newton ’ s method in section, we introduced second - order gradient methods. in contrast to ﬁrst - 4. 3 order methods, second - order methods make use of second derivatives to improve optimization. the most widely used second - order method is newton ’ s method. we now describe newton ’ s method in more detail, with emphasis on its application to neural network training. 310


chapter 8. optimization for training deep models algorithm 8. 7 the adam algorithm require : step size ( suggested default : ) 0 001. require : exponential decay rates for moment estimates, ρ1 and ρ2 in [ 0, 1 ). ( suggested defaults : and respectively ) 0 9. 0 999. require : small constant δ used for numerical stabilization. ( suggested default : 10−8 ) require : initial parameters θ initialize 1st and 2nd moment variables, s = 0 r = 0 initialize time step t = 0 while do stopping criterion not met sample a minibatch of m examples from the training set { x ( 1 ),..., x ( ) m } with corresponding targets y ( ) i. compute gradient : g ←1 m∇θ i l f ( ( x ( ) i ; ) θ, y ( ) i ) t t ← + 1 update biased ﬁrst moment estimate : s ←ρ1s + ( 1 −ρ1 ) g update biased second moment estimate : r ←ρ2r + ( 1 −ρ2 ) g g correct bias in ﬁrst moment : [UNK] ← s 1−ρt 1 correct bias in second moment : [UNK] ← r 1−ρt 2 compute update : ∆ = θ − [UNK]


←ρ2r + ( 1 −ρ2 ) g g correct bias in ﬁrst moment : [UNK] ← s 1−ρt 1 correct bias in second moment : [UNK] ← r 1−ρt 2 compute update : ∆ = θ − [UNK] √ [UNK] + δ ( operations applied element - wise ) apply update : θ θ θ ← + ∆ end while newton ’ s method is an optimization scheme based on using a second - order tay - lor series expansion to approximate j ( θ ) near some point θ 0, ignoring derivatives of higher order : j j ( ) θ ≈ ( θ0 ) + ( θ θ − 0 ) ∇θj ( θ0 ) + 1 2 ( θ θ − 0 ) h θ θ ( − 0 ), ( 8. 26 ) where h is the hessian of j with respect to θ evaluated at θ0. if we then solve for the critical point of this function, we obtain the newton parameter update rule : θ∗ = θ0 −h−1∇θj ( θ 0 ) ( 8. 27 ) thus for a locally quadratic function ( with positive deﬁnite h ), by rescaling the gradient by h −1, newton ’ s method jumps directly to the minimum. if the objective function


θ 0 ) ( 8. 27 ) thus for a locally quadratic function ( with positive deﬁnite h ), by rescaling the gradient by h −1, newton ’ s method jumps directly to the minimum. if the objective function is convex but not quadratic ( there are higher - order terms ), this update can be iterated, yielding the training algorithm associated with newton ’ s method, given in algorithm. 8. 8 311


chapter 8. optimization for training deep models algorithm 8. 8 newton ’ s method with objective j ( θ ) = 1 m m i = 1 l f ( ( x ( ) i ; ) θ, y ( ) i ). require : initial parameter θ0 require : training set of examples m while do stopping criterion not met compute gradient : g ←1 m∇θ i l f ( ( x ( ) i ; ) θ, y ( ) i ) compute hessian : h ←1 m ∇2 θ i l f ( ( x ( ) i ; ) θ, y ( ) i ) compute hessian inverse : h−1 compute update : ∆ = θ −h−1 g apply update : θ θ θ = + ∆ end while for surfaces that are not quadratic, as long as the hessian remains positive deﬁnite, newton ’ s method can be applied iteratively. this implies a two - step iterative procedure. first, update or compute the inverse hessian ( i. e. by updat - ing the quadratic approximation ). second, update the parameters according to equation. 8. 27 in section, we discussed how newton ’ s method is appropriate only when 8. 2. 3 the hessian is positive deﬁ


by updat - ing the quadratic approximation ). second, update the parameters according to equation. 8. 27 in section, we discussed how newton ’ s method is appropriate only when 8. 2. 3 the hessian is positive deﬁnite. in deep learning, the surface of the objective function is typically non - convex with many features, such as saddle points, that are problematic for newton ’ s method. if the eigenvalues of the hessian are not all positive, for example, near a saddle point, then newton ’ s method can actually cause updates to move in the wrong direction. this situation can be avoided by regularizing the hessian. common regularization strategies include adding a constant,, along the diagonal of the hessian. the regularized update becomes α θ∗ = θ0 − [ ( ( h f θ0 ) ) + ] αi −1 ∇θ f ( θ0 ). ( 8. 28 ) this regularization strategy is used in approximations to newton ’ s method, such as the levenberg – marquardt algorithm ( levenberg 1944 marquardt 1963, ;, ), and works fairly well as long as the negative eigenvalues of the hessian are still relatively close


’ s method, such as the levenberg – marquardt algorithm ( levenberg 1944 marquardt 1963, ;, ), and works fairly well as long as the negative eigenvalues of the hessian are still relatively close to zero. in cases where there are more extreme directions of curvature, the value of α would have to be [UNK] large to [UNK] the negative eigenvalues. however, as α increases in size, the hessian becomes dominated by the αi diagonal and the direction chosen by newton ’ s method converges to the standard gradient divided by α. when strong negative curvature is present, α may need to be so large that newton ’ s method would make smaller steps than gradient descent with a properly chosen learning rate. beyond the challenges created by certain features of the objective function, 312


chapter 8. optimization for training deep models such as saddle points, the application of newton ’ s method for training large neural networks is limited by the signiﬁcant computational burden it imposes. the number of elements in the hessian is squared in the number of parameters, so with k parameters ( and for even very small neural networks the number of parameters k can be in the millions ), newton ’ s method would require the inversion of a k k × matrix — with computational complexity of o ( k3 ). also, since the parameters will change with every update, the inverse hessian has to be computed at every training iteration. as a consequence, only networks with a very small number of parameters can be practically trained via newton ’ s method. in the remainder of this section, we will discuss alternatives that attempt to gain some of the advantages of newton ’ s method while side - stepping the computational hurdles. 8. 6. 2 conjugate gradients conjugate gradients is a method to [UNK] avoid the calculation of the inverse hessian by iteratively descending conjugate directions. the inspiration for this approach follows from a careful study of the weakness of the method of steepest descent ( see section for details ), where line searches are applied iterative


calculation of the inverse hessian by iteratively descending conjugate directions. the inspiration for this approach follows from a careful study of the weakness of the method of steepest descent ( see section for details ), where line searches are applied iteratively in 4. 3 the direction associated with the gradient. figure illustrates how the method of 8. 6 steepest descent, when applied in a quadratic bowl, progresses in a rather [UNK] back - and - forth, zig - zag pattern. this happens because each line search direction, when given by the gradient, is guaranteed to be orthogonal to the previous line search direction. let the previous search direction be dt−1. at the minimum, where the line search terminates, the directional derivative is zero in direction dt−1 : ∇θj ( θ ) · dt−1 = 0. since the gradient at this point deﬁnes the current search direction, dt = ∇θj ( θ ) will have no contribution in the direction dt−1. thus dt is orthogonal to dt−1. this relationship between dt−1 and d t is illustrated in ﬁgure for 8. 6 multiple iterations of steepest descent. as demonstrated in the ﬁgure, the choice of orthogonal directions of descent do not preserve the minimum along the


##−1. this relationship between dt−1 and d t is illustrated in ﬁgure for 8. 6 multiple iterations of steepest descent. as demonstrated in the ﬁgure, the choice of orthogonal directions of descent do not preserve the minimum along the previous search directions. this gives rise to the zig - zag pattern of progress, where by descending to the minimum in the current gradient direction, we must re - minimize the objective in the previous gradient direction. thus, by following the gradient at the end of each line search we are, in a sense, undoing progress we have already made in the direction of the previous line search. the method of conjugate gradients seeks to address this problem. in the method of conjugate gradients, we seek to ﬁnd a search direction that is conjugate to the previous line search direction, i. e. it will not undo progress made in that direction. at training iteration t, the next search direction dt takes 313


chapter 8. optimization for training deep models figure 8. 6 : the method of steepest descent applied to a quadratic cost surface. the method of steepest descent involves jumping to the point of lowest cost along the line deﬁned by the gradient at the initial point on each step. this resolves some of the problems seen with using a ﬁxed learning rate in ﬁgure, but even with the optimal step size 4. 6 the algorithm still makes back - and - forth progress toward the optimum. by deﬁnition, at the minimum of the objective along a given direction, the gradient at the ﬁnal point is orthogonal to that direction. the form : dt = ∇θj β ( ) + θ tdt−1 ( 8. 29 ) where βt is a [UNK] whose magnitude controls how much of the direction, dt−1, we should add back to the current search direction. two directions, dt and dt−1, are deﬁned as conjugate if d t hdt−1 = 0, where h is the hessian matrix. the straightforward way to impose conjugacy would involve calculation of the eigenvectors of h to choose βt, which would not satisfy our goal of developing a method that is more computationally viable


, where h is the hessian matrix. the straightforward way to impose conjugacy would involve calculation of the eigenvectors of h to choose βt, which would not satisfy our goal of developing a method that is more computationally viable than newton ’ s method for large problems. can we calculate the conjugate directions without resorting to these calculations? fortunately the answer to that is yes. two popular methods for computing the βt are : 1. fletcher - reeves : βt = ∇θj ( θt ) ∇θj ( θt ) ∇θj ( θt−1 ) ∇θj ( θt−1 ) ( 8. 30 ) 314


chapter 8. optimization for training deep models 2. polak - ribiere : βt = ( ∇θj ( θt ) −∇θj ( θt−1 ) ) ∇θj ( θt ) ∇θj ( θt−1 ) ∇θj ( θt−1 ) ( 8. 31 ) for a quadratic surface, the conjugate directions ensure that the gradient along the previous direction does not increase in magnitude. we therefore stay at the minimum along the previous directions. as a consequence, in a k - dimensional parameter space, the conjugate gradient method requires at most k line searches to achieve the minimum. the conjugate gradient algorithm is given in algorithm. 8. 9 algorithm 8. 9 the conjugate gradient method require : initial parameters θ0 require : training set of examples m initialize ρ0 = 0 initialize g0 = 0 initialize t = 1 while do stopping criterion not met initialize the gradient gt = 0 compute gradient : gt ←1 m∇θ i l f ( ( x ( ) i ; ) θ, y ( ) i ) compute βt = ( g t−gt−1 ) gt g t−1gt−1 ( polak - ribiere ) ( nonlinear conjugate gradient : optional


i l f ( ( x ( ) i ; ) θ, y ( ) i ) compute βt = ( g t−gt−1 ) gt g t−1gt−1 ( polak - ribiere ) ( nonlinear conjugate gradient : optionally reset βt to zero, for example if t is a multiple of some constant, such as ) k k = 5 compute search direction : ρt = −gt + βtρt−1 perform line search to ﬁnd : ∗ = argmin1 m m i = 1 l f ( ( x ( ) i ; θt + ρt ), y ( ) i ) ( on a truly quadratic cost function, analytically solve for ∗rather than explicitly searching for it ) apply update : θt + 1 = θt + ∗ρt t t ← + 1 end while nonlinear conjugate gradients : so far we have discussed the method of conjugate gradients as it is applied to quadratic objective functions. of course, our primary interest in this chapter is to explore optimization methods for training neural networks and other related deep learning models where the corresponding objective function is far from quadratic. perhaps surprisingly, the method of conjugate gradients is still applicable in this setting, though


primary interest in this chapter is to explore optimization methods for training neural networks and other related deep learning models where the corresponding objective function is far from quadratic. perhaps surprisingly, the method of conjugate gradients is still applicable in this setting, though with some modiﬁcation. without any assurance that the objective is quadratic, the conjugate directions 315


chapter 8. optimization for training deep models are no longer assured to remain at the minimum of the objective for previous directions. as a result, the nonlinear conjugate gradients algorithm includes occasional resets where the method of conjugate gradients is restarted with line search along the unaltered gradient. practitioners report reasonable results in applications of the nonlinear conjugate gradients algorithm to training neural networks, though it is often beneﬁcial to initialize the optimization with a few iterations of stochastic gradient descent before commencing nonlinear conjugate gradients. also, while the ( nonlinear ) conjugate gradients algorithm has traditionally been cast as a batch method, minibatch versions have been used successfully for the training of neural networks (, le et al. 2011 ). adaptations of conjugate gradients speciﬁcally for neural networks have been proposed earlier, such as the scaled conjugate gradients algorithm (, moller 1993 ). 8. 6. 3 bfgs the broyden – fletcher – goldfarb – shanno ( bfgs ) algorithm attempts to bring some of the advantages of newton ’ s method without the computational burden. in that respect, bfgs is similar to the conjugate gradient method. however,


##n – fletcher – goldfarb – shanno ( bfgs ) algorithm attempts to bring some of the advantages of newton ’ s method without the computational burden. in that respect, bfgs is similar to the conjugate gradient method. however, bfgs takes a more direct approach to the approximation of newton ’ s update. recall that newton ’ s update is given by θ∗ = θ0 −h−1∇θj ( θ0 ), ( 8. 32 ) where h is the hessian of j with respect to θ evaluated at θ0. the primary computational [UNK] in applying newton ’ s update is the calculation of the inverse hessian h −1. the approach adopted by quasi - newton methods ( of which the bfgs algorithm is the most prominent ) is to approximate the inverse with a matrix mt that is iteratively reﬁned by low rank updates to become a better approximation of h−1. the speciﬁcation and derivation of the bfgs approximation is given in many textbooks on optimization, including luenberger 1984 ( ). once the inverse hessian approximation mt is updated, the direction of descent ρt is determined by ρt = mtgt. a line search is performed in this direction to determine the size of the step, ∗,


luenberger 1984 ( ). once the inverse hessian approximation mt is updated, the direction of descent ρt is determined by ρt = mtgt. a line search is performed in this direction to determine the size of the step, ∗, taken in this direction. the ﬁnal update to the parameters is given by : θt + 1 = θt + ∗ρt. ( 8. 33 ) like the method of conjugate gradients, the bfgs algorithm iterates a series of line searches with the direction incorporating second - order information. however 316


chapter 8. optimization for training deep models unlike conjugate gradients, the success of the approach is not heavily dependent on the line search ﬁnding a point very close to the true minimum along the line. thus, relative to conjugate gradients, bfgs has the advantage that it can spend less time reﬁning each line search. on the other hand, the bfgs algorithm must store the inverse hessian matrix, m, that requires o ( n2 ) memory, making bfgs impractical for most modern deep learning models that typically have millions of parameters. limited memory bfgs ( or l - bfgs ) the memory costs of the bfgs algorithm can be signiﬁcantly decreased by avoiding storing the complete inverse hessian approximationm. the l - bfgs algorithm computes the approximation m using the same method as the bfgs algorithm, but beginning with the assumption that m ( 1 ) t− is the identity matrix, rather than storing the approximation from one step to the next. if used with exact line searches, the directions deﬁned by l - bfgs are mutually conjugate. however, unlike the method of conjugate gradients, this procedure remains well behaved when the minimum of the line search is reached only approximately


used with exact line searches, the directions deﬁned by l - bfgs are mutually conjugate. however, unlike the method of conjugate gradients, this procedure remains well behaved when the minimum of the line search is reached only approximately. the l - bfgs strategy with no storage described here can be generalized to include more information about the hessian by storing some of the vectors used to update at each time step, which costs only per step. m o n ( ) 8. 7 optimization strategies and meta - algorithms many optimization techniques are not exactly algorithms, but rather general templates that can be specialized to yield algorithms, or subroutines that can be incorporated into many [UNK] algorithms. 8. 7. 1 batch normalization batch normalization (, ) is one of the most exciting recent [UNK] and szegedy 2015 innovations in optimizing deep neural networks and it is actually not an optimization algorithm at all. instead, it is a method of adaptive reparametrization, motivated by the [UNK] of training very deep models. very deep models involve the composition of several functions or layers. the gradient tells how to update each parameter, under the assumption that the other layers do not change. in practice, we update all of the layers simultaneously. when


of training very deep models. very deep models involve the composition of several functions or layers. the gradient tells how to update each parameter, under the assumption that the other layers do not change. in practice, we update all of the layers simultaneously. when we make the update, unexpected results can happen because many functions composed together are changed simultaneously, using updates that were computed under the assumption that the other functions remain constant. as a simple 317


chapter 8. optimization for training deep models example, suppose we have a deep neural network that has only one unit per layer and does not use an activation function at each hidden layer : [UNK] = xw1w2w3... wl. here, wi provides the weight used by layer i. the output of layer i is hi = hi−1wi. the output [UNK] is a linear function of the input x, but a nonlinear function of the weights wi. suppose our cost function has put a gradient of on 1 [UNK], so we wish to decrease [UNK] slightly. the back - propagation algorithm can then compute a gradient g = [UNK]. consider what happens when we make an update w w g ← −. the ﬁrst - order taylor series approximation of [UNK] predicts that the value of [UNK] will decrease by gg. if we wanted to decrease [UNK] by. 1, this ﬁrst - order information available in the gradient suggests we could set the learning rate to. 1 g g. however, the actual update will include second - order and third - order [UNK], on up to [UNK] of order l. the new value of [UNK] is given by x w ( 1 −g1 ) ( w2 −g 2 ) (... wl −gl )


include second - order and third - order [UNK], on up to [UNK] of order l. the new value of [UNK] is given by x w ( 1 −g1 ) ( w2 −g 2 ) (... wl −gl ). ( 8. 34 ) an example of one second - order term arising from this update is 2g1 g2 l i = 3 wi. this term might be negligible if l i = 3wi is small, or might be exponentially large if the weights on layers through 3 l are greater than. this makes it very hard 1 to choose an appropriate learning rate, because the [UNK] of an update to the parameters for one layer depends so strongly on all of the other layers. second - order optimization algorithms address this issue by computing an update that takes these second - order interactions into account, but we can see that in very deep networks, even higher - order interactions can be signiﬁcant. even second - order optimization algorithms are expensive and usually require numerous approximations that prevent them from truly accounting for all signiﬁcant second - order interactions. building an n - th order optimization algorithm for n > 2 thus seems hopeless. what can we do instead? batch normalization provides an elegant way of


approximations that prevent them from truly accounting for all signiﬁcant second - order interactions. building an n - th order optimization algorithm for n > 2 thus seems hopeless. what can we do instead? batch normalization provides an elegant way of reparametrizing almost any deep network. the reparametrization signiﬁcantly reduces the problem of coordinating updates across many layers. batch normalization can be applied to any input or hidden layer in a network. let h be a minibatch of activations of the layer to normalize, arranged as a design matrix, with the activations for each example appearing in a row of the matrix. to normalize, we replace it with h h = h µ − σ, ( 8. 35 ) where µ is a vector containing the mean of each unit and σ is a vector containing the standard deviation of each unit. the arithmetic here is based on broadcasting the vector µ and the vector σ to be applied to every row of the matrix h. within each row, the arithmetic is element - wise, so hi, j is normalized by subtracting µj 318


chapter 8. optimization for training deep models and dividing by σj. the rest of the network then operates on h in exactly the same way that the original network operated on. h at training time, µ = 1 m i hi, : ( 8. 36 ) and σ = δ + 1 m i ( ) h µ − 2 i, ( 8. 37 ) where δ is a small positive value such as 10−8 imposed to avoid encountering the undeﬁned gradient of √z at z = 0. crucially, we back - propagate through these operations for computing the mean and the standard deviation, and for applying them to normalize h. this means that the gradient will never propose an operation that acts simply to increase the standard deviation or mean of hi ; the normalization operations remove the [UNK] of such an action and zero out its component in the gradient. this was a major innovation of the batch normalization approach. previous approaches had involved adding penalties to the cost function to encourage units to have normalized activation statistics or involved intervening to renormalize unit statistics after each gradient descent step. the former approach usually resulted in imperfect normalization and the latter usually resulted in signiﬁcant wasted time as the learning algorithm repeatedly proposed changing the mean


##ized activation statistics or involved intervening to renormalize unit statistics after each gradient descent step. the former approach usually resulted in imperfect normalization and the latter usually resulted in signiﬁcant wasted time as the learning algorithm repeatedly proposed changing the mean and variance and the normalization step repeatedly undid this change. batch normalization reparametrizes the model to make some units always be standardized by deﬁnition, deftly sidestepping both problems. at test time, µ and σ may be replaced by running averages that were collected during training time. this allows the model to be evaluated on a single example, without needing to use deﬁnitions of µ and σ that depend on an entire minibatch. revisiting the [UNK] = xw1w2... wl example, we see that we can mostly resolve the [UNK] in learning this model by normalizing hl−1. suppose that x is drawn from a unit gaussian. then hl−1 will also come from a gaussian, because the transformation from x to hl is linear. however, h l−1 will no longer have zero mean and unit variance. after applying batch normalization, we obtain the normalized [UNK] that restores the zero mean and unit


gaussian, because the transformation from x to hl is linear. however, h l−1 will no longer have zero mean and unit variance. after applying batch normalization, we obtain the normalized [UNK] that restores the zero mean and unit variance properties. for almost any update to the lower layers, [UNK] will remain a unit gaussian. the output [UNK] may then be learned as a simple linear function [UNK] = [UNK] l−1. learning in this model is now very simple because the parameters at the lower layers simply do not have an [UNK] in most cases ; their output is always renormalized to a unit gaussian. in some corner cases, the lower layers can have an [UNK]. changing one of the lower layer weights to can make the output become degenerate, and changing the sign 0 319


chapter 8. optimization for training deep models of one of the lower weights can ﬂip the relationship between [UNK] and y. these situations are very rare. without normalization, nearly every update would have an extreme [UNK] on the statistics of hl−1. batch normalization has thus made this model signiﬁcantly easier to learn. in this example, the ease of learning of course came at the cost of making the lower layers useless. in our linear example, the lower layers no longer have any harmful [UNK], but they also no longer have any beneﬁcial [UNK]. this is because we have normalized out the ﬁrst and second order statistics, which is all that a linear network can inﬂuence. in a deep neural network with nonlinear activation functions, the lower layers can perform nonlinear transformations of the data, so they remain useful. batch normalization acts to standardize only the mean and variance of each unit in order to stabilize learning, but allows the relationships between units and the nonlinear statistics of a single unit to change. because the ﬁnal layer of the network is able to learn a linear transformation, we may actually wish to remove all linear relationships between units within a layer. indeed, this is the approach taken by ( ), who provided desjardin


to change. because the ﬁnal layer of the network is able to learn a linear transformation, we may actually wish to remove all linear relationships between units within a layer. indeed, this is the approach taken by ( ), who provided desjardins et al. 2015 the inspiration for batch normalization. unfortunately, eliminating all linear interactions is much more expensive than standardizing the mean and standard deviation of each individual unit, and so far batch normalization remains the most practical approach. normalizing the mean and standard deviation of a unit can reduce the expressive power of the neural network containing that unit. in order to maintain the expressive power of the network, it is common to replace the batch of hidden unit activations h with γh + β rather than simply the normalized h. the variables γ and β are learned parameters that allow the new variable to have any mean and standard deviation. at ﬁrst glance, this may seem useless — why did we set the mean to 0, and then introduce a parameter that allows it to be set back to any arbitrary value β? the answer is that the new parametrization can represent the same family of functions of the input as the old parametrization, but the new parametrization has [UNK] learning dynamics. in the old para


be set back to any arbitrary value β? the answer is that the new parametrization can represent the same family of functions of the input as the old parametrization, but the new parametrization has [UNK] learning dynamics. in the old parametrization, the mean of h was determined by a complicated interaction between the parameters in the layers below h. in the new parametrization, the mean of γh + β is determined solely by β. the new parametrization is much easier to learn with gradient descent. most neural network layers take the form of φ ( xw + b ) where φ is some ﬁxed nonlinear activation function such as the rectiﬁed linear transformation. it is natural to wonder whether we should apply batch normalization to the input x, or to the transformed value xw + b. ( ) recommend [UNK] and szegedy 2015 320


chapter 8. optimization for training deep models the latter. more speciﬁcally, xw + b should be replaced by a normalized version of xw. the bias term should be omitted because it becomes redundant with the β parameter applied by the batch normalization reparametrization. the input to a layer is usually the output of a nonlinear activation function such as the rectiﬁed linear function in a previous layer. the statistics of the input are thus more non - gaussian and less amenable to standardization by linear operations. in convolutional networks, described in chapter, it is important to apply the 9 same normalizing µ and σ at every spatial location within a feature map, so that the statistics of the feature map remain the same regardless of spatial location. 8. 7. 2 coordinate descent in some cases, it may be possible to solve an optimization problem quickly by breaking it into separate pieces. if we minimize f ( x ) with respect to a single variable xi, then minimize it with respect to another variable x j and so on, repeatedly cycling through all variables, we are guaranteed to arrive at a ( local ) minimum. this practice is known as coordinate descent, because we optimize one coordinate at a time. more


it with respect to another variable x j and so on, repeatedly cycling through all variables, we are guaranteed to arrive at a ( local ) minimum. this practice is known as coordinate descent, because we optimize one coordinate at a time. more generally, block coordinate descent refers to minimizing with respect to a subset of the variables simultaneously. the term “ coordinate descent ” is often used to refer to block coordinate descent as well as the strictly individual coordinate descent. coordinate descent makes the most sense when the [UNK] variables in the optimization problem can be clearly separated into groups that play relatively isolated roles, or when optimization with respect to one group of variables is signiﬁcantly more [UNK] than optimization with respect to all of the variables. for example, consider the cost function j, ( h w ) = i, j | hi, j | + i, j x w − h 2 i, j. ( 8. 38 ) this function describes a learning problem called sparse coding, where the goal is to ﬁnd a weight matrix w that can linearly decode a matrix of activation values h to reconstruct the training set x. most applications of sparse coding also involve weight decay or a constraint on the norms of the columns of w, in order to prevent the


weight matrix w that can linearly decode a matrix of activation values h to reconstruct the training set x. most applications of sparse coding also involve weight decay or a constraint on the norms of the columns of w, in order to prevent the pathological solution with extremely small and large. h w the function j is not convex. however, we can divide the inputs to the training algorithm into two sets : the dictionary parameters w and the code representations h. minimizing the objective function with respect to either one of these sets of variables is a convex problem. block coordinate descent thus gives 321


chapter 8. optimization for training deep models us an optimization strategy that allows us to use [UNK] convex optimization algorithms, by alternating between optimizing w with h ﬁxed, then optimizing h w with ﬁxed. coordinate descent is not a very good strategy when the value of one variable strongly inﬂuences the optimal value of another variable, as in the function f ( x ) = ( x1 −x2 ) 2 + α x2 1 + x2 2 where α is a positive constant. the ﬁrst term encourages the two variables to have similar value, while the second term encourages them to be near zero. the solution is to set both to zero. newton ’ s method can solve the problem in a single step because it is a positive deﬁnite quadratic problem. however, for small α, coordinate descent will make very slow progress because the ﬁrst term does not allow a single variable to be changed to a value that [UNK] signiﬁcantly from the current value of the other variable. 8. 7. 3 polyak averaging polyak averaging ( polyak and juditsky 1992, ) consists of averaging together several points in the trajectory through parameter space visited by an optimization algorithm. if t iterations of gradient descent visit points θ


. 8. 7. 3 polyak averaging polyak averaging ( polyak and juditsky 1992, ) consists of averaging together several points in the trajectory through parameter space visited by an optimization algorithm. if t iterations of gradient descent visit points θ ( 1 ),..., θ ( ) t, then the output of the polyak averaging algorithm is [UNK] ( ) t = 1 t i θ ( ) i. on some problem classes, such as gradient descent applied to convex problems, this approach has strong convergence guarantees. when applied to neural networks, its justiﬁcation is more heuristic, but it performs well in practice. the basic idea is that the optimization algorithm may leap back and forth across a valley several times without ever visiting a point near the bottom of the valley. the average of all of the locations on either side should be close to the bottom of the valley though. in non - convex problems, the path taken by the optimization trajectory can be very complicated and visit many [UNK] regions. including points in parameter space from the distant past that may be separated from the current point by large barriers in the cost function does not seem like a useful behavior. as a result, when applying polyak averaging to non - convex problems, it is typical to


points in parameter space from the distant past that may be separated from the current point by large barriers in the cost function does not seem like a useful behavior. as a result, when applying polyak averaging to non - convex problems, it is typical to use an exponentially decaying running average : [UNK] ( ) t = [UNK] ( 1 ) t− + ( 1 ) −α θ ( ) t. ( 8. 39 ) the running average approach is used in numerous applications. see szegedy et al. ( ) for a recent example. 2015 322


chapter 8. optimization for training deep models 8. 7. 4 supervised pretraining sometimes, directly training a model to solve a speciﬁc task can be too ambitious if the model is complex and hard to optimize or if the task is very [UNK]. it is sometimes more [UNK] to train a simpler model to solve the task, then make the model more complex. it can also be more [UNK] to train the model to solve a simpler task, then move on to confront the ﬁnal task. these strategies that involve training simple models on simple tasks before confronting the challenge of training the desired model to perform the desired task are collectively known as pretraining. greedy algorithms break a problem into many components, then solve for the optimal version of each component in isolation. unfortunately, combining the individually optimal components is not guaranteed to yield an optimal complete solution. however, greedy algorithms can be computationally much cheaper than algorithms that solve for the best joint solution, and the quality of a greedy solution is often acceptable if not optimal. greedy algorithms may also be followed by a ﬁne - tuning stage in which a joint optimization algorithm searches for an optimal solution to the full problem. initializing the joint optimization algorithm with a greedy solution can greatly speed it up and improve the quality of the


greedy algorithms may also be followed by a ﬁne - tuning stage in which a joint optimization algorithm searches for an optimal solution to the full problem. initializing the joint optimization algorithm with a greedy solution can greatly speed it up and improve the quality of the solution it ﬁnds. pretraining, and especially greedy pretraining, algorithms are ubiquitous in deep learning. in this section, we describe speciﬁcally those pretraining algorithms that break supervised learning problems into other simpler supervised learning problems. this approach is known as. greedy supervised pretraining in the original (, ) version of greedy supervised pretraining, bengio et al. 2007 each stage consists of a supervised learning training task involving only a subset of the layers in the ﬁnal neural network. an example of greedy supervised pretraining is illustrated in ﬁgure, in which each added hidden layer is pretrained as part 8. 7 of a shallow supervised mlp, taking as input the output of the previously trained hidden layer. instead of pretraining one layer at a time, simonyan and zisserman ( ) pretrain a deep convolutional network ( eleven weight layers ) and then use 2015 the ﬁrst four and last three layers from this network to initialize


pretraining one layer at a time, simonyan and zisserman ( ) pretrain a deep convolutional network ( eleven weight layers ) and then use 2015 the ﬁrst four and last three layers from this network to initialize even deeper networks ( with up to nineteen layers of weights ). the middle layers of the new, very deep network are initialized randomly. the new network is then jointly trained. another option, explored by yu 2010 et al. ( ) is to use the of the previously outputs trained mlps, as well as the raw input, as inputs for each added stage. why would greedy supervised pretraining help? the hypothesis initially discussed by ( ) is that it helps to provide better guidance to the bengio et al. 2007 323


chapter 8. optimization for training deep models y h ( 1 ) h ( 1 ) x ( a ) u ( 1 ) u ( 1 ) w ( 1 ) w ( 1 ) y h ( 1 ) h ( 1 ) x ( b ) u ( 1 ) u ( 1 ) w ( 1 ) w ( 1 ) y h ( 1 ) h ( 1 ) x ( c ) u ( 1 ) u ( 1 ) w ( 1 ) w ( 1 ) h ( 2 ) h ( 2 ) y u ( 2 ) u ( 2 ) w ( 2 ) w ( 2 ) y h ( 1 ) h ( 1 ) x ( d ) u ( 1 ) u ( 1 ) w ( 1 ) w ( 1 ) h ( 2 ) h ( 2 ) y u ( 2 ) u ( 2 ) w ( 2 ) w ( 2 ) figure 8. 7 : illustration of one form of greedy supervised pretraining (, ). bengio et al. 2007 ( a ) we start by training a [UNK] shallow architecture. another drawing of the ( b ) same architecture. we keep only the input - to - hidden layer of the original network and ( c ) discard the hidden - to - output layer. we send the output of the ﬁ


a [UNK] shallow architecture. another drawing of the ( b ) same architecture. we keep only the input - to - hidden layer of the original network and ( c ) discard the hidden - to - output layer. we send the output of the ﬁrst hidden layer as input to another supervised single hidden layer mlp that is trained with the same objective as the ﬁrst network was, thus adding a second hidden layer. this can be repeated for as many layers as desired. another drawing of the result, viewed as a feedforward network. ( d ) to further improve the optimization, we can jointly ﬁne - tune all the layers, either only at the end or at each stage of this process. 324


chapter 8. optimization for training deep models intermediate levels of a deep hierarchy. in general, pretraining may help both in terms of optimization and in terms of generalization. an approach related to supervised pretraining extends the idea to the context of transfer learning : yosinski 2014 et al. ( ) pretrain a deep convolutional net with 8 layers of weights on a set of tasks ( a subset of the 1000 imagenet object categories ) and then initialize a same - size network with the ﬁrst k layers of the ﬁrst net. all the layers of the second network ( with the upper layers initialized randomly ) are then jointly trained to perform a [UNK] set of tasks ( another subset of the 1000 imagenet object categories ), with fewer training examples than for the ﬁrst set of tasks. other approaches to transfer learning with neural networks are discussed in section. 15. 2 another related line of work is the fitnets (, ) approach. romero et al. 2015 this approach begins by training a network that has low enough depth and great enough width ( number of units per layer ) to be easy to train. this network then becomes a teacher for a second network, designated the student. the student network is much deeper and thinner ( eleven to


a network that has low enough depth and great enough width ( number of units per layer ) to be easy to train. this network then becomes a teacher for a second network, designated the student. the student network is much deeper and thinner ( eleven to nineteen layers ) and would be [UNK] to train with sgd under normal circumstances. the training of the student network is made easier by training the student network not only to predict the output for the original task, but also to predict the value of the middle layer of the teacher network. this extra task provides a set of hints about how the hidden layers should be used and can simplify the optimization problem. additional parameters are introduced to regress the middle layer of the 5 - layer teacher network from the middle layer of the deeper student network. however, instead of predicting the ﬁnal classiﬁcation target, the objective is to predict the middle hidden layer of the teacher network. the lower layers of the student networks thus have two objectives : to help the outputs of the student network accomplish their task, as well as to predict the intermediate layer of the teacher network. although a thin and deep network appears to be more [UNK] to train than a wide and shallow network, the thin and deep network may generalize better and certainly has lower computational


their task, as well as to predict the intermediate layer of the teacher network. although a thin and deep network appears to be more [UNK] to train than a wide and shallow network, the thin and deep network may generalize better and certainly has lower computational cost if it is thin enough to have far fewer parameters. without the hints on the hidden layer, the student network performs very poorly in the experiments, both on the training and test set. hints on middle layers may thus be one of the tools to help train neural networks that otherwise seem [UNK] to train, but other optimization techniques or changes in the architecture may also solve the problem. 325


chapter 8. optimization for training deep models 8. 7. 5 designing models to aid optimization to improve optimization, the best strategy is not always to improve the optimization algorithm. instead, many improvements in the optimization of deep models have come from designing the models to be easier to optimize. in principle, we could use activation functions that increase and decrease in jagged non - monotonic patterns. however, this would make optimization extremely [UNK]. in practice, it is more important to choose a model family that is easy to optimize than to use a powerful optimization algorithm. most of the advances in neural network learning over the past 30 years have been obtained by changing the model family rather than changing the optimization procedure. stochastic gradient descent with momentum, which was used to train neural networks in the 1980s, remains in use in modern state of the art neural network applications. speciﬁcally, modern neural networks reﬂect a design choice to use linear trans - formations between layers and activation functions that are [UNK] almost everywhere and have signiﬁcant slope in large portions of their domain. in par - ticular, model innovations like the lstm, rectiﬁed linear units and maxout units have all moved toward using more linear functions than previous models like


have signiﬁcant slope in large portions of their domain. in par - ticular, model innovations like the lstm, rectiﬁed linear units and maxout units have all moved toward using more linear functions than previous models like deep networks based on sigmoidal units. these models have nice properties that make optimization easier. the gradient ﬂows through many layers provided that the jacobian of the linear transformation has reasonable singular values. moreover, linear functions consistently increase in a single direction, so even if the model ’ s output is very far from correct, it is clear simply from computing the gradient which direction its output should move to reduce the loss function. in other words, modern neural nets have been designed so that their local gradient information corresponds reasonably well to moving toward a distant solution. other model design strategies can help to make optimization easier. for example, linear paths or skip connections between layers reduce the length of the shortest path from the lower layer ’ s parameters to the output, and thus mitigate the vanishing gradient problem ( srivastava 2015 et al., ). a related idea to skip connections is adding extra copies of the output that are attached to the intermediate hidden layers of the network, as in googlenet (, ) szeged


vanishing gradient problem ( srivastava 2015 et al., ). a related idea to skip connections is adding extra copies of the output that are attached to the intermediate hidden layers of the network, as in googlenet (, ) szegedy et al. 2014a and deeply - supervised nets (, ). these “ auxiliary heads ” are trained lee et al. 2014 to perform the same task as the primary output at the top of the network in order to ensure that the lower layers receive a large gradient. when training is complete the auxiliary heads may be discarded. this is an alternative to the pretraining strategies, which were introduced in the previous section. in this way, one can train jointly all the layers in a single phase but change the architecture, so that intermediate layers ( especially the lower ones ) can get some hints about what they 326


chapter 8. optimization for training deep models should do, via a shorter path. these hints provide an error signal to lower layers. 8. 7. 6 continuation methods and curriculum learning as argued in section, many of the challenges in optimization arise from the 8. 2. 7 global structure of the cost function and cannot be resolved merely by making better estimates of local update directions. the predominant strategy for overcoming this problem is to attempt to initialize the parameters in a region that is connected to the solution by a short path through parameter space that local descent can discover. continuation methods are a family of strategies that can make optimization easier by choosing initial points to ensure that local optimization spends most of its time in well - behaved regions of space. the idea behind continuation methods is to construct a series of objective functions over the same parameters. in order to minimize a cost function j ( θ ), we will construct new cost functions { j ( 0 ),..., j ( ) n }. these cost functions are designed to be increasingly [UNK], with j ( 0 ) being fairly easy to minimize, and j ( ) n, the most [UNK], being j ( θ ), the true cost function motivating the entire process. when we say that j ( ) i is easier


[UNK], with j ( 0 ) being fairly easy to minimize, and j ( ) n, the most [UNK], being j ( θ ), the true cost function motivating the entire process. when we say that j ( ) i is easier than j ( + 1 ) i, we mean that it is well behaved over more of θ space. a random initialization is more likely to land in the region where local descent can minimize the cost function successfully because this region is larger. the series of cost functions are designed so that a solution to one is a good initial point of the next. we thus begin by solving an easy problem then reﬁne the solution to solve incrementally harder problems until we arrive at a solution to the true underlying problem. traditional continuation methods ( predating the use of continuation methods for neural network training ) are usually based on smoothing the objective function. see wu 1997 ( ) for an example of such a method and a review of some related methods. continuation methods are also closely related to simulated annealing, which adds noise to the parameters ( kirkpatrick 1983 et al., ). continuation methods have been extremely successful in recent years. see mobahi and fisher ( ) for an overview of recent literature, especially for ai applications


simulated annealing, which adds noise to the parameters ( kirkpatrick 1983 et al., ). continuation methods have been extremely successful in recent years. see mobahi and fisher ( ) for an overview of recent literature, especially for ai applications. 2015 continuation methods traditionally were mostly designed with the goal of overcoming the challenge of local minima. speciﬁcally, they were designed to reach a global minimum despite the presence of many local minima. to do so, these continuation methods would construct easier cost functions by “ blurring ” the original cost function. this blurring operation can be done by approximating j ( ) i ( ) = θ [UNK] ( θ ; θ, σ ( ) 2 i ) j ( θ ) ( 8. 40 ) via sampling. the intuition for this approach is that some non - convex functions 327


chapter 8. optimization for training deep models become approximately convex when blurred. in many cases, this blurring preserves enough information about the location of a global minimum that we can ﬁnd the global minimum by solving progressively less blurred versions of the problem. this approach can break down in three [UNK] ways. first, it might successfully deﬁne a series of cost functions where the ﬁrst is convex and the optimum tracks from one function to the next arriving at the global minimum, but it might require so many incremental cost functions that the cost of the entire procedure remains high. np - hard optimization problems remain np - hard, even when continuation methods are applicable. the other two ways that continuation methods fail both correspond to the method not being applicable. first, the function might not become convex, no matter how much it is blurred. consider for example the function j ( θ ) = −θθ. second, the function may become convex as a result of blurring, but the minimum of this blurred function may track to a local rather than a global minimum of the original cost function. though continuation methods were mostly originally designed to deal with the problem of local minima, local minima are no longer believed to be the primary problem for neural network optimization. fortunately, continuation methods


local rather than a global minimum of the original cost function. though continuation methods were mostly originally designed to deal with the problem of local minima, local minima are no longer believed to be the primary problem for neural network optimization. fortunately, continuation methods can still help. the easier objective functions introduced by the continuation method can eliminate ﬂat regions, decrease variance in gradient estimates, improve conditioning of the hessian matrix, or do anything else that will either make local updates easier to compute or improve the correspondence between local update directions and progress toward a global solution. bengio 2009 et al. ( ) observed that an approach called curriculum learning or shaping can be interpreted as a continuation method. curriculum learning is based on the idea of planning a learning process to begin by learning simple concepts and progress to learning more complex concepts that depend on these simpler concepts. this basic strategy was previously known to accelerate progress in animal training (, ;, ; skinner 1958 peterson 2004 krueger and dayan 2009, ) and machine learning (, ;, ;, ). ( ) [UNK] elman 1993 sanger 1994 bengio et al. 2009 justiﬁed this strategy as a continuation method, where earlier j ( ) i are made easier by increasing the inﬂuence of simpler examples ( either


, ). ( ) [UNK] elman 1993 sanger 1994 bengio et al. 2009 justiﬁed this strategy as a continuation method, where earlier j ( ) i are made easier by increasing the inﬂuence of simpler examples ( either by assigning their contributions to the cost function larger [UNK], or by sampling them more frequently ), and experimentally demonstrated that better results could be obtained by following a curriculum on a large - scale neural language modeling task. curriculum learning has been successful on a wide range of natural language ( spitkovsky 2010 et al., ; collobert 2011a mikolov 2011b tu and honavar 2011 et al., ; et al., ;, ) and computer vision (, ;, ;, ) kumar et al. 2010 lee and grauman 2011 supancic and ramanan 2013 tasks. curriculum learning was also veriﬁed as being consistent with the way in which humans teach (, ) : teachers start by showing easier and khan et al. 2011 328


chapter 8. optimization for training deep models more prototypical examples and then help the learner reﬁne the decision surface with the less obvious cases. curriculum - based strategies are more [UNK] for teaching humans than strategies based on uniform sampling of examples, and can also increase the [UNK] of other teaching strategies (, basu and christensen 2013 ). another important contribution to research on curriculum learning arose in the context of training recurrent neural networks to capture long - term dependencies : zaremba and sutskever 2014 ( ) found that much better results were obtained with a stochastic curriculum, in which a random mix of easy and [UNK] examples is always presented to the learner, but where the average proportion of the more [UNK] examples ( here, those with longer - term dependencies ) is gradually increased. with a deterministic curriculum, no improvement over the baseline ( ordinary training from the full training set ) was observed. we have now described the basic family of neural network models and how to regularize and optimize them. in the chapters ahead, we turn to specializations of the neural network family, that allow neural networks to scale to very large sizes and process input data that has special structure. the optimization methods discussed in this chapter are often directly applicable to these specialized


. in the chapters ahead, we turn to specializations of the neural network family, that allow neural networks to scale to very large sizes and process input data that has special structure. the optimization methods discussed in this chapter are often directly applicable to these specialized architectures with little or no modiﬁcation. 329


chapter 9 convolutional networks convolutional networks (, ), also known as lecun 1989 convolutional neural networks or cnns, are a specialized kind of neural network for processing data that has a known, grid - like topology. examples include time - series data, which can be thought of as a 1d grid taking samples at regular time intervals, and image data, which can be thought of as a 2d grid of pixels. convolutional networks have been tremendously successful in practical applications. the name “ convolutional neural network ” indicates that the network employs a mathematical operation called convolution. convolution is a specialized kind of linear operation. convolutional networks are simply neural networks that use convolution in place of general matrix multiplication in at least one of their layers. in this chapter, we will ﬁrst describe what convolution is. next, we will explain the motivation behind using convolution in a neural network. we will then describe an operation called pooling, which almost all convolutional networks employ. usually, the operation used in a convolutional neural network does not correspond precisely to the deﬁni


##tion in a neural network. we will then describe an operation called pooling, which almost all convolutional networks employ. usually, the operation used in a convolutional neural network does not correspond precisely to the deﬁnition of convolution as used in other ﬁelds such as engineering or pure mathematics. we will describe several variants on the convolution function that are widely used in practice for neural networks. we will also show how convolution may be applied to many kinds of data, with [UNK] numbers of dimensions. we then discuss means of making convolution more [UNK]. convolutional networks stand out as an example of neuroscientiﬁc principles inﬂuencing deep learning. we will discuss these neuroscientiﬁc principles, then conclude with comments about the role convolutional networks have played in the history of deep learning. one topic this chapter does not address is how to choose the architecture of your convolutional network. the goal of this chapter is to describe the kinds of tools that convolutional networks provide, while chapter 11 330


chapter 9. convolutional networks describes general guidelines for choosing which tools to use in which circumstances. research into convolutional network architectures proceeds so rapidly that a new best architecture for a given benchmark is announced every few weeks to months, rendering it impractical to describe the best architecture in print. however, the best architectures have consistently been composed of the building blocks described here. 9. 1 the convolution operation in its most general form, convolution is an operation on two functions of a real - valued argument. to motivate the deﬁnition of convolution, we start with examples of two functions we might use. suppose we are tracking the location of a spaceship with a laser sensor. our laser sensor provides a single output x ( t ), the position of the spaceship at time t. both x and t are real - valued, i. e., we can get a [UNK] reading from the laser sensor at any instant in time. now suppose that our laser sensor is somewhat noisy. to obtain a less noisy estimate of the spaceship ’ s position, we would like to average together several measurements. of course, more recent measurements are more relevant, so we will want this to be a


now suppose that our laser sensor is somewhat noisy. to obtain a less noisy estimate of the spaceship ’ s position, we would like to average together several measurements. of course, more recent measurements are more relevant, so we will want this to be a weighted average that gives more weight to recent measurements. we can do this with a weighting function w ( a ), where a is the age of a measurement. if we apply such a weighted average operation at every moment, we obtain a new function providing a smoothed estimate of the position of the spaceship : s s t ( ) = x a w t a da ( ) ( − ) ( 9. 1 ) this operation is called convolution. the convolution operation is typically denoted with an asterisk : s t x w t ( ) = ( ∗ ) ( ) ( 9. 2 ) in our example, w needs to be a valid probability density function, or the output is not a weighted average. also, w needs to be for all negative arguments, 0 or it will look into the future, which is presumably beyond our capabilities. these limitations are particular to our example though. in general, convolution is deﬁned for any functions for which the above integral is deﬁ


negative arguments, 0 or it will look into the future, which is presumably beyond our capabilities. these limitations are particular to our example though. in general, convolution is deﬁned for any functions for which the above integral is deﬁned, and may be used for other purposes besides taking weighted averages. in convolutional network terminology, the ﬁrst argument ( in this example, the function x ) to the convolution is often referred to as the input and the second 331


chapter 9. convolutional networks argument ( in this example, the function w ) as the kernel. the output is sometimes referred to as the. feature map in our example, the idea of a laser sensor that can provide measurements at every instant in time is not realistic. usually, when we work with data on a computer, time will be discretized, and our sensor will provide data at regular intervals. in our example, it might be more realistic to assume that our laser provides a measurement once per second. the time index t can then take on only integer values. if we now assume that x and w are deﬁned only on integer t, we can deﬁne the discrete convolution : s t x w t ( ) = ( ∗ ) ( ) = ∞ a = −∞ x a w t a ( ) ( − ) ( 9. 3 ) in machine learning applications, the input is usually a multidimensional array of data and the kernel is usually a multidimensional array of parameters that are adapted by the learning algorithm. we will refer to these multidimensional arrays as tensors. because each element of the input and kernel must be explicitly stored separately, we usually assume that these functions are zero everywhere


##ensional array of parameters that are adapted by the learning algorithm. we will refer to these multidimensional arrays as tensors. because each element of the input and kernel must be explicitly stored separately, we usually assume that these functions are zero everywhere but the ﬁnite set of points for which we store the values. this means that in practice we can implement the inﬁnite summation as a summation over a ﬁnite number of array elements. finally, we often use convolutions over more than one axis at a time. for example, if we use a two - dimensional image i as our input, we probably also want to use a two - dimensional kernel : k s i, j i k i, j ( ) = ( ∗ ) ( ) = m n i m, n k i m, j n. ( ) ( − − ) ( 9. 4 ) convolution is commutative, meaning we can equivalently write : s i, j k i i, j ( ) = ( ∗ ) ( ) = m n i i m, j n k m, n. ( − − ) ( ) ( 9. 5 ) usually the latter formula is more straightforward to implement in a machine learning library,


, j ( ) = ( ∗ ) ( ) = m n i i m, j n k m, n. ( − − ) ( ) ( 9. 5 ) usually the latter formula is more straightforward to implement in a machine learning library, because there is less variation in the range of valid values of m and. n the commutative property of convolution arises because we have ﬂipped the kernel relative to the input, in the sense that as m increases, the index into the input increases, but the index into the kernel decreases. the only reason to ﬂip the kernel is to obtain the commutative property. while the commutative property 332


chapter 9. convolutional networks is useful for writing proofs, it is not usually an important property of a neural network implementation. instead, many neural network libraries implement a related function called the cross - correlation, which is the same as convolution but without ﬂipping the kernel : s i, j i k i, j ( ) = ( ∗ ) ( ) = m n i i m, j n k m, n. ( + + ) ( ) ( 9. 6 ) many machine learning libraries implement cross - correlation but call it convolution. in this text we will follow this convention of calling both operations convolution, and specify whether we mean to ﬂip the kernel or not in contexts where kernel ﬂipping is relevant. in the context of machine learning, the learning algorithm will learn the appropriate values of the kernel in the appropriate place, so an algorithm based on convolution with kernel ﬂipping will learn a kernel that is ﬂipped relative to the kernel learned by an algorithm without the ﬂipping. it is also rare for convolution to be used alone in machine learning ; instead convolution is used simultaneously with other functions, and the combination of these functions does not


to the kernel learned by an algorithm without the ﬂipping. it is also rare for convolution to be used alone in machine learning ; instead convolution is used simultaneously with other functions, and the combination of these functions does not commute regardless of whether the convolution operation ﬂips its kernel or not. see ﬁgure for an example of convolution ( without kernel ﬂipping ) applied 9. 1 to a 2 - d tensor. discrete convolution can be viewed as multiplication by a matrix. however, the matrix has several entries constrained to be equal to other entries. for example, for univariate discrete convolution, each row of the matrix is constrained to be equal to the row above shifted by one element. this is known as a toeplitz matrix. in two dimensions, a doubly block circulant matrix corresponds to convolution. in addition to these constraints that several elements be equal to each other, convolution usually corresponds to a very sparse matrix ( a matrix whose entries are mostly equal to zero ). this is because the kernel is usually much smaller than the input image. any neural network algorithm that works with matrix multiplication and does not depend on


##volution usually corresponds to a very sparse matrix ( a matrix whose entries are mostly equal to zero ). this is because the kernel is usually much smaller than the input image. any neural network algorithm that works with matrix multiplication and does not depend on speciﬁc properties of the matrix structure should work with convolution, without requiring any further changes to the neural network. typical convolutional neural networks do make use of further specializations in order to deal with large inputs [UNK], but these are not strictly necessary from a theoretical perspective. 333


chapter 9. convolutional networks a b c d e f g h i j k l w x y z aw + bx + ey + fz aw + bx + ey + fz bw + cx + fy + gz bw + cx + fy + gz cw + dx + gy + hz cw + dx + gy + hz ew + fx + iy + jz ew + fx + iy + jz fw + gx + jy + kz fw + gx + jy + kz gw + hx + ky + lz gw + hx + ky + lz input kernel output figure 9. 1 : an example of 2 - d convolution without kernel - ﬂipping. in this case we restrict the output to only positions where the kernel lies entirely within the image, called “ valid ” convolution in some contexts. we draw boxes with arrows to indicate how the upper - left element of the output tensor is formed by applying the kernel to the corresponding upper - left region of the input tensor. 334


chapter 9. convolutional networks 9. 2 motivation convolution leverages three important ideas that can help improve a machine learning system : sparse interactions, parameter sharing and equivariant representations. moreover, convolution provides a means for working with inputs of variable size. we now describe each of these ideas in turn. traditional neural network layers use matrix multiplication by a matrix of parameters with a separate parameter describing the interaction between each input unit and each output unit. this means every output unit interacts with every input unit. convolutional networks, however, typically have sparse interactions ( also referred to as sparse connectivity or sparse weights ). this is accomplished by making the kernel smaller than the input. for example, when processing an image, the input image might have thousands or millions of pixels, but we can detect small, meaningful features such as edges with kernels that occupy only tens or hundreds of pixels. this means that we need to store fewer parameters, which both reduces the memory requirements of the model and improves its statistical [UNK]. it also means that computing the output requires fewer operations. these improvements in [UNK] are usually quite large. if there are m inputs and n outputs, then matrix multiplication requires m n × parameters and the algorithms used


of the model and improves its statistical [UNK]. it also means that computing the output requires fewer operations. these improvements in [UNK] are usually quite large. if there are m inputs and n outputs, then matrix multiplication requires m n × parameters and the algorithms used in practice have o ( m n × ) runtime ( per example ). if we limit the number of connections each output may have to k, then the sparsely connected approach requires only k n × parameters and o ( k n × ) runtime. for many practical applications, it is possible to obtain good performance on the machine learning task while keeping k several orders of magnitude smaller than m. for graphical demonstrations of sparse connectivity, see ﬁgure and ﬁgure. in a deep convolutional network, 9. 2 9. 3 units in the deeper layers may indirectly interact with a larger portion of the input, as shown in ﬁgure. this allows the network to [UNK] describe complicated 9. 4 interactions between many variables by constructing such interactions from simple building blocks that each describe only sparse interactions. parameter sharing refers to using the same parameter for more than one function in a model. in a traditional neural net, each element of the weight matrix is used exactly once when computing the output of a layer. it


blocks that each describe only sparse interactions. parameter sharing refers to using the same parameter for more than one function in a model. in a traditional neural net, each element of the weight matrix is used exactly once when computing the output of a layer. it is multiplied by one element of the input and then never revisited. as a synonym for parameter sharing, one can say that a network has tied weights, because the value of the weight applied to one input is tied to the value of a weight applied elsewhere. in a convolutional neural net, each member of the kernel is used at every position of the input ( except perhaps some of the boundary pixels, depending on the design decisions regarding the boundary ). the parameter sharing used by the convolution operation means that rather than learning a separate set of parameters 335


chapter 9. convolutional networks x1 x1 x2 x2 x3 x3 s2 s2 s1 s1 s3 s3 x4 x4 s4 s4 x5 x5 s5 s5 x1 x1 x2 x2 x3 x3 s2 s2 s1 s1 s3 s3 x4 x4 s4 s4 x5 x5 s5 s5 figure 9. 2 : sparse connectivity, viewed from below : we highlight one input unit, x3, and also highlight the output units in s that are [UNK] by this unit. ( top ) when s is formed by convolution with a kernel of width, only three outputs are [UNK] by 3 x. ( bottom ) when is formed by matrix multiplication, connectivity is no longer sparse, so s all of the outputs are [UNK] by x3. 336


chapter 9. convolutional networks x1 x1 x2 x2 x3 x3 s2 s2 s1 s1 s3 s3 x4 x4 s4 s4 x5 x5 s5 s5 x1 x1 x2 x2 x3 x3 s2 s2 s1 s1 s3 s3 x4 x4 s4 s4 x5 x5 s5 s5 figure 9. 3 : sparse connectivity, viewed from above : we highlight one output unit, s3, and also highlight the input units in x that [UNK] this unit. these units are known as the receptive ﬁeld of s3. ( top ) when s is formed by convolution with a kernel of width, only three inputs [UNK] 3 s3. when ( bottom ) s is formed by matrix multiplication, connectivity is no longer sparse, so all of the inputs [UNK] s3. x1 x1 x2 x2 x3 x3 h2h2 h1h1 h3 h3 x4 x4 h4h4 x5 x5 h5h5 g2g2 g1g1 g3 g3 g4g4 g5g5 figure 9. 4 : the rec


##1h1 h3 h3 x4 x4 h4h4 x5 x5 h5h5 g2g2 g1g1 g3 g3 g4g4 g5g5 figure 9. 4 : the receptive ﬁeld of the units in the deeper layers of a convolutional network is larger than the receptive ﬁeld of the units in the shallow layers. this [UNK] increases if the network includes architectural features like strided convolution ( ﬁgure ) or pooling 9. 12 ( section ). this means that even though 9. 3 direct connections in a convolutional net are very sparse, units in the deeper layers can be indirectly connected to all or most of the input image. 337


chapter 9. convolutional networks x1 x1 x2 x2 x3 x3 s2 s2 s1 s1 s3 s3 x4 x4 s4 s4 x5 x5 s5 s5 x1 x1 x2 x2 x3 x3 x4 x4 x5 x5 s2 s2 s1 s1 s3 s3 s4 s4 s5 s5 figure 9. 5 : parameter sharing : black arrows indicate the connections that use a particular parameter in two [UNK] models. ( top ) the black arrows indicate uses of the central element of a 3 - element kernel in a convolutional model. due to parameter sharing, this single parameter is used at all input locations. the single black arrow indicates ( bottom ) the use of the central element of the weight matrix in a fully connected model. this model has no parameter sharing so the parameter is used only once. for every location, we learn only one set. this does not [UNK] the runtime of forward propagation — it is still o ( k n × ) — but it does further reduce the storage requirements of the model to k parameters. recall that k is usually several orders of magnitude less than m. since m and n are


the runtime of forward propagation — it is still o ( k n × ) — but it does further reduce the storage requirements of the model to k parameters. recall that k is usually several orders of magnitude less than m. since m and n are usually roughly the same size, k is practically insigniﬁcant compared to m n ×. convolution is thus dramatically more [UNK] than dense matrix multiplication in terms of the memory requirements and statistical [UNK]. for a graphical depiction of how parameter sharing works, see ﬁgure. 9. 5 as an example of both of these ﬁrst two principles in action, ﬁgure shows 9. 6 how sparse connectivity and parameter sharing can dramatically improve the [UNK] of a linear function for detecting edges in an image. in the case of convolution, the particular form of parameter sharing causes the layer to have a property called equivariance to translation. to say a function is equivariant means that if the input changes, the output changes in the same way. speciﬁcally, a function f ( x ) is equivariant to a function g if f ( g ( x ) ) = g ( f ( x ) ). in the case of convolu


the same way. speciﬁcally, a function f ( x ) is equivariant to a function g if f ( g ( x ) ) = g ( f ( x ) ). in the case of convolution, if we let g be any function that translates the input, i. e., shifts it, then the convolution function is equivariant to g. for example, let i be a function giving image brightness at integer coordinates. let g be a function 338


chapter 9. convolutional networks mapping one image function to another image function, such that i = g ( i ) is the image function with i ( x, y ) = i ( x −1, y ). this shifts every pixel of i one unit to the right. if we apply this transformation to i, then apply convolution, the result will be the same as if we applied convolution to i, then applied the transformation g to the output. when processing time series data, this means that convolution produces a sort of timeline that shows when [UNK] features appear in the input. if we move an event later in time in the input, the exact same representation of it will appear in the output, just later in time. similarly with images, convolution creates a 2 - d map of where certain features appear in the input. if we move the object in the input, its representation will move the same amount in the output. this is useful for when we know that some function of a small number of neighboring pixels is useful when applied to multiple input locations. for example, when processing images, it is useful to detect edges in the ﬁrst layer of a convolutional network. the same edges


know that some function of a small number of neighboring pixels is useful when applied to multiple input locations. for example, when processing images, it is useful to detect edges in the ﬁrst layer of a convolutional network. the same edges appear more or less everywhere in the image, so it is practical to share parameters across the entire image. in some cases, we may not wish to share parameters across the entire image. for example, if we are processing images that are cropped to be centered on an individual ’ s face, we probably want to extract [UNK] features at [UNK] locations — the part of the network processing the top of the face needs to look for eyebrows, while the part of the network processing the bottom of the face needs to look for a chin. convolution is not naturally equivariant to some other transformations, such as changes in the scale or rotation of an image. other mechanisms are necessary for handling these kinds of transformations. finally, some kinds of data cannot be processed by neural networks deﬁned by matrix multiplication with a ﬁxed - shape matrix. convolution enables processing of some of these kinds of data. we discuss this further in section. 9. 7 9. 3 pooling a typical layer of a


neural networks deﬁned by matrix multiplication with a ﬁxed - shape matrix. convolution enables processing of some of these kinds of data. we discuss this further in section. 9. 7 9. 3 pooling a typical layer of a convolutional network consists of three stages ( see ﬁgure ). 9. 7 in the ﬁrst stage, the layer performs several convolutions in parallel to produce a set of linear activations. in the second stage, each linear activation is run through a nonlinear activation function, such as the rectiﬁed linear activation function. this stage is sometimes called the detector stage. in the third stage, we use a pooling function to modify the output of the layer further. a pooling function replaces the output of the net at a certain location with a summary statistic of the nearby outputs. for example, the max pooling ( zhou 339


chapter 9. convolutional networks figure 9. 6 : [UNK] of edge detection. the image on the right was formed by taking each pixel in the original image and subtracting the value of its neighboring pixel on the left. this shows the strength of all of the vertically oriented edges in the input image, which can be a useful operation for object detection. both images are 280 pixels tall. the input image is 320 pixels wide while the output image is 319 pixels wide. this transformation can be described by a convolution kernel containing two elements, and requires 319 × 280 × 3 = 267, 960 ﬂoating point operations ( two multiplications and one addition per output pixel ) to compute using convolution. to describe the same transformation with a matrix multiplication would take 320× 280× 319 × 280, or over eight billion, entries in the matrix, making convolution four billion times more [UNK] for representing this transformation. the straightforward matrix multiplication algorithm performs over sixteen billion ﬂoating point operations, making convolution roughly 60, 000 times more [UNK] computationally. of course, most of the entries of the matrix would be zero. if we stored only the nonzero entries of the matrix, then both matrix multiplication and convo


, making convolution roughly 60, 000 times more [UNK] computationally. of course, most of the entries of the matrix would be zero. if we stored only the nonzero entries of the matrix, then both matrix multiplication and convolution would require the same number of ﬂoating point operations to compute. the matrix would still need to contain 2 × 319 × 280 = 178, 640 entries. convolution is an extremely [UNK] way of describing transformations that apply the same linear transformation of a small, local region across the entire input. ( photo credit : paula goodfellow ) 340