good in many contexts. warde - farley 2014 et al. ( ) compared dropout training to training of large ensembles and concluded that dropout [UNK] additional improvements to generalization error beyond those obtained by ensembles of independent models. it is important to understand that a large portion of the power of dropout arises from the fact that the masking noise is applied to the hidden units. this can be seen as a form of highly intelligent, adaptive destruction of the information content of the input rather than destruction of the raw values of the input. for example, if the model learns a hidden unit hi that detects a face by ﬁnding the nose, then dropping hi corresponds to erasing the information that there is a nose in the image. the model must learn another hi, either that redundantly encodes the presence of a nose, or that detects the face by another feature, such as the mouth. traditional noise injection techniques that add unstructured noise at the input are not able to randomly erase the information about a nose from an image of a face unless the magnitude of the noise is so great that nearly all of the information in 267


chapter 7. regularization for deep learning the image is removed. destroying extracted features rather than original values allows the destruction process to make use of all of the knowledge about the input distribution that the model has acquired so far. another important aspect of dropout is that the noise is multiplicative. if the noise were additive with ﬁxed scale, then a rectiﬁed linear hidden unit hi with added noise could simply learn to have hi become very large in order to make the added noise insigniﬁcant by comparison. multiplicative noise does not allow such a pathological solution to the noise robustness problem. another deep learning algorithm, batch normalization, reparametrizes the model in a way that introduces both additive and multiplicative noise on the hidden units at training time. the primary purpose of batch normalization is to improve optimization, but the noise can have a regularizing [UNK], and sometimes makes dropout unnecessary. batch normalization is described further in section. 8. 7. 1 7. 13 adversarial training in many cases, neural networks have begun to reach human performance when evaluated on an i. i. d. test set. it is natural therefore to wonder whether these models have obtained a true human - level understanding of

/home/ricoiban/GEMMA/mnlp_chatsplaining/RAG/Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville (z-lib.org).pdf


. 13 adversarial training in many cases, neural networks have begun to reach human performance when evaluated on an i. i. d. test set. it is natural therefore to wonder whether these models have obtained a true human - level understanding of these tasks. in order to probe the level of understanding a network has of the underlying task, we can search for examples that the model misclassiﬁes. ( ) found that szegedy et al. 2014b even neural networks that perform at human level accuracy have a nearly 100 % error rate on examples that are intentionally constructed by using an optimization procedure to search for an input xnear a data point x such that the model output is very [UNK] at x. in many cases, xcan be so similar to x that a human observer cannot tell the [UNK] between the original example and the adversarial example, but the network can make highly [UNK] predictions. see ﬁgure for an example. 7. 8 adversarial examples have many implications, for example, in computer security, that are beyond the scope of this chapter. however, they are interesting in the context of regularization because one can reduce the error rate on the original i. i. d. test set via adversarial


for example, in computer security, that are beyond the scope of this chapter. however, they are interesting in the context of regularization because one can reduce the error rate on the original i. i. d. test set via adversarial training — training on adversarially perturbed examples from the training set (, ; szegedy et al. 2014b goodfellow 2014b et al., ). goodfellow 2014b et al. ( ) showed that one of the primary causes of these adversarial examples is excessive linearity. neural networks are built out of primarily linear building blocks. in some experiments the overall function they implement proves to be highly linear as a result. these linear functions are easy 268


chapter 7. regularization for deep learning +. 007 × = x sign ( ∇xj ( θ x,, y ) ) x + sign ( ∇xj ( θ x,, y ) ) y = “ panda ” “ nematode ” “ gibbon ” w / 57. 7 % conﬁdence w / 8. 2 % conﬁdence w / 99. 3 % conﬁdence figure 7. 8 : a demonstration of adversarial example generation applied to googlenet (, ) on imagenet. by adding an imperceptibly small vector whose szegedy et al. 2014a elements are equal to the sign of the elements of the gradient of the cost function with respect to the input, we can change googlenet ’ s classiﬁcation of the image. reproduced with permission from ( ). goodfellow et al. 2014b to optimize. unfortunately, the value of a linear function can change very rapidly if it has numerous inputs. if we change each input by, then a linear function with weights w can change by as much as | | | | w 1, which can be a very large amount if w is high - dimensional. adversarial training discourages this


. if we change each input by, then a linear function with weights w can change by as much as | | | | w 1, which can be a very large amount if w is high - dimensional. adversarial training discourages this highly sensitive locally linear behavior by encouraging the network to be locally constant in the neighborhood of the training data. this can be seen as a way of explicitly introducing a local constancy prior into supervised neural nets. adversarial training helps to illustrate the power of using a large function family in combination with aggressive regularization. purely linear models, like logistic regression, are not able to resist adversarial examples because they are forced to be linear. neural networks are able to represent functions that can range from nearly linear to nearly locally constant and thus have the ﬂexibility to capture linear trends in the training data while still learning to resist local perturbation. adversarial examples also provide a means of accomplishing semi - supervised learning. at a point x that is not associated with a label in the dataset, the model itself assigns some label [UNK]. the model ’ s label [UNK] may not be the true label, but if the model is high quality, then [UNK] has a high probability of providing the true


is not associated with a label in the dataset, the model itself assigns some label [UNK]. the model ’ s label [UNK] may not be the true label, but if the model is high quality, then [UNK] has a high probability of providing the true label. we can seek an adversarial example xthat causes the classiﬁer to output a label ywith y = [UNK]. adversarial examples generated using not the true label but a label provided by a trained model are called virtual adversarial examples ( miyato 2015 et al., ). the classiﬁer may then be trained to assign the same label to x and x. this encourages the classiﬁer to learn a function that is 269


chapter 7. regularization for deep learning robust to small changes anywhere along the manifold where the unlabeled data lies. the assumption motivating this approach is that [UNK] classes usually lie on disconnected manifolds, and a small perturbation should not be able to jump from one class manifold to another class manifold. 7. 14 tangent distance, tangent prop, and manifold tangent classiﬁer many machine learning algorithms aim to overcome the curse of dimensionality by assuming that the data lies near a low - dimensional manifold, as described in section. 5. 11. 3 one of the early attempts to take advantage of the manifold hypothesis is the tangent distance algorithm (,, ). it is a non - parametric simard et al. 1993 1998 nearest - neighbor algorithm in which the metric used is not the generic euclidean distance but one that is derived from knowledge of the manifolds near which probability concentrates. it is assumed that we are trying to classify examples and that examples on the same manifold share the same category. since the classiﬁer should be invariant to the local factors of variation that correspond to movement on the manifold, it would make sense to use as nearest - neighbor distance between points x1 and x2 the distance between the manifolds m1 and m2 to


the classiﬁer should be invariant to the local factors of variation that correspond to movement on the manifold, it would make sense to use as nearest - neighbor distance between points x1 and x2 the distance between the manifolds m1 and m2 to which they respectively belong. although that may be computationally [UNK] ( it would require solving an optimization problem, to ﬁnd the nearest pair of points on m1 and m2 ), a cheap alternative that makes sense locally is to approximate mi by its tangent plane at xi and measure the distance between the two tangents, or between a tangent plane and a point. that can be achieved by solving a low - dimensional linear system ( in the dimension of the manifolds ). of course, this algorithm requires one to specify the tangent vectors. in a related spirit, the tangent prop algorithm (, ) ( ﬁgure ) simard et al. 1992 7. 9 trains a neural net classiﬁer with an extra penalty to make each output f ( x ) of the neural net locally invariant to known factors of variation. these factors of variation correspond to movement along the manifold near which examples of the same class concentrate. local invariance is achieved by requiring ∇xf ( x ) to be orthogonal to the known manifold tangent vectors


locally invariant to known factors of variation. these factors of variation correspond to movement along the manifold near which examples of the same class concentrate. local invariance is achieved by requiring ∇xf ( x ) to be orthogonal to the known manifold tangent vectors v ( ) i at x, or equivalently that the directional derivative of f at x in the directions v ( ) i be small by adding a regularization penalty : ω ω ( ) = f i ( ∇xf ( ) ) x v ( ) i 2. ( 7. 67 ) 270


chapter 7. regularization for deep learning this regularizer can of course be scaled by an appropriate hyperparameter, and, for most neural networks, we would need to sum over many outputs rather than the lone output f ( x ) described here for simplicity. as with the tangent distance algorithm, the tangent vectors are derived a priori, usually from the formal knowledge of the [UNK] of transformations such as translation, rotation, and scaling in images. tangent prop has been used not just for supervised learning (, ) simard et al. 1992 but also in the context of reinforcement learning (, ). thrun 1995 tangent propagation is closely related to dataset augmentation. in both cases, the user of the algorithm encodes his or her prior knowledge of the task by specifying a set of transformations that should not alter the output of the network. the [UNK] is that in the case of dataset augmentation, the network is explicitly trained to correctly classify distinct inputs that were created by applying more than an inﬁnitesimal amount of these transformations. tangent propagation does not require explicitly visiting a new input point. instead, it analytically regularizes the model to resist perturbation in the directions corresponding to the speciﬁed transformation. while this analytical approach


##tesimal amount of these transformations. tangent propagation does not require explicitly visiting a new input point. instead, it analytically regularizes the model to resist perturbation in the directions corresponding to the speciﬁed transformation. while this analytical approach is intellectually elegant, it has two major drawbacks. first, it only regularizes the model to resist inﬁnitesimal perturbation. explicit dataset augmentation confers resistance to larger perturbations. second, the inﬁnitesimal approach poses [UNK] for models based on rectiﬁed linear units. these models can only shrink their derivatives by turning units [UNK] shrinking their weights. they are not able to shrink their derivatives by saturating at a high value with large weights, as sigmoid or tanh units can. dataset augmentation works well with rectiﬁed linear units because [UNK] subsets of rectiﬁed units can activate for [UNK] transformed versions of each original input. tangent propagation is also related to double backprop ( drucker and lecun, 1992 ) and adversarial training (, ;, ). szegedy et al. 2014b goodfellow et al. 2014b double backprop regular


to double backprop ( drucker and lecun, 1992 ) and adversarial training (, ;, ). szegedy et al. 2014b goodfellow et al. 2014b double backprop regularizes the jacobian to be small, while adversarial training ﬁnds inputs near the original inputs and trains the model to produce the same output on these as on the original inputs. tangent propagation and dataset augmentation using manually speciﬁed transformations both require that the model should be invariant to certain speciﬁed directions of change in the input. double backprop and adversarial training both require that the model should be invariant to directions of change in the input so long as the change is small. just all as dataset augmentation is the non - inﬁnitesimal version of tangent propagation, adversarial training is the non - inﬁnitesimal version of double backprop. the manifold tangent classiﬁer (, ), eliminates the need to rifai et al. 2011c know the tangent vectors a priori. as we will see in chapter, autoencoders can 14 271


chapter 7. regularization for deep learning x1 x2 normal tangent figure 7. 9 : illustration of the main idea of the tangent prop algorithm (, simard et al. 1992 rifai 2011c ) and manifold tangent classiﬁer ( et al., ), which both regularize the classiﬁer output function f ( x ). each curve represents the manifold for a [UNK] class, illustrated here as a one - dimensional manifold embedded in a two - dimensional space. on one curve, we have chosen a single point and drawn a vector that is tangent to the class manifold ( parallel to and touching the manifold ) and a vector that is normal to the class manifold ( orthogonal to the manifold ). in multiple dimensions there may be many tangent directions and many normal directions. we expect the classiﬁcation function to change rapidly as it moves in the direction normal to the manifold, and not to change as it moves along the class manifold. both tangent propagation and the manifold tangent classiﬁer regularize f ( x ) to not change very much asx moves along the manifold. tangent propagation requires the user to manually specify functions that compute the tangent directions ( such as specifying that small translations of images remain in the same class manifold ) while the manifold


f ( x ) to not change very much asx moves along the manifold. tangent propagation requires the user to manually specify functions that compute the tangent directions ( such as specifying that small translations of images remain in the same class manifold ) while the manifold tangent classiﬁer estimates the manifold tangent directions by training an autoencoder to ﬁt the training data. the use of autoencoders to estimate manifolds will be described in chapter. 14 estimate the manifold tangent vectors. the manifold tangent classiﬁer makes use of this technique to avoid needing user - speciﬁed tangent vectors. as illustrated in ﬁgure, these estimated tangent vectors go beyond the classical invariants 14. 10 that arise out of the geometry of images ( such as translation, rotation and scaling ) and include factors that must be learned because they are object - speciﬁc ( such as moving body parts ). the algorithm proposed with the manifold tangent classiﬁer is therefore simple : ( 1 ) use an autoencoder to learn the manifold structure by unsupervised learning, and ( 2 ) use these tangents to regularize a neural net classiﬁer as in tangent prop ( equation ). 7. 67 this chapter has described most of the


##r to learn the manifold structure by unsupervised learning, and ( 2 ) use these tangents to regularize a neural net classiﬁer as in tangent prop ( equation ). 7. 67 this chapter has described most of the general strategies used to regularize neural networks. regularization is a central theme of machine learning and as such 272


chapter 7. regularization for deep learning will be revisited periodically by most of the remaining chapters. another central theme of machine learning is optimization, described next. 273


chapter 8 optimization for training deep models deep learning algorithms involve optimization in many contexts. for example, performing inference in models such as pca involves solving an optimization problem. we often use analytical optimization to write proofs or design algorithms. of all of the many optimization problems involved in deep learning, the most [UNK] is neural network training. it is quite common to invest days to months of time on hundreds of machines in order to solve even a single instance of the neural network training problem. because this problem is so important and so expensive, a specialized set of optimization techniques have been developed for solving it. this chapter presents these optimization techniques for neural network training. if you are unfamiliar with the basic principles of gradient - based optimization, we suggest reviewing chapter. that chapter includes a brief overview of numerical 4 optimization in general. this chapter focuses on one particular case of optimization : ﬁnding the param - eters θ of a neural network that signiﬁcantly reduce a cost function j ( θ ), which typically includes a performance measure evaluated on the entire training set as well as additional regularization terms. we begin with a description of how optimization used as a training algorithm for a machine learning task [UNK] from pure optimization. next, we present several of the concrete challenges that make optimization of


evaluated on the entire training set as well as additional regularization terms. we begin with a description of how optimization used as a training algorithm for a machine learning task [UNK] from pure optimization. next, we present several of the concrete challenges that make optimization of neural networks [UNK]. we then deﬁne several practical algorithms, including both optimization algorithms themselves and strategies for initializing the parameters. more advanced algorithms adapt their learning rates during training or leverage information contained in 274


chapter 8. optimization for training deep models the second derivatives of the cost function. finally, we conclude with a review of several optimization strategies that are formed by combining simple optimization algorithms into higher - level procedures. 8. 1 how learning [UNK] from pure optimization optimization algorithms used for training of deep models [UNK] from traditional optimization algorithms in several ways. machine learning usually acts indirectly. in most machine learning scenarios, we care about some performance measure p, that is deﬁned with respect to the test set and may also be intractable. we therefore optimize p only indirectly. we reduce a [UNK] cost function j ( θ ) in the hope that doing so will improve p. this is in contrast to pure optimization, where minimizing j is a goal in and of itself. optimization algorithms for training deep models also typically include some specialization on the speciﬁc structure of machine learning objective functions. typically, the cost function can be written as an average over the training set, such as j ( ) = θ e ( ) [UNK] x, y [UNK] f, y, ( ( ; ) x θ ) ( 8. 1 ) where l is the per - example loss function, f ( x ; θ ) is the predicted output when the input is x, [UNK] is the empirical


x, y [UNK] f, y, ( ( ; ) x θ ) ( 8. 1 ) where l is the per - example loss function, f ( x ; θ ) is the predicted output when the input is x, [UNK] is the empirical distribution. in the supervised learning case, y is the target output. throughout this chapter, we develop the unregularized supervised case, where the arguments to l are f ( x ; θ ) and y. however, it is trivial to extend this development, for example, to include θ or x as arguments, or to exclude y as arguments, in order to develop various forms of regularization or unsupervised learning. equation deﬁnes an objective function with respect to the training set. we 8. 1 would usually prefer to minimize the corresponding objective function where the expectation is taken across the data generating distribution pdata rather than just over the ﬁnite training set : j∗ ( ) = θ e ( ) x, y [UNK] f, y. ( ( ; ) x θ ) ( 8. 2 ) 8. 1. 1 empirical risk minimization the goal of a machine learning algorithm is to reduce the expected generalization error given by equation. this quantity is known as the 8. 2


( ( ; ) x θ ) ( 8. 2 ) 8. 1. 1 empirical risk minimization the goal of a machine learning algorithm is to reduce the expected generalization error given by equation. this quantity is known as the 8. 2 risk. we emphasize here that the expectation is taken over the true underlying distribution pdata. if we knew the true distribution pdata ( x, y ), risk minimization would be an optimization task 275


chapter 8. optimization for training deep models solvable by an optimization algorithm. however, when we do not know pdata ( x, y ) but only have a training set of samples, we have a machine learning problem. the simplest way to convert a machine learning problem back into an op - timization problem is to minimize the expected loss on the training set. this means replacing the true distribution p ( x, y ) with the empirical distribution [UNK] ( x, y ) deﬁned by the training set. we now minimize the empirical risk ex, [UNK] ( ) x, y [ ( ( ; ) ) ] = l f x θ, y 1 m m i = 1 l f ( ( x ( ) i ; ) θ, y ( ) i ) ( 8. 3 ) where is the number of training examples. m the training process based on minimizing this average training error is known as empirical risk minimization. in this setting, machine learning is still very similar to straightforward optimization. rather than optimizing the risk directly, we optimize the empirical risk, and hope that the risk decreases signiﬁcantly as well. a variety of theoretical results establish conditions under which the true risk can be expected to decrease by various amounts. however


##imizing the risk directly, we optimize the empirical risk, and hope that the risk decreases signiﬁcantly as well. a variety of theoretical results establish conditions under which the true risk can be expected to decrease by various amounts. however, empirical risk minimization is prone to overﬁtting. models with high capacity can simply memorize the training set. in many cases, empirical risk minimization is not really feasible. the most [UNK] modern optimization algorithms are based on gradient descent, but many useful loss functions, such as 0 - 1 loss, have no useful derivatives ( the derivative is either zero or undeﬁned everywhere ). these two problems mean that, in the context of deep learning, we rarely use empirical risk minimization. instead, we must use a slightly [UNK] approach, in which the quantity that we actually optimize is even more [UNK] from the quantity that we truly want to optimize. 8. 1. 2 surrogate loss functions and early stopping sometimes, the loss function we actually care about ( say classiﬁcation error ) is not one that can be optimized [UNK]. for example, exactly minimizing expected 0 - 1 loss is typically intractable ( exponential in the input dimension ), even for


function we actually care about ( say classiﬁcation error ) is not one that can be optimized [UNK]. for example, exactly minimizing expected 0 - 1 loss is typically intractable ( exponential in the input dimension ), even for a linear classiﬁer ( marcotte and savard 1992, ). in such situations, one typically optimizes a surrogate loss function instead, which acts as a proxy but has advantages. for example, the negative log - likelihood of the correct class is typically used as a surrogate for the 0 - 1 loss. the negative log - likelihood allows the model to estimate the conditional probability of the classes, given the input, and if the model can do that well, then it can pick the classes that yield the least classiﬁcation error in expectation. 276


chapter 8. optimization for training deep models in some cases, a surrogate loss function actually results in being able to learn more. for example, the test set 0 - 1 loss often continues to decrease for a long time after the training set 0 - 1 loss has reached zero, when training using the log - likelihood surrogate. this is because even when the expected 0 - 1 loss is zero, one can improve the robustness of the classiﬁer by further pushing the classes apart from each other, obtaining a more conﬁdent and reliable classiﬁer, thus extracting more information from the training data than would have been possible by simply minimizing the average 0 - 1 loss on the training set. a very important [UNK] between optimization in general and optimization as we use it for training algorithms is that training algorithms do not usually halt at a local minimum. instead, a machine learning algorithm usually minimizes a surrogate loss function but halts when a convergence criterion based on early stopping ( section ) is satisﬁed. typically the early stopping criterion is based 7. 8 on the true underlying loss function, such as 0 - 1 loss measured on a validation set, and is designed to cause the algorithm to halt whenever overﬁtting begins to occur. training


##ﬁed. typically the early stopping criterion is based 7. 8 on the true underlying loss function, such as 0 - 1 loss measured on a validation set, and is designed to cause the algorithm to halt whenever overﬁtting begins to occur. training often halts while the surrogate loss function still has large derivatives, which is very [UNK] from the pure optimization setting, where an optimization algorithm is considered to have converged when the gradient becomes very small. 8. 1. 3 batch and minibatch algorithms one aspect of machine learning algorithms that separates them from general optimization algorithms is that the objective function usually decomposes as a sum over the training examples. optimization algorithms for machine learning typically compute each update to the parameters based on an expected value of the cost function estimated using only a subset of the terms of the full cost function. for example, maximum likelihood estimation problems, when viewed in log space, decompose into a sum over each example : θml = arg max θ m i = 1 log pmodel ( x ( ) i, y ( ) i ; ) θ. ( 8. 4 ) maximizing this sum is equivalent to maximizing the expectation over the empirical distribution deﬁned by the training set : j ( ) = θ ex, [UNK]


( x ( ) i, y ( ) i ; ) θ. ( 8. 4 ) maximizing this sum is equivalent to maximizing the expectation over the empirical distribution deﬁned by the training set : j ( ) = θ ex, [UNK] log pmodel ( ; ) x, y θ. ( 8. 5 ) most of the properties of the objective function j used by most of our opti - mization algorithms are also expectations over the training set. for example, the 277


chapter 8. optimization for training deep models most commonly used property is the gradient : ∇θj ( ) = θ ex, [UNK] log pmodel ( ; ) x, y θ. ( 8. 6 ) computing this expectation exactly is very expensive because it requires evaluating the model on every example in the entire dataset. in practice, we can compute these expectations by randomly sampling a small number of examples from the dataset, then taking the average over only those examples. recall that the standard error of the mean ( equation ) estimated from 5. 46 n samples is given by σ / √n, where σ is the true standard deviation of the value of the samples. the denominator of √n shows that there are less than linear returns to using more examples to estimate the gradient. compare two hypothetical estimates of the gradient, one based on 100 examples and another based on 10, 000 examples. the latter requires 100 times more computation than the former, but reduces the standard error of the mean only by a factor of 10. most optimization algorithms converge much faster ( in terms of total computation, not in terms of number of updates ) if they are allowed to rapidly compute approximate estimates of the gradient rather than slowly computing the exact gradient. another consideration motivating statistical estimation of


. most optimization algorithms converge much faster ( in terms of total computation, not in terms of number of updates ) if they are allowed to rapidly compute approximate estimates of the gradient rather than slowly computing the exact gradient. another consideration motivating statistical estimation of the gradient from a small number of samples is redundancy in the training set. in the worst case, all m samples in the training set could be identical copies of each other. a sampling - based estimate of the gradient could compute the correct gradient with a single sample, using m times less computation than the naive approach. in practice, we are unlikely to truly encounter this worst - case situation, but we may ﬁnd large numbers of examples that all make very similar contributions to the gradient. optimization algorithms that use the entire training set are called batch or deterministic gradient methods, because they process all of the training examples simultaneously in a large batch. this terminology can be somewhat confusing because the word “ batch ” is also often used to describe the minibatch used by minibatch stochastic gradient descent. typically the term “ batch gradient descent ” implies the use of the full training set, while the use of the term “ batch ” to describe a group of examples does not. for example, it is very common


##batch stochastic gradient descent. typically the term “ batch gradient descent ” implies the use of the full training set, while the use of the term “ batch ” to describe a group of examples does not. for example, it is very common to use the term “ batch size ” to describe the size of a minibatch. optimization algorithms that use only a single example at a time are sometimes called stochastic or sometimes online methods. the term online is usually reserved for the case where the examples are drawn from a stream of continually created examples rather than from a ﬁxed - size training set over which several passes are made. most algorithms used for deep learning fall somewhere in between, using more 278


chapter 8. optimization for training deep models than one but less than all of the training examples. these were traditionally called minibatch or minibatch stochastic methods and it is now common to simply call them stochastic methods. the canonical example of a stochastic method is stochastic gradient descent, presented in detail in section. 8. 3. 1 minibatch sizes are generally driven by the following factors : • larger batches provide a more accurate estimate of the gradient, but with less than linear returns. • multicore architectures are usually underutilized by extremely small batches. this motivates using some absolute minimum batch size, below which there is no reduction in the time to process a minibatch. • if all examples in the batch are to be processed in parallel ( as is typically the case ), then the amount of memory scales with the batch size. for many hardware setups this is the limiting factor in batch size. • some kinds of hardware achieve better runtime with speciﬁc sizes of arrays. especially when using gpus, it is common for power of 2 batch sizes to [UNK] better runtime. typical power of 2 batch sizes range from 32 to 256, with 16 sometimes being attempted for large models.


speciﬁc sizes of arrays. especially when using gpus, it is common for power of 2 batch sizes to [UNK] better runtime. typical power of 2 batch sizes range from 32 to 256, with 16 sometimes being attempted for large models. • small batches can [UNK] a regularizing [UNK] (, ), wilson and martinez 2003 perhaps due to the noise they add to the learning process. generalization error is often best for a batch size of 1. training with such a small batch size might require a small learning rate to maintain stability due to the high variance in the estimate of the gradient. the total runtime can be very high due to the need to make more steps, both because of the reduced learning rate and because it takes more steps to observe the entire training set. [UNK] kinds of algorithms use [UNK] kinds of information from the mini - batch in [UNK] ways. some algorithms are more sensitive to sampling error than others, either because they use information that is [UNK] to estimate accurately with few samples, or because they use information in ways that amplify sampling errors more. methods that compute updates based only on the gradient g are usually relatively robust and can handle smaller batch sizes like 100. second - order methods, which use also the hessian matrix h and compute updates


information in ways that amplify sampling errors more. methods that compute updates based only on the gradient g are usually relatively robust and can handle smaller batch sizes like 100. second - order methods, which use also the hessian matrix h and compute updates such as h−1g, typically require much larger batch sizes like 10, 000. these large batch sizes are required to minimize ﬂuctuations in the estimates of h−1g. suppose that h is estimated perfectly but has a poor condition number. multiplication by 279


chapter 8. optimization for training deep models h or its inverse ampliﬁes pre - existing errors, in this case, estimation errors in g. very small changes in the estimate of g can thus cause large changes in the update h−1g, even if h were estimated perfectly. of course, h will be estimated only approximately, so the update h−1g will contain even more error than we would predict from applying a poorly conditioned operation to the estimate of. g it is also crucial that the minibatches be selected randomly. computing an unbiased estimate of the expected gradient from a set of samples requires that those samples be independent. we also wish for two subsequent gradient estimates to be independent from each other, so two subsequent minibatches of examples should also be independent from each other. many datasets are most naturally arranged in a way where successive examples are highly correlated. for example, we might have a dataset of medical data with a long list of blood sample test results. this list might be arranged so that ﬁrst we have ﬁve blood samples taken at [UNK] times from the ﬁrst patient, then we have three blood samples taken from the second patient, then the blood samples from the third patient, and so on. if we were to draw examples


that ﬁrst we have ﬁve blood samples taken at [UNK] times from the ﬁrst patient, then we have three blood samples taken from the second patient, then the blood samples from the third patient, and so on. if we were to draw examples in order from this list, then each of our minibatches would be extremely biased, because it would represent primarily one patient out of the many patients in the dataset. in cases such as these where the order of the dataset holds some signiﬁcance, it is necessary to [UNK] the examples before selecting minibatches. for very large datasets, for example datasets containing billions of examples in a data center, it can be impractical to sample examples truly uniformly at random every time we want to construct a minibatch. fortunately, in practice it is usually [UNK] to [UNK] the order of the dataset once and then store it in [UNK] fashion. this will impose a ﬁxed set of possible minibatches of consecutive examples that all models trained thereafter will use, and each individual model will be forced to reuse this ordering every time it passes through the training data. however, this deviation from true random selection does not seem to have a signiﬁcant detrimental [UNK]. failing to


trained thereafter will use, and each individual model will be forced to reuse this ordering every time it passes through the training data. however, this deviation from true random selection does not seem to have a signiﬁcant detrimental [UNK]. failing to ever [UNK] the examples in any way can seriously reduce the [UNK] of the algorithm. many optimization problems in machine learning decompose over examples well enough that we can compute entire separate updates over [UNK] examples in parallel. in other words, we can compute the update that minimizes j ( x ) for one minibatch of examples x at the same time that we compute the update for several other minibatches. such asynchronous parallel distributed approaches are discussed further in section. 12. 1. 3 an interesting motivation for minibatch stochastic gradient descent is that it follows the gradient of the true generalization error ( equation ) so long as no 8. 2 examples are repeated. most implementations of minibatch stochastic gradient 280


chapter 8. optimization for training deep models descent [UNK] the dataset once and then pass through it multiple times. on the ﬁrst pass, each minibatch is used to compute an unbiased estimate of the true generalization error. on the second pass, the estimate becomes biased because it is formed by re - sampling values that have already been used, rather than obtaining new fair samples from the data generating distribution. the fact that stochastic gradient descent minimizes generalization error is easiest to see in the online learning case, where examples or minibatches are drawn from a stream of data. in other words, instead of receiving a ﬁxed - size training set, the learner is similar to a living being who sees a new example at each instant, with every example ( x, y ) coming from the data generating distribution p data ( x, y ). in this scenario, examples are never repeated ; every experience is a fair sample from pdata. the equivalence is easiest to derive when both x and y are discrete. in this case, the generalization error ( equation ) can be written as a sum 8. 2 j∗ ( ) = θ x y pdata ( ) ( ( ; ) ) x, y l f x θ, y,


are discrete. in this case, the generalization error ( equation ) can be written as a sum 8. 2 j∗ ( ) = θ x y pdata ( ) ( ( ; ) ) x, y l f x θ, y, ( 8. 7 ) with the exact gradient g = ∇θj ∗ ( ) = θ x y pdata ( ) x, y ∇θl f, y. ( ( ; ) x θ ) ( 8. 8 ) we have already seen the same fact demonstrated for the log - likelihood in equa - tion and equation ; we observe now that this holds for other functions 8. 5 8. 6 l besides the likelihood. a similar result can be derived when x and y are continuous, under mild assumptions regarding pdata and. l hence, we can obtain an unbiased estimator of the exact gradient of the generalization error by sampling a minibatch of examples { x ( 1 ),... x ( ) m } with cor - responding targets y ( ) i from the data generating distribution pdata, and computing the gradient of the loss with respect to the parameters for that minibatch : [UNK] = 1 m∇θ i l f ( ( x ( ) i ;


##r - responding targets y ( ) i from the data generating distribution pdata, and computing the gradient of the loss with respect to the parameters for that minibatch : [UNK] = 1 m∇θ i l f ( ( x ( ) i ; ) θ, y ( ) i ). ( 8. 9 ) updating in the direction of θ [UNK] performs sgd on the generalization error. of course, this interpretation only applies when examples are not reused. nonetheless, it is usually best to make several passes through the training set, unless the training set is extremely large. when multiple such epochs are used, only the ﬁrst epoch follows the unbiased gradient of the generalization error, but 281


chapter 8. optimization for training deep models of course, the additional epochs usually provide enough beneﬁt due to decreased training error to [UNK] the harm they cause by increasing the gap between training error and test error. with some datasets growing rapidly in size, faster than computing power, it is becoming more common for machine learning applications to use each training example only once or even to make an incomplete pass through the training set. when using an extremely large training set, overﬁtting is not an issue, so underﬁtting and computational [UNK] become the predominant concerns. see also ( ) for a discussion of the [UNK] of computational bottou and bousquet 2008 bottlenecks on generalization error, as the number of training examples grows. 8. 2 challenges in neural network optimization optimization in general is an extremely [UNK] task. traditionally, machine learning has avoided the [UNK] of general optimization by carefully designing the objective function and constraints to ensure that the optimization problem is convex. when training neural networks, we must confront the general non - convex case. even convex optimization is not without its complications. in this section, we summarize several of the most prominent challenges involved in optimization for training deep models. 8. 2. 1 ill - conditioning some challenges arise even when optimi


- convex case. even convex optimization is not without its complications. in this section, we summarize several of the most prominent challenges involved in optimization for training deep models. 8. 2. 1 ill - conditioning some challenges arise even when optimizing convex functions. of these, the most prominent is ill - conditioning of the hessian matrix h. this is a very general problem in most numerical optimization, convex or otherwise, and is described in more detail in section. 4. 3. 1 the ill - conditioning problem is generally believed to be present in neural network training problems. ill - conditioning can manifest by causing sgd to get “ stuck ” in the sense that even very small steps increase the cost function. recall from equation that a second - order taylor series expansion of the 4. 9 cost function predicts that a gradient descent step of will add −g 1 22ghg g −g ( 8. 10 ) to the cost. ill - conditioning of the gradient becomes a problem when 1 22ghg exceeds gg. to determine whether ill - conditioning is detrimental to a neural network training task, one can monitor the squared gradient norm gg and 282


chapter 8. optimization for training deep models −50 0 50 100 150 200 250 training time ( epochs ) −2 0 2 4 6 8 10 12 14 16 gradient norm 0 50 100 150 200 250 training time ( epochs ) 0 1. 0 2. 0 3. 0 4. 0 5. 0 6. 0 7. 0 8. 0 9. 1 0. classiﬁcation error rate figure 8. 1 : gradient descent often does not arrive at a critical point of any kind. in this example, the gradient norm increases throughout training of a convolutional network used for object detection. ( left ) a scatterplot showing how the norms of individual gradient evaluations are distributed over time. to improve legibility, only one gradient norm is plotted per epoch. the running average of all gradient norms is plotted as a solid curve. the gradient norm clearly increases over time, rather than decreasing as we would expect if the training process converged to a critical point. despite the increasing ( right ) gradient, the training process is reasonably successful. the validation set classiﬁcation error decreases to a low level. the ghg term. in many cases, the gradient norm does not shrink signiﬁcantly throughout learning, but the g


) gradient, the training process is reasonably successful. the validation set classiﬁcation error decreases to a low level. the ghg term. in many cases, the gradient norm does not shrink signiﬁcantly throughout learning, but the ghg term grows by more than an order of magnitude. the result is that learning becomes very slow despite the presence of a strong gradient because the learning rate must be shrunk to compensate for even stronger curvature. figure shows an example of the gradient increasing signiﬁcantly 8. 1 during the successful training of a neural network. though ill - conditioning is present in other settings besides neural network training, some of the techniques used to combat it in other contexts are less applicable to neural networks. for example, newton ’ s method is an excellent tool for minimizing convex functions with poorly conditioned hessian matrices, but in the subsequent sections we will argue that newton ’ s method requires signiﬁcant modiﬁcation before it can be applied to neural networks. 8. 2. 2 local minima one of the most prominent features of a convex optimization problem is that it can be reduced to the problem of ﬁnding a local minimum. any local minimum is 283


chapter 8. optimization for training deep models guaranteed to be a global minimum. some convex functions have a ﬂat region at the bottom rather than a single global minimum point, but any point within such a ﬂat region is an acceptable solution. when optimizing a convex function, we know that we have reached a good solution if we ﬁnd a critical point of any kind. with non - convex functions, such as neural nets, it is possible to have many local minima. indeed, nearly any deep model is essentially guaranteed to have an extremely large number of local minima. however, as we will see, this is not necessarily a major problem. neural networks and any models with multiple equivalently parametrized latent variables all have multiple local minima because of the model identiﬁability problem. a model is said to be identiﬁable if a [UNK] large training set can rule out all but one setting of the model ’ s parameters. models with latent variables are often not identiﬁable because we can obtain equivalent models by exchanging latent variables with each other. for example, we could take a neural network and modify layer 1 by swapping the incoming weight vector for unit i with the incoming weight vector for unit j, then


##ﬁable because we can obtain equivalent models by exchanging latent variables with each other. for example, we could take a neural network and modify layer 1 by swapping the incoming weight vector for unit i with the incoming weight vector for unit j, then doing the same for the outgoing weight vectors. if we have m layers with n units each, then there are n! m ways of arranging the hidden units. this kind of non - identiﬁability is known as weight space symmetry. in addition to weight space symmetry, many kinds of neural networks have additional causes of non - identiﬁability. for example, in any rectiﬁed linear or maxout network, we can scale all of the incoming weights and biases of a unit by α if we also scale all of its outgoing weights by 1 α. this means that — if the cost function does not include terms such as weight decay that depend directly on the weights rather than the models ’ outputs — every local minimum of a rectiﬁed linear or maxout network lies on an ( m n × ) - dimensional hyperbola of equivalent local minima. these model identiﬁability issues mean that there can be an extremely large or even uncountably inﬁnite amount of


or maxout network lies on an ( m n × ) - dimensional hyperbola of equivalent local minima. these model identiﬁability issues mean that there can be an extremely large or even uncountably inﬁnite amount of local minima in a neural network cost function. however, all of these local minima arising from non - identiﬁability are equivalent to each other in cost function value. as a result, these local minima are not a problematic form of non - convexity. local minima can be problematic if they have high cost in comparison to the global minimum. one can construct small neural networks, even without hidden units, that have local minima with higher cost than the global minimum ( sontag and sussman 1989 brady 1989 gori and tesi 1992, ; et al., ;, ). if local minima with high cost are common, this could pose a serious problem for gradient - based optimization algorithms. it remains an open question whether there are many local minima of high cost 284


chapter 8. optimization for training deep models for networks of practical interest and whether optimization algorithms encounter them. for many years, most practitioners believed that local minima were a common problem plaguing neural network optimization. today, that does not appear to be the case. the problem remains an active area of research, but experts now suspect that, for [UNK] large neural networks, most local minima have a low cost function value, and that it is not important to ﬁnd a true global minimum rather than to ﬁnd a point in parameter space that has low but not minimal cost (, ;, ;, ; saxe et al. 2013 dauphin et al. 2014 goodfellow et al. 2015 choromanska et al., ). 2014 many practitioners attribute nearly all [UNK] with neural network optimiza - tion to local minima. we encourage practitioners to carefully test for speciﬁc problems. a test that can rule out local minima as the problem is to plot the norm of the gradient over time. if the norm of the gradient does not shrink to insigniﬁcant size, the problem is neither local minima nor any other kind of critical point. this kind of negative test can rule out local minima. in high dimensional spaces,


if the norm of the gradient does not shrink to insigniﬁcant size, the problem is neither local minima nor any other kind of critical point. this kind of negative test can rule out local minima. in high dimensional spaces, it can be very [UNK] to positively establish that local minima are the problem. many structures other than local minima also have small gradients. 8. 2. 3 plateaus, saddle points and other flat regions for many high - dimensional non - convex functions, local minima ( and maxima ) are in fact rare compared to another kind of point with zero gradient : a saddle point. some points around a saddle point have greater cost than the saddle point, while others have a lower cost. at a saddle point, the hessian matrix has both positive and negative eigenvalues. points lying along eigenvectors associated with positive eigenvalues have greater cost than the saddle point, while points lying along negative eigenvalues have lower value. we can think of a saddle point as being a local minimum along one cross - section of the cost function and a local maximum along another cross - section. see ﬁgure for an illustration. 4. 5 many classes of random functions exhibit the following behavior :


think of a saddle point as being a local minimum along one cross - section of the cost function and a local maximum along another cross - section. see ﬁgure for an illustration. 4. 5 many classes of random functions exhibit the following behavior : in low - dimensional spaces, local minima are common. in higher dimensional spaces, local minima are rare and saddle points are more common. for a function f : rn →r of this type, the expected ratio of the number of saddle points to local minima grows exponentially with n. to understand the intuition behind this behavior, observe that the hessian matrix at a local minimum has only positive eigenvalues. the hessian matrix at a saddle point has a mixture of positive and negative eigenvalues. imagine that the sign of each eigenvalue is generated by ﬂipping a coin. in a single dimension, it is easy to obtain a local minimum by tossing a coin and getting heads once. in n - dimensional space, it is exponentially unlikely that all n coin tosses will 285


chapter 8. optimization for training deep models be heads. see ( ) for a review of the relevant theoretical work. dauphin et al. 2014 an amazing property of many random functions is that the eigenvalues of the hessian become more likely to be positive as we reach regions of lower cost. in our coin tossing analogy, this means we are more likely to have our coin come up heads n times if we are at a critical point with low cost. this means that local minima are much more likely to have low cost than high cost. critical points with high cost are far more likely to be saddle points. critical points with extremely high cost are more likely to be local maxima. this happens for many classes of random functions. does it happen for neural networks? ( ) showed theoretically that shallow autoencoders baldi and hornik 1989 ( feedforward networks trained to copy their input to their output, described in chapter ) with no nonlinearities have global minima and saddle points but no 14 local minima with higher cost than the global minimum. they observed without proof that these results extend to deeper networks without nonlinearities. the output of such networks is a linear function of their input, but they are useful to study as a model of nonlinear neural networks because


with higher cost than the global minimum. they observed without proof that these results extend to deeper networks without nonlinearities. the output of such networks is a linear function of their input, but they are useful to study as a model of nonlinear neural networks because their loss function is a non - convex function of their parameters. such networks are essentially just multiple matrices composed together. ( ) provided exact solutions saxe et al. 2013 to the complete learning dynamics in such networks and showed that learning in these models captures many of the qualitative features observed in the training of deep models with nonlinear activation functions. ( ) showed dauphin et al. 2014 experimentally that real neural networks also have loss functions that contain very many high - cost saddle points. choromanska 2014 et al. ( ) provided additional theoretical arguments, showing that another class of high - dimensional random functions related to neural networks does so as well. what are the implications of the proliferation of saddle points for training algo - rithms? for ﬁrst - order optimization algorithms that use only gradient information, the situation is unclear. the gradient can often become very small near a saddle point. on the other hand, gradient descent empirically seems to be able to escape saddle points in many cases. ( ) provided visualizations of


that use only gradient information, the situation is unclear. the gradient can often become very small near a saddle point. on the other hand, gradient descent empirically seems to be able to escape saddle points in many cases. ( ) provided visualizations of goodfellow et al. 2015 several learning trajectories of state - of - the - art neural networks, with an example given in ﬁgure. these visualizations show a ﬂattening of the cost function near 8. 2 a prominent saddle point where the weights are all zero, but they also show the gradient descent trajectory rapidly escaping this region. ( ) goodfellow et al. 2015 also argue that continuous - time gradient descent may be shown analytically to be repelled from, rather than attracted to, a nearby saddle point, but the situation may be [UNK] for more realistic uses of gradient descent. for newton ’ s method, it is clear that saddle points constitute a problem. 286


chapter 8. optimization for training deep models projection 2 of θ projection 1 of θ j ( ) θ figure 8. 2 : a visualization of the cost function of a neural network. image adapted with permission from goodfellow 2015 et al. ( ). these visualizations appear similar for feedforward neural networks, convolutional networks, and recurrent networks applied to real object recognition and natural language processing tasks. surprisingly, these visualizations usually do not show many conspicuous obstacles. prior to the success of stochastic gradient descent for training very large models beginning in roughly 2012, neural net cost function surfaces were generally believed to have much more non - convex structure than is revealed by these projections. the primary obstacle revealed by this projection is a saddle point of high cost near where the parameters are initialized, but, as indicated by the blue path, the sgd training trajectory escapes this saddle point readily. most of training time is spent traversing the relatively ﬂat valley of the cost function, which may be due to high noise in the gradient, poor conditioning of the hessian matrix in this region, or simply the need to circumnavigate the tall “ mountain ” visible in the ﬁgure via an indirect arcing path. 287


chapter 8. optimization for training deep models gradient descent is designed to move “ downhill ” and is not explicitly designed to seek a critical point. newton ’ s method, however, is designed to solve for a point where the gradient is zero. without appropriate modiﬁcation, it can jump to a saddle point. the proliferation of saddle points in high dimensional spaces presumably explains why second - order methods have not succeeded in replacing gradient descent for neural network training. ( ) introduced a dauphin et al. 2014 saddle - free newton method for second - order optimization and showed that it improves signiﬁcantly over the traditional version. second - order methods remain [UNK] to scale to large neural networks, but this saddle - free approach holds promise if it could be scaled. there are other kinds of points with zero gradient besides minima and saddle points. there are also maxima, which are much like saddle points from the perspective of optimization — many algorithms are not attracted to them, but unmodiﬁed newton ’ s method is. maxima of many classes of random functions become exponentially rare in high dimensional space, just like minima do. there may also be wide, ﬂat regions of constant value. in these locations, the gradient and also the hessian are all zero


maxima of many classes of random functions become exponentially rare in high dimensional space, just like minima do. there may also be wide, ﬂat regions of constant value. in these locations, the gradient and also the hessian are all zero. such degenerate locations pose major problems for all numerical optimization algorithms. in a convex problem, a wide, ﬂat region must consist entirely of global minima, but in a general optimization problem, such a region could correspond to a high value of the objective function. 8. 2. 4 [UNK] and exploding gradients neural networks with many layers often have extremely steep regions resembling [UNK], as illustrated in ﬁgure. these result from the multiplication of several 8. 3 large weights together. on the face of an extremely steep [UNK], the gradient update step can move the parameters extremely far, usually jumping [UNK] of the [UNK] altogether. 288


chapter 8. optimization for training deep models figure 8. 3 : the objective function for highly nonlinear deep neural networks or for recurrent neural networks often contains sharp nonlinearities in parameter space resulting from the multiplication of several parameters. these nonlinearities give rise to very high derivatives in some places. when the parameters get close to such a [UNK], a gradient descent update can catapult the parameters very far, possibly losing most of the optimization work that had been done. figure adapted with permission from pascanu et al. ( ). 2013 the [UNK] be dangerous whether we approach it from above or from below, but fortunately its most serious consequences can be avoided using the gradient clipping heuristic described in section. the basic idea is to recall that 10. 11. 1 the gradient does not specify the optimal step size, but only the optimal direction within an inﬁnitesimal region. when the traditional gradient descent algorithm proposes to make a very large step, the gradient clipping heuristic intervenes to reduce the step size to be small enough that it is less likely to go outside the region where the gradient indicates the direction of approximately steepest descent. [UNK] structures are most common in the cost functions for recurrent neural networks, because such models involve a multiplication of many


size to be small enough that it is less likely to go outside the region where the gradient indicates the direction of approximately steepest descent. [UNK] structures are most common in the cost functions for recurrent neural networks, because such models involve a multiplication of many factors, with one factor for each time step. long temporal sequences thus incur an extreme amount of multiplication. 8. 2. 5 long - term dependencies another [UNK] that neural network optimization algorithms must overcome arises when the computational graph becomes extremely deep. feedforward networks with many layers have such deep computational graphs. so do recurrent networks, described in chapter, which construct very deep computational graphs 10 289


chapter 8. optimization for training deep models by repeatedly applying the same operation at each time step of a long temporal sequence. repeated application of the same parameters gives rise to especially pronounced [UNK]. for example, suppose that a computational graph contains a path that consists of repeatedly multiplying by a matrix w. after t steps, this is equivalent to mul - tiplying by wt. suppose that w has an eigendecomposition w = v diag ( λ ) v −1. in this simple case, it is straightforward to see that w t = v λ v diag ( ) −1t = ( ) v diag λ tv −1. ( 8. 11 ) any eigenvalues λi that are not near an absolute value of will either explode if they 1 are greater than in magnitude or vanish if they are less than in magnitude. the 1 1 vanishing and exploding gradient problem refers to the fact that gradients through such a graph are also scaled according to diag ( λ ) t. vanishing gradients make it [UNK] to know which direction the parameters should move to improve the cost function, while exploding gradients can make learning unstable. the [UNK] structures described earlier that motivate gradient clipping are an example of the exploding gradient phenomenon


. vanishing gradients make it [UNK] to know which direction the parameters should move to improve the cost function, while exploding gradients can make learning unstable. the [UNK] structures described earlier that motivate gradient clipping are an example of the exploding gradient phenomenon. the repeated multiplication by w at each time step described here is very similar to the power method algorithm used to ﬁnd the largest eigenvalue of a matrix w and the corresponding eigenvector. from this point of view it is not surprising that xw t will eventually discard all components of x that are orthogonal to the principal eigenvector of. w recurrent networks use the same matrix w at each time step, but feedforward networks do not, so even very deep feedforward networks can largely avoid the vanishing and exploding gradient problem (, ). sussillo 2014 we defer a further discussion of the challenges of training recurrent networks until section, after recurrent networks have been described in more detail. 10. 7 8. 2. 6 inexact gradients most optimization algorithms are designed with the assumption that we have access to the exact gradient or hessian matrix. in practice, we usually only have a noisy or even biased estimate of these quantities. nearly every deep learning algorithm


. 6 inexact gradients most optimization algorithms are designed with the assumption that we have access to the exact gradient or hessian matrix. in practice, we usually only have a noisy or even biased estimate of these quantities. nearly every deep learning algorithm relies on sampling - based estimates at least insofar as using a minibatch of training examples to compute the gradient. in other cases, the objective function we want to minimize is actually intractable. when the objective function is intractable, typically its gradient is intractable as well. in such cases we can only approximate the gradient. these issues mostly arise 290


chapter 8. optimization for training deep models with the more advanced models in part. for example, contrastive divergence iii gives a technique for approximating the gradient of the intractable log - likelihood of a boltzmann machine. various neural network optimization algorithms are designed to account for imperfections in the gradient estimate. one can also avoid the problem by choosing a surrogate loss function that is easier to approximate than the true loss. 8. 2. 7 poor correspondence between local and global structure many of the problems we have discussed so far correspond to properties of the loss function at a single point — it can be [UNK] to make a single step if j ( θ ) is poorly conditioned at the current point θ, or if θ lies on a [UNK], or if θ is a saddle point hiding the opportunity to make progress downhill from the gradient. it is possible to overcome all of these problems at a single point and still perform poorly if the direction that results in the most improvement locally does not point toward distant regions of much lower cost. goodfellow 2015 et al. ( ) argue that much of the runtime of training is due to the length of the trajectory needed to arrive at the solution. figure shows that 8. 2 the learning trajectory spends most of its time tracing out a


goodfellow 2015 et al. ( ) argue that much of the runtime of training is due to the length of the trajectory needed to arrive at the solution. figure shows that 8. 2 the learning trajectory spends most of its time tracing out a wide arc around a mountain - shaped structure. much of research into the [UNK] of optimization has focused on whether training arrives at a global minimum, a local minimum, or a saddle point, but in practice neural networks do not arrive at a critical point of any kind. figure 8. 1 shows that neural networks often do not arrive at a region of small gradient. indeed, such critical points do not even necessarily exist. for example, the loss function −log p ( y | x ; θ ) can lack a global minimum point and instead asymptotically approach some value as the model becomes more conﬁdent. for a classiﬁer with discrete y and p ( y | x ) provided by a softmax, the negative log - likelihood can become arbitrarily close to zero if the model is able to correctly classify every example in the training set, but it is impossible to actually reach the value of zero. likewise, a model of real values p ( y | x ) = n ( y ; f


##ily close to zero if the model is able to correctly classify every example in the training set, but it is impossible to actually reach the value of zero. likewise, a model of real values p ( y | x ) = n ( y ; f ( θ ), β−1 ) can have negative log - likelihood that asymptotes to negative inﬁnity — if f ( θ ) is able to correctly predict the value of all training set y targets, the learning algorithm will increase β without bound. see ﬁgure for an example of a failure of local optimization to 8. 4 ﬁnd a good cost function value even in the absence of any local minima or saddle points. future research will need to develop further understanding of the factors that inﬂuence the length of the learning trajectory and better characterize the outcome 291


chapter 8. optimization for training deep models θ j ( ) θ figure 8. 4 : optimization based on local downhill moves can fail if the local surface does not point toward the global solution. here we provide an example of how this can occur, even if there are no saddle points and no local minima. this example cost function contains only asymptotes toward low values, not minima. the main cause of [UNK] in this case is being initialized on the wrong side of the “ mountain ” and not being able to traverse it. in higher dimensional space, learning algorithms can often circumnavigate such mountains but the trajectory associated with doing so may be long and result in excessive training time, as illustrated in ﬁgure. 8. 2 of the process. many existing research directions are aimed at ﬁnding good initial points for problems that have [UNK] global structure, rather than developing algorithms that use non - local moves. gradient descent and essentially all learning algorithms that are [UNK] for training neural networks are based on making small, local moves. the previous sections have primarily focused on how the correct direction of these local moves can be [UNK] to compute. we may be able to compute some properties of the objective function, such as its gradient, only approximately, with bias


making small, local moves. the previous sections have primarily focused on how the correct direction of these local moves can be [UNK] to compute. we may be able to compute some properties of the objective function, such as its gradient, only approximately, with bias or variance in our estimate of the correct direction. in these cases, local descent may or may not deﬁne a reasonably short path to a valid solution, but we are not actually able to follow the local descent path. the objective function may have issues such as poor conditioning or discontinuous gradients, causing the region where the gradient provides a good model of the objective function to be very small. in these cases, local descent with steps of size may deﬁne a reasonably short path to the solution, but we are only able to compute the local descent direction with steps of size δ. in these cases, local descent may or may not deﬁne a path to the solution, but the path contains many steps, so following the path incurs a 292


chapter 8. optimization for training deep models high computational cost. sometimes local information provides us no guide, when the function has a wide ﬂat region, or if we manage to land exactly on a critical point ( usually this latter scenario only happens to methods that solve explicitly for critical points, such as newton ’ s method ). in these cases, local descent does not deﬁne a path to a solution at all. in other cases, local moves can be too greedy and lead us along a path that moves downhill but away from any solution, as in ﬁgure, or along an unnecessarily long trajectory to the solution, as in ﬁgure. 8. 4 8. 2 currently, we do not understand which of these problems are most relevant to making neural network optimization [UNK], and this is an active area of research. regardless of which of these problems are most signiﬁcant, all of them might be avoided if there exists a region of space connected reasonably directly to a solution by a path that local descent can follow, and if we are able to initialize learning within that well - behaved region. this last view suggests research into choosing good initial points for traditional optimization algorithms to use. 8. 2. 8 theoretical limits of optimization several theoretical


that local descent can follow, and if we are able to initialize learning within that well - behaved region. this last view suggests research into choosing good initial points for traditional optimization algorithms to use. 8. 2. 8 theoretical limits of optimization several theoretical results show that there are limits on the performance of any optimization algorithm we might design for neural networks ( blum and rivest, 1992 judd 1989 wolpert and macready 1997 ;, ;, ). typically these results have little bearing on the use of neural networks in practice. some theoretical results apply only to the case where the units of a neural network output discrete values. however, most neural network units output smoothly increasing values that make optimization via local search feasible. some theoretical results show that there exist problem classes that are intractable, but it can be [UNK] to tell whether a particular problem falls into that class. other results show that ﬁnding a solution for a network of a given size is intractable, but in practice we can ﬁnd a solution easily by using a larger network for which many more parameter settings correspond to an acceptable solution. moreover, in the context of neural network training, we usually do not care about ﬁnding the exact minimum of a function, but seek only to reduce its value [UNK] to


using a larger network for which many more parameter settings correspond to an acceptable solution. moreover, in the context of neural network training, we usually do not care about ﬁnding the exact minimum of a function, but seek only to reduce its value [UNK] to obtain good generalization error. theoretical analysis of whether an optimization algorithm can accomplish this goal is extremely [UNK]. developing more realistic bounds on the performance of optimization algorithms therefore remains an important goal for machine learning research. 293


chapter 8. optimization for training deep models 8. 3 basic algorithms we have previously introduced the gradient descent ( section ) algorithm that 4. 3 follows the gradient of an entire training set downhill. this may be accelerated considerably by using stochastic gradient descent to follow the gradient of randomly selected minibatches downhill, as discussed in section and section. 5. 9 8. 1. 3 8. 3. 1 stochastic gradient descent stochastic gradient descent ( sgd ) and its variants are probably the most used optimization algorithms for machine learning in general and for deep learning in particular. as discussed in section, it is possible to obtain an unbiased 8. 1. 3 estimate of the gradient by taking the average gradient on a minibatch of m examples drawn i. i. d from the data generating distribution. algorithm shows how to follow this estimate of the gradient downhill. 8. 1 algorithm 8. 1 stochastic gradient descent ( sgd ) update at training iteration k require : learning rate k. require : initial parameter θ while do stopping criterion not met sample a minibatch of m examples from the training set { x ( 1 ),..., x ( ) m } with corresponding targets y ( ) i. compute gradient estimate : [UNK]


: initial parameter θ while do stopping criterion not met sample a minibatch of m examples from the training set { x ( 1 ),..., x ( ) m } with corresponding targets y ( ) i. compute gradient estimate : [UNK] ← + 1 m∇θ i l f ( ( x ( ) i ; ) θ, y ( ) i ) apply update : θ θ ← [UNK] end while a crucial parameter for the sgd algorithm is the learning rate. previously, we have described sgd as using a ﬁxed learning rate. in practice, it is necessary to gradually decrease the learning rate over time, so we now denote the learning rate at iteration as k k. this is because the sgd gradient estimator introduces a source of noise ( the random sampling of m training examples ) that does not vanish even when we arrive at a minimum. by comparison, the true gradient of the total cost function becomes small and then 0 when we approach and reach a minimum using batch gradient descent, so batch gradient descent can use a ﬁxed learning rate. a [UNK] condition to guarantee convergence of sgd is that ∞ k = 1 k = and ∞, ( 8. 12 ) 294


chapter 8. optimization for training deep models ∞ k = 1 2 k <. ∞ ( 8. 13 ) in practice, it is common to decay the learning rate linearly until iteration : τ k = ( 1 ) −α 0 + ατ ( 8. 14 ) with α = k τ. after iteration, it is common to leave constant. τ the learning rate may be chosen by trial and error, but it is usually best to choose it by monitoring learning curves that plot the objective function as a function of time. this is more of an art than a science, and most guidance on this subject should be regarded with some skepticism. when using the linear schedule, the parameters to choose are 0, τ, and τ. usually τ may be set to the number of iterations required to make a few hundred passes through the training set. usually τ should be set to roughly the value of 1 % 0. the main question is how to set 0. if it is too large, the learning curve will show violent oscillations, with the cost function often increasing signiﬁcantly. gentle oscillations are ﬁne, especially if training with a stochastic cost function such as the cost function arising from the use of dropout. if the


##cillations, with the cost function often increasing signiﬁcantly. gentle oscillations are ﬁne, especially if training with a stochastic cost function such as the cost function arising from the use of dropout. if the learning rate is too low, learning proceeds slowly, and if the initial learning rate is too low, learning may become stuck with a high cost value. typically, the optimal initial learning rate, in terms of total training time and the ﬁnal cost value, is higher than the learning rate that yields the best performance after the ﬁrst 100 iterations or so. therefore, it is usually best to monitor the ﬁrst several iterations and use a learning rate that is higher than the best - performing learning rate at this time, but not so high that it causes severe instability. the most important property of sgd and related minibatch or online gradient - based optimization is that computation time per update does not grow with the number of training examples. this allows convergence even when the number of training examples becomes very large. for a large enough dataset, sgd may converge to within some ﬁxed tolerance of its ﬁnal test set error before it has processed the entire training set. to study the convergence rate of an optimization algorithm it


of training examples becomes very large. for a large enough dataset, sgd may converge to within some ﬁxed tolerance of its ﬁnal test set error before it has processed the entire training set. to study the convergence rate of an optimization algorithm it is common to measure the excess error j ( θ ) −minθ j ( θ ), which is the amount that the current cost function exceeds the minimum possible cost. when sgd is applied to a convex problem, the excess error is o ( 1 √ k ) after k iterations, while in the strongly convex case it is o ( 1 k ). these bounds cannot be improved unless extra conditions are assumed. batch gradient descent enjoys better convergence rates than stochastic gradient descent in theory. however, the cramer - rao bound (, ;, cramer 1946 rao 1945 ) states that generalization error cannot decrease faster than o ( 1 k ). bottou 295


chapter 8. optimization for training deep models and bousquet 2008 ( ) argue that it therefore may not be worthwhile to pursue an optimization algorithm that converges faster than o ( 1 k ) for machine learning tasks — faster convergence presumably corresponds to overﬁtting. moreover, the asymptotic analysis obscures many advantages that stochastic gradient descent has after a small number of steps. with large datasets, the ability of sgd to make rapid initial progress while evaluating the gradient for only very few examples outweighs its slow asymptotic convergence. most of the algorithms described in the remainder of this chapter achieve beneﬁts that matter in practice but are lost in the constant factors obscured by the o ( 1 k ) asymptotic analysis. one can also trade [UNK] beneﬁts of both batch and stochastic gradient descent by gradually increasing the minibatch size during the course of learning. for more information on sgd, see ( ). bottou 1998 8. 3. 2 momentum while stochastic gradient descent remains a very popular optimization strategy, learning with it can sometimes be slow. the method of momentum ( polyak 1964, ) is designed to accelerate learning, especially in the face of high curvature, small but consistent


2 momentum while stochastic gradient descent remains a very popular optimization strategy, learning with it can sometimes be slow. the method of momentum ( polyak 1964, ) is designed to accelerate learning, especially in the face of high curvature, small but consistent gradients, or noisy gradients. the momentum algorithm accumulates an exponentially decaying moving average of past gradients and continues to move in their direction. the [UNK] of momentum is illustrated in ﬁgure. 8. 5 formally, the momentum algorithm introduces a variable v that plays the role of velocity — it is the direction and speed at which the parameters move through parameter space. the velocity is set to an exponentially decaying average of the negative gradient. the name momentum derives from a physical analogy, in which the negative gradient is a force moving a particle through parameter space, according to newton ’ s laws of motion. momentum in physics is mass times velocity. in the momentum learning algorithm, we assume unit mass, so the velocity vectorv may also be regarded as the momentum of the particle. a hyperparameter α ∈ [ 0, 1 ) determines how quickly the contributions of previous gradients exponentially decay. the update rule is given by : v v ←α −∇ θ 1 m m i = 1 l


momentum of the particle. a hyperparameter α ∈ [ 0, 1 ) determines how quickly the contributions of previous gradients exponentially decay. the update rule is given by : v v ←α −∇ θ 1 m m i = 1 l ( ( f x ( ) i ; ) θ, y ( ) i ), ( 8. 15 ) θ θ v ← +. ( 8. 16 ) the velocity v accumulates the gradient elements ∇θ 1 m m i = 1 l ( ( f x ( ) i ; ) θ, y ( ) i ). the larger α is relative to, the more previous gradients [UNK] the current direction. the sgd algorithm with momentum is given in algorithm. 8. 2 296


chapter 8. optimization for training deep models − − − 30 20 10 0 10 20 −30 −20 −10 0 10 20 figure 8. 5 : momentum aims primarily to solve two problems : poor conditioning of the hessian matrix and variance in the stochastic gradient. here, we illustrate how momentum overcomes the ﬁrst of these two problems. the contour lines depict a quadratic loss function with a poorly conditioned hessian matrix. the red path cutting across the contours indicates the path followed by the momentum learning rule as it minimizes this function. at each step along the way, we draw an arrow indicating the step that gradient descent would take at that point. we can see that a poorly conditioned quadratic objective looks like a long, narrow valley or canyon with steep sides. momentum correctly traverses the canyon lengthwise, while gradient steps waste time moving back and forth across the narrow axis of the canyon. compare also ﬁgure, which shows the behavior of gradient 4. 6 descent without momentum. 297


chapter 8. optimization for training deep models previously, the size of the step was simply the norm of the gradient multiplied by the learning rate. now, the size of the step depends on how large and how aligned a sequence of gradients are. the step size is largest when many successive gradients point in exactly the same direction. if the momentum algorithm always observes gradient g, then it will accelerate in the direction of −g, until reaching a terminal velocity where the size of each step is | | | | g 1 −α. ( 8. 17 ) it is thus helpful to think of the momentum hyperparameter in terms of 1 1−α. for example, α =. 9 corresponds to multiplying the maximum speed by relative to 10 the gradient descent algorithm. common values of α used in practice include. 5,. 9, and. 99. like the learning rate, α may also be adapted over time. typically it begins with a small value and is later raised. it is less important to adapt α over time than to shrink over time. algorithm 8. 2 stochastic gradient descent ( sgd ) with momentum require : learning rate, momentum parameter. α require : initial parameter, initial velocity. θ v while do stopping criterion not met sample a


over time than to shrink over time. algorithm 8. 2 stochastic gradient descent ( sgd ) with momentum require : learning rate, momentum parameter. α require : initial parameter, initial velocity. θ v while do stopping criterion not met sample a minibatch of m examples from the training set { x ( 1 ),..., x ( ) m } with corresponding targets y ( ) i. compute gradient estimate : g ←1 m∇θ i l f ( ( x ( ) i ; ) θ, y ( ) i ) compute velocity update : v v g ←α − apply update : θ θ v ← + end while we can view the momentum algorithm as simulating a particle subject to continuous - time newtonian dynamics. the physical analogy can help to build intuition for how the momentum and gradient descent algorithms behave. the position of the particle at any point in time is given by θ ( t ). the particle experiences net force. this force causes the particle to accelerate : f ( ) t f ( ) = t ∂2 ∂t2 θ ( ) t. ( 8. 18 ) rather than viewing this as a second - order [UNK] equation of the position, we can introduce the variable v ( t ) representing the velocity of the particle at


( ) = t ∂2 ∂t2 θ ( ) t. ( 8. 18 ) rather than viewing this as a second - order [UNK] equation of the position, we can introduce the variable v ( t ) representing the velocity of the particle at time t and rewrite the newtonian dynamics as a ﬁrst - order [UNK] equation : v ( ) = t ∂ ∂tθ ( ) t, ( 8. 19 ) 298


chapter 8. optimization for training deep models f ( ) = t ∂ ∂tv ( ) t. ( 8. 20 ) the momentum algorithm then consists of solving the [UNK] equations via numerical simulation. a simple numerical method for solving [UNK] equations is euler ’ s method, which simply consists of simulating the dynamics deﬁned by the equation by taking small, ﬁnite steps in the direction of each gradient. this explains the basic form of the momentum update, but what speciﬁcally are the forces? one force is proportional to the negative gradient of the cost function : −∇θj ( θ ). this force pushes the particle downhill along the cost function surface. the gradient descent algorithm would simply take a single step based on each gradient, but the newtonian scenario used by the momentum algorithm instead uses this force to alter the velocity of the particle. we can think of the particle as being like a hockey puck sliding down an icy surface. whenever it descends a steep part of the surface, it gathers speed and continues sliding in that direction until it begins to go uphill again. one other force is necessary. if the only force is the gradient of the cost function, then the particle might never come to rest. imagine a hockey puck sliding down one side of


continues sliding in that direction until it begins to go uphill again. one other force is necessary. if the only force is the gradient of the cost function, then the particle might never come to rest. imagine a hockey puck sliding down one side of a valley and straight up the other side, oscillating back and forth forever, assuming the ice is perfectly frictionless. to resolve this problem, we add one other force, proportional to −v ( t ). in physics terminology, this force corresponds to viscous drag, as if the particle must push through a resistant medium such as syrup. this causes the particle to gradually lose energy over time and eventually converge to a local minimum. why do we use −v ( t ) and viscous drag in particular? part of the reason to use −v ( t ) is mathematical convenience — an integer power of the velocity is easy to work with. however, other physical systems have other kinds of drag based on other integer powers of the velocity. for example, a particle traveling through the air experiences turbulent drag, with force proportional to the square of the velocity, while a particle moving along the ground experiences dry friction, with a force of constant magnitude. we can reject each of these options. turbulent drag, proportional to the


traveling through the air experiences turbulent drag, with force proportional to the square of the velocity, while a particle moving along the ground experiences dry friction, with a force of constant magnitude. we can reject each of these options. turbulent drag, proportional to the square of the velocity, becomes very weak when the velocity is small. it is not powerful enough to force the particle to come to rest. a particle with a non - zero initial velocity that experiences only the force of turbulent drag will move away from its initial position forever, with the distance from the starting point growing like o ( log t ). we must therefore use a lower power of the velocity. if we use a power of zero, representing dry friction, then the force is too strong. when the force due to the gradient of the cost function is small but non - zero, the constant force due to friction can cause the particle to come to rest before reaching a local minimum. viscous drag avoids both of these problems — it is weak enough 299


chapter 8. optimization for training deep models that the gradient can continue to cause motion until a minimum is reached, but strong enough to prevent motion if the gradient does not justify moving. 8. 3. 3 nesterov momentum sutskever 2013 et al. ( ) introduced a variant of the momentum algorithm that was inspired by nesterov ’ s accelerated gradient method (,, ). the nesterov 1983 2004 update rules in this case are given by : v v ←α −∇ θ 1 m m i = 1 l f x ( ( ) i ; + ) θ αv, y ( ) i, ( 8. 21 ) θ θ v ← +, ( 8. 22 ) where the parameters α and play a similar role as in the standard momentum method. the [UNK] between nesterov momentum and standard momentum is where the gradient is evaluated. with nesterov momentum the gradient is evaluated after the current velocity is applied. thus one can interpret nesterov momentum as attempting to add a correction factor to the standard method of momentum. the complete nesterov momentum algorithm is presented in algorithm. 8. 3 in the convex batch gradient case, nesterov momentum brings the rate of convergence of the excess error from o ( 1 / k ) ( after


the standard method of momentum. the complete nesterov momentum algorithm is presented in algorithm. 8. 3 in the convex batch gradient case, nesterov momentum brings the rate of convergence of the excess error from o ( 1 / k ) ( after k steps ) to o ( 1 / k2 ) as shown by nesterov 1983 ( ). unfortunately, in the stochastic gradient case, nesterov momentum does not improve the rate of convergence. algorithm 8. 3 stochastic gradient descent ( sgd ) with nesterov momentum require : learning rate, momentum parameter. α require : initial parameter, initial velocity. θ v while do stopping criterion not met sample a minibatch of m examples from the training set { x ( 1 ),..., x ( ) m } with corresponding labels y ( ) i. apply interim update : [UNK] θ v ← + α compute gradient ( at interim point ) : g ←1 [UNK] i l f ( ( x ( ) i ; [UNK] y ), ( ) i ) compute velocity update : v v g ←α − apply update : θ θ v ← + end while 300


chapter 8. optimization for training deep models 8. 4 parameter initialization strategies some optimization algorithms are not iterative by nature and simply solve for a solution point. other optimization algorithms are iterative by nature but, when applied to the right class of optimization problems, converge to acceptable solutions in an acceptable amount of time regardless of initialization. deep learning training algorithms usually do not have either of these luxuries. training algorithms for deep learning models are usually iterative in nature and thus require the user to specify some initial point from which to begin the iterations. moreover, training deep models is a [UNK] [UNK] task that most algorithms are strongly [UNK] by the choice of initialization. the initial point can determine whether the algorithm converges at all, with some initial points being so unstable that the algorithm encounters numerical [UNK] and fails altogether. when learning does converge, the initial point can determine how quickly learning converges and whether it converges to a point with high or low cost. also, points of comparable cost can have wildly varying generalization error, and the initial point can [UNK] the generalization as well. modern initialization strategies are simple and heuristic. designing improved initialization strategies is a [UNK] task because neural network optimization is not yet well understood. most initialization strategies are based


error, and the initial point can [UNK] the generalization as well. modern initialization strategies are simple and heuristic. designing improved initialization strategies is a [UNK] task because neural network optimization is not yet well understood. most initialization strategies are based on achieving some nice properties when the network is initialized. however, we do not have a good understanding of which of these properties are preserved under which circumstances after learning begins to proceed. a further [UNK] is that some initial points may be beneﬁcial from the viewpoint of optimization but detrimental from the viewpoint of generalization. our understanding of how the initial point [UNK] generalization is especially primitive, [UNK] little to no guidance for how to select the initial point. perhaps the only property known with complete certainty is that the initial parameters need to “ break symmetry ” between [UNK] units. if two hidden units with the same activation function are connected to the same inputs, then these units must have [UNK] initial parameters. if they have the same initial parameters, then a deterministic learning algorithm applied to a deterministic cost and model will constantly update both of these units in the same way. even if the model or training algorithm is capable of using stochasticity to compute [UNK] updates for [UNK] units ( for example, if one trains with


to a deterministic cost and model will constantly update both of these units in the same way. even if the model or training algorithm is capable of using stochasticity to compute [UNK] updates for [UNK] units ( for example, if one trains with dropout ), it is usually best to initialize each unit to compute a [UNK] function from all of the other units. this may help to make sure that no input patterns are lost in the null space of forward propagation and no gradient patterns are lost in the null space of back - propagation. the goal of having each unit compute a [UNK] function 301


chapter 8. optimization for training deep models motivates random initialization of the parameters. we could explicitly search for a large set of basis functions that are all mutually [UNK] from each other, but this often incurs a noticeable computational cost. for example, if we have at most as many outputs as inputs, we could use gram - schmidt orthogonalization on an initial weight matrix, and be guaranteed that each unit computes a very [UNK] function from each other unit. random initialization from a high - entropy distribution over a high - dimensional space is computationally cheaper and unlikely to assign any units to compute the same function as each other. typically, we set the biases for each unit to heuristically chosen constants, and initialize only the weights randomly. extra parameters, for example, parameters encoding the conditional variance of a prediction, are usually set to heuristically chosen constants much like the biases are. we almost always initialize all the weights in the model to values drawn randomly from a gaussian or uniform distribution. the choice of gaussian or uniform distribution does not seem to matter very much, but has not been exhaustively studied. the scale of the initial distribution, however, does have a large [UNK] on both the outcome of the


##ian or uniform distribution. the choice of gaussian or uniform distribution does not seem to matter very much, but has not been exhaustively studied. the scale of the initial distribution, however, does have a large [UNK] on both the outcome of the optimization procedure and on the ability of the network to generalize. larger initial weights will yield a stronger symmetry breaking [UNK], helping to avoid redundant units. they also help to avoid losing signal during forward or back - propagation through the linear component of each layer — larger values in the matrix result in larger outputs of matrix multiplication. initial weights that are too large may, however, result in exploding values during forward propagation or back - propagation. in recurrent networks, large weights can also result in chaos ( such extreme sensitivity to small perturbations of the input that the behavior of the deterministic forward propagation procedure appears random ). to some extent, the exploding gradient problem can be mitigated by gradient clipping ( thresholding the values of the gradients before performing a gradient descent step ). large weights may also result in extreme values that cause the activation function to saturate, causing complete loss of gradient through saturated units. these competing factors determine the ideal initial scale of the weights. the perspectives of regularization and optimization


descent step ). large weights may also result in extreme values that cause the activation function to saturate, causing complete loss of gradient through saturated units. these competing factors determine the ideal initial scale of the weights. the perspectives of regularization and optimization can give very [UNK] insights into how we should initialize a network. the optimization perspective suggests that the weights should be large enough to propagate information success - fully, but some regularization concerns encourage making them smaller. the use of an optimization algorithm such as stochastic gradient descent that makes small incremental changes to the weights and tends to halt in areas that are nearer to the initial parameters ( whether due to getting stuck in a region of low gradient, or 302


chapter 8. optimization for training deep models due to triggering some early stopping criterion based on overﬁtting ) expresses a prior that the ﬁnal parameters should be close to the initial parameters. recall from section that gradient descent with early stopping is equivalent to weight 7. 8 decay for some models. in the general case, gradient descent with early stopping is not the same as weight decay, but does provide a loose analogy for thinking about the [UNK] of initialization. we can think of initializing the parameters θ to θ0 as being similar to imposing a gaussian prior p ( θ ) with mean θ0. from this point of view, it makes sense to choose θ0 to be near 0. this prior says that it is more likely that units do not interact with each other than that they do interact. units interact only if the likelihood term of the objective function expresses a strong preference for them to interact. on the other hand, if we initialize θ0 to large values, then our prior speciﬁes which units should interact with each other, and how they should interact. some heuristics are available for choosing the initial scale of the weights. one heuristic is to initialize the weights of a fully connected layer with m inputs and n outputs by


should interact with each other, and how they should interact. some heuristics are available for choosing the initial scale of the weights. one heuristic is to initialize the weights of a fully connected layer with m inputs and n outputs by sampling each weight from u ( − 1 √m, 1 √m ), while glorot and bengio ( ) suggest using the 2010 normalized initialization wi, j [UNK] − 6 m n +, 6 m n +. ( 8. 23 ) this latter heuristic is designed to compromise between the goal of initializing all layers to have the same activation variance and the goal of initializing all layers to have the same gradient variance. the formula is derived using the assumption that the network consists only of a chain of matrix multiplications, with no nonlinearities. real neural networks obviously violate this assumption, but many strategies designed for the linear model perform reasonably well on its nonlinear counterparts. saxe 2013 et al. ( ) recommend initializing to random orthogonal matrices, with a carefully chosen scaling or gain factor g that accounts for the nonlinearity applied at each layer. they derive speciﬁc values of the scaling factor for [UNK] types of nonlinear activation functions. this initialization scheme is also motivated by a model of a