use the vector w to indicate all of the weights that should be [UNK] by a norm penalty, while the vector θ denotes all of the parameters, including both w and the unregularized parameters. in the context of neural networks, it is sometimes desirable to use a separate penalty with a [UNK] α [UNK] for each layer of the network. because it can be expensive to search for the correct value of multiple hyperparameters, it is still reasonable to use the same weight decay at all layers just to reduce the size of search space. 230


chapter 7. regularization for deep learning 7. 1. 1 l2 parameter regularization we have already seen, in section, one of the simplest and most common kinds 5. 2. 2 of parameter norm penalty : the l2 parameter norm penalty commonly known as weight decay. this regularization strategy drives the weights closer to the origin1 by adding a regularization term ω ( θ ) = 1 2 w 2 2 to the objective function. in other academic communities, l2 regularization is also known as ridge regression or tikhonov regularization. we can gain some insight into the behavior of weight decay regularization by studying the gradient of the regularized objective function. to simplify the presentation, we assume no bias parameter, so θ is just w. such a model has the following total objective function : [UNK], ( ; w x y ) = α 2 ww w x y + ( j ;, ), ( 7. 2 ) with the corresponding parameter gradient ∇w [UNK], α ( ; w x y ) = w + ∇wj,. ( ; w x y ) ( 7. 3 ) to take a single gradient step to update the weights, we perform this update : w w w ← −α ( + ∇wj,.

/home/ricoiban/GEMMA/mnlp_chatsplaining/RAG/Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville (z-lib.org).pdf


y ) = w + ∇wj,. ( ; w x y ) ( 7. 3 ) to take a single gradient step to update the weights, we perform this update : w w w ← −α ( + ∇wj,. ( ; w x y ) ) ( 7. 4 ) written another way, the update is : w w ← − ( 1 α ) −∇ wj,. ( ; w x y ) ( 7. 5 ) we can see that the addition of the weight decay term has modiﬁed the learning rule to multiplicatively shrink the weight vector by a constant factor on each step, just before performing the usual gradient update. this describes what happens in a single step. but what happens over the entire course of training? we will further simplify the analysis by making a quadratic approximation to the objective function in the neighborhood of the value of the weights that obtains minimal unregularized training cost, w∗ = arg minw j ( w ). if the objective function is truly quadratic, as in the case of ﬁtting a linear regression model with 1more generally, we could regularize the parameters to be near any speciﬁc point in space and, surprisingly, still get


). if the objective function is truly quadratic, as in the case of ﬁtting a linear regression model with 1more generally, we could regularize the parameters to be near any speciﬁc point in space and, surprisingly, still get a regularization [UNK], but better results will be obtained for a value closer to the true one, with zero being a default value that makes sense when we do not know if the correct value should be positive or negative. since it is far more common to regularize the model parameters towards zero, we will focus on this special case in our exposition. 231


chapter 7. regularization for deep learning mean squared error, then the approximation is perfect. the approximation [UNK] is given by [UNK] j ( ) = θ ( w∗ ) + 1 2 ( w w − ∗ ) h w w ( − ∗ ), ( 7. 6 ) where h is the hessian matrix of j with respect to w evaluated at w∗. there is no ﬁrst - order term in this quadratic approximation, because w∗is deﬁned to be a minimum, where the gradient vanishes. likewise, because w∗is the location of a minimum of, we can conclude that is positive semideﬁnite. j h the minimum of [UNK] occurs where its gradient ∇w [UNK] ( ) = ( w h w w − ∗ ) ( 7. 7 ) is equal to. 0 to study the [UNK] of weight decay, we modify equation by adding the 7. 7 weight decay gradient. we can now solve for the minimum of the regularized version of [UNK]. we use the variable [UNK] to represent the location of the minimum. α [UNK] h + ( [UNK] w − ∗ ) = 0 ( 7. 8 ) ( + ) h αi [UNK] hw = ∗ ( 7. 9 ) [UNK] h i = ( + α ) −


to represent the location of the minimum. α [UNK] h + ( [UNK] w − ∗ ) = 0 ( 7. 8 ) ( + ) h αi [UNK] hw = ∗ ( 7. 9 ) [UNK] h i = ( + α ) −1hw∗. ( 7. 10 ) as α approaches 0, the regularized solution [UNK] approaches w∗. but what happens as α grows? because h is real and symmetric, we can decompose it into a diagonal matrix λ and an orthonormal basis of eigenvectors, q, such that h q q = λ. applying the decomposition to equation, we obtain : 7. 10 [UNK] q q = ( λ + ) αi −1q q λ w∗ ( 7. 11 ) = q i q ( + λ α ) −1 q q λ w∗ ( 7. 12 ) = ( + ) q λ αi −1λqw∗. ( 7. 13 ) we see that the [UNK] of weight decay is to rescale w ∗along the axes deﬁned by the eigenvectors of h. speciﬁcally, the component of w ∗that is aligned with the i - th eigenvector of h is


is to rescale w ∗along the axes deﬁned by the eigenvectors of h. speciﬁcally, the component of w ∗that is aligned with the i - th eigenvector of h is rescaled by a factor of λi λi + α. ( you may wish to review how this kind of scaling works, ﬁrst explained in ﬁgure ). 2. 3 along the directions where the eigenvalues of h are relatively large, for example, where λi α, the [UNK] of regularization is relatively small. however, components with λi α will be shrunk to have nearly zero magnitude. this [UNK] is illustrated in ﬁgure. 7. 1 232


chapter 7. regularization for deep learning w1 w2 w∗ [UNK] w figure 7. 1 : an illustration of the [UNK] ofl2 ( or weight decay ) regularization on the value of the optimal w. the solid ellipses represent contours of equal value of the unregularized objective. the dotted circles represent contours of equal value of thel2 regularizer. at the point [UNK] w, these competing objectives reach an equilibrium. in the ﬁrst dimension, the eigenvalue of the hessian of j is small. the objective function does not increase much when moving horizontally away from w∗. because the objective function does not express a strong preference along this direction, the regularizer has a strong [UNK] on this axis. the regularizer pulls w 1 close to zero. in the second dimension, the objective function is very sensitive to movements away from w∗. the corresponding eigenvalue is large, indicating high curvature. as a result, weight decay [UNK] the position ofw2 relatively little. only directions along which the parameters contribute signiﬁcantly to reducing the objective function are preserved relatively intact. in directions that do not contribute to reducing the objective function, a small eigenvalue of the hessian tells


##w2 relatively little. only directions along which the parameters contribute signiﬁcantly to reducing the objective function are preserved relatively intact. in directions that do not contribute to reducing the objective function, a small eigenvalue of the hessian tells us that movement in this direction will not signiﬁcantly increase the gradient. components of the weight vector corresponding to such unimportant directions are decayed away through the use of the regularization throughout training. so far we have discussed weight decay in terms of its [UNK] on the optimization of an abstract, general, quadratic cost function. how do these [UNK] relate to machine learning in particular? we can ﬁnd out by studying linear regression, a model for which the true cost function is quadratic and therefore amenable to the same kind of analysis we have used so far. applying the analysis again, we will be able to obtain a special case of the same results, but with the solution now phrased in terms of the training data. for linear regression, the cost function is 233


chapter 7. regularization for deep learning the sum of squared errors : ( ) xw y − ( ) xw y −. ( 7. 14 ) when we add l2 regularization, the objective function changes to ( ) xw y − ( ) + xw y − 1 2αww. ( 7. 15 ) this changes the normal equations for the solution from w x = ( x ) −1x y ( 7. 16 ) to w x = ( x i + α ) −1xy. ( 7. 17 ) the matrix xx in equation is proportional to the covariance matrix 7. 16 1 mxx. using l2 regularization replaces this matrix with xx i + α −1 in equation. 7. 17 the new matrix is the same as the original one, but with the addition of α to the diagonal. the diagonal entries of this matrix correspond to the variance of each input feature. we can see that l2 regularization causes the learning algorithm to “ perceive ” the input x as having higher variance, which makes it shrink the weights on features whose covariance with the output target is low compared to this added variance. 7. 1. 2 l1 regularization while l2 weight decay is the most common form


input x as having higher variance, which makes it shrink the weights on features whose covariance with the output target is low compared to this added variance. 7. 1. 2 l1 regularization while l2 weight decay is the most common form of weight decay, there are other ways to penalize the size of the model parameters. another option is to use l1 regularization. formally, l1 regularization on the model parameter is deﬁned as : w ω ( ) = θ | | | | w 1 = i | wi |, ( 7. 18 ) that is, as the sum of absolute values of the individual parameters. 2 we will now discuss the [UNK] of l1 regularization on the simple linear regression model, with no bias parameter, that we studied in our analysis of l2 regularization. in particular, we are interested in delineating the [UNK] between l1 and l2 forms 2as with l2 regularization, we could regularize the parameters towards a value that is not zero, but instead towards some parameter value w ( ) o. in that case the l1 regularization would introduce the term ω ( ) = θ | | − w w ( ) o | | 1 = i | wi −w ( ) o


zero, but instead towards some parameter value w ( ) o. in that case the l1 regularization would introduce the term ω ( ) = θ | | − w w ( ) o | | 1 = i | wi −w ( ) o i |. 234


chapter 7. regularization for deep learning of regularization. as with l2 weight decay, l1 weight decay controls the strength of the regularization by scaling the penalty using a positive hyperparameter ω α. thus, the regularized objective function [UNK], ( ; w x y ) is given by [UNK], α ( ; w x y ) = | | | | w 1 + ( ; ) j w x y,, ( 7. 19 ) with the corresponding gradient ( actually, sub - gradient ) : ∇w [UNK], α ( ; w x y ) = sign ( ) + w ∇wj, ( x y w ; ) ( 7. 20 ) where is simply the sign of applied element - wise. sign ( ) w w by inspecting equation, we can see immediately that the [UNK] of 7. 20 l1 regularization is quite [UNK] from that of l2 regularization. speciﬁcally, we can see that the regularization contribution to the gradient no longer scales linearly with each wi ; instead it is a constant factor with a sign equal to sign ( wi ). one consequence of this form of the gradient is that we will not necessarily see clean algebraic solutions to quadratic approximations of j ( x y, ; w )


wi ; instead it is a constant factor with a sign equal to sign ( wi ). one consequence of this form of the gradient is that we will not necessarily see clean algebraic solutions to quadratic approximations of j ( x y, ; w ) as we did for l2 regularization. our simple linear model has a quadratic cost function that we can represent via its taylor series. alternately, we could imagine that this is a truncated taylor series approximating the cost function of a more sophisticated model. the gradient in this setting is given by ∇w [UNK] ( ) = ( w h w w − ∗ ), ( 7. 21 ) where, again, is the hessian matrix of with respect to evaluated at h j w w∗. because the l1 penalty does not admit clean algebraic expressions in the case of a fully general hessian, we will also make the further simplifying assumption that the hessian is diagonal, h = diag ( [ h1 1,,..., hn, n ] ), where each hi, i > 0. this assumption holds if the data for the linear regression problem has been preprocessed to remove all correlation between the input features, which may be accomplished using pca. our quadratic approximation


n ] ), where each hi, i > 0. this assumption holds if the data for the linear regression problem has been preprocessed to remove all correlation between the input features, which may be accomplished using pca. our quadratic approximation of the l1 regularized objective function decom - poses into a sum over the parameters : [UNK], j ( ; w x y ) = ( w∗ ; ) + x y, i 1 2h i, i ( wi −w∗ i ) 2 + α w | i |. ( 7. 22 ) the problem of minimizing this approximate cost function has an analytical solution ( for each dimension ), with the following form : i wi = sign ( w∗ i ) max | w∗ i | − α hi, i, 0. ( 7. 23 ) 235


chapter 7. regularization for deep learning consider the situation where w∗ i > i 0 for all. there are two possible outcomes : 1. the case where w∗ i ≤ α hi, i. here the optimal value of wi under the regularized objective is simply wi = 0. this occurs because the contribution of j ( w ; x y, ) to the regularized objective [UNK] ( w ; x y, ) is overwhelmed — in direction i — by the l1 regularization which pushes the value of wi to zero. 2. the case where w∗ i > α hi, i. in this case, the regularization does not move the optimal value of wi to zero but instead it just shifts it in that direction by a distance equal to α hi, i. a similar process happens when w∗ i < 0, but with the l1 penalty making wi less negative by α hi, i, or 0. in comparison to l2 regularization, l1 regularization results in a solution that is more sparse. sparsity in this context refers to the fact that some parameters have an optimal value of zero. the sparsity of l1 regularization is a qualitatively [UNK] behavior than arises with l2 regularization. equation gave the


sparse. sparsity in this context refers to the fact that some parameters have an optimal value of zero. the sparsity of l1 regularization is a qualitatively [UNK] behavior than arises with l2 regularization. equation gave the 7. 13 solution [UNK] for l2 regularization. if we revisit that equation using the assumption of a diagonal and positive deﬁnite hessian h that we introduced for our analysis of l1 regularization, we ﬁnd that [UNK] = hi, i hi, i + αw∗ i. if w∗ i was nonzero, then [UNK] remains nonzero. this demonstrates that l2 regularization does not cause the parameters to become sparse, while l1 regularization may do so for large enough. α the sparsity property induced by l1 regularization has been used extensively as a feature selection mechanism. feature selection simpliﬁes a machine learning problem by choosing which subset of the available features should be used. in particular, the well known lasso (, ) ( least absolute shrinkage and tibshirani 1995 selection operator ) model integrates an l1 penalty with a linear model and a least squares cost function. the l1 penalty causes a subset of the weights to


well known lasso (, ) ( least absolute shrinkage and tibshirani 1995 selection operator ) model integrates an l1 penalty with a linear model and a least squares cost function. the l1 penalty causes a subset of the weights to become zero, suggesting that the corresponding features may safely be discarded. in section, we saw that many regularization strategies can be interpreted 5. 6. 1 as map bayesian inference, and that in particular, l2 regularization is equivalent to map bayesian inference with a gaussian prior on the weights. for l1 regu - larization, the penalty αω ( w ) = α i | wi | used to regularize a cost function is equivalent to the log - prior term that is maximized by map bayesian inference when the prior is an isotropic laplace distribution ( equation ) over 3. 26 w ∈rn : log ( ) = p w i log laplace ( wi ; 0, 1 α ) = − | | | | α w 1 + log log 2 n α n −. ( 7. 24 ) 236


chapter 7. regularization for deep learning from the point of view of learning via maximization with respect to w, we can ignore the terms because they do not depend on. log log 2 α − w 7. 2 norm penalties as constrained optimization consider the cost function regularized by a parameter norm penalty : [UNK], j, α. ( ; θ x y ) = ( ; θ x y ) + ω ( ) θ ( 7. 25 ) recall from section that we can minimize a function subject to constraints 4. 4 by constructing a generalized lagrange function, consisting of the original objective function plus a set of penalties. each penalty is a product between a [UNK], called a karush – kuhn – tucker ( kkt ) multiplier, and a function representing whether the constraint is satisﬁed. if we wanted to constrain ω ( θ ) to be less than some constant, we could construct a generalized lagrange function k l − ( ; ) = ( ; ) + ( ω ( ) θ, α x y, j θ x y, α θ k. ) ( 7. 26 ) the solution to the constrained problem is given by θ∗ = arg min θ max α, α≥0l ( ) θ,


( ) θ, α x y, j θ x y, α θ k. ) ( 7. 26 ) the solution to the constrained problem is given by θ∗ = arg min θ max α, α≥0l ( ) θ, α. ( 7. 27 ) as described in section, solving this problem requires modifying both 4. 4 θ and α. section provides a worked example of linear regression with an 4. 5 l2 constraint. many [UNK] procedures are possible — some may use gradient descent, while others may use analytical solutions for where the gradient is zero — but in all procedures α must increase whenever ω ( θ ) > k and decrease whenever ω ( θ ) < k. all positive α encourage ω ( θ ) to shrink. the optimal value α∗will encourage ω ( θ ) to shrink, but not so strongly to make become less than. ω ( ) θ k to gain some insight into the [UNK] of the constraint, we can ﬁx α∗and view the problem as just a function of : θ θ∗ = arg min θ l ( θ, α∗ ) = arg min θ j, α ( ; θ x y ) + ∗ω ( ) θ. ( 7. 28 ) this is exactly the same as


function of : θ θ∗ = arg min θ l ( θ, α∗ ) = arg min θ j, α ( ; θ x y ) + ∗ω ( ) θ. ( 7. 28 ) this is exactly the same as the regularized training problem of minimizing [UNK]. we can thus think of a parameter norm penalty as imposing a constraint on the weights. if is the ω l2 norm, then the weights are constrained to lie in an l2 ball. if is the ω l1 norm, then the weights are constrained to lie in a region of 237


chapter 7. regularization for deep learning limited l1 norm. usually we do not know the size of the constraint region that we impose by using weight decay with [UNK] α∗because the value of α∗does not directly tell us the value of k. in principle, one can solve for k, but the relationship between k and α∗depends on the form of j. while we do not know the exact size of the constraint region, we can control it roughly by increasing or decreasing α in order to grow or shrink the constraint region. larger α will result in a smaller constraint region. smaller will result in a larger constraint region. α sometimes we may wish to use explicit constraints rather than penalties. as described in section, we can modify algorithms such as stochastic gradient 4. 4 descent to take a step downhill on j ( θ ) and then project θ back to the nearest point that satisﬁes ω ( θ ) < k. this can be useful if we have an idea of what value of k is appropriate and do not want to spend time searching for the value of α that corresponds to this. k another reason to use explicit constraints and reprojection rather than enforcing constraints with penalties is that penalties can cause non - convex optimization


value of k is appropriate and do not want to spend time searching for the value of α that corresponds to this. k another reason to use explicit constraints and reprojection rather than enforcing constraints with penalties is that penalties can cause non - convex optimization procedures to get stuck in local minima corresponding to small θ. when training neural networks, this usually manifests as neural networks that train with several “ dead units. ” these are units that do not contribute much to the behavior of the function learned by the network because the weights going into or out of them are all very small. when training with a penalty on the norm of the weights, these conﬁgurations can be locally optimal, even if it is possible to signiﬁcantly reduce j by making the weights larger. explicit constraints implemented by re - projection can work much better in these cases because they do not encourage the weights to approach the origin. explicit constraints implemented by re - projection only have an [UNK] when the weights become large and attempt to leave the constraint region. finally, explicit constraints with reprojection can be useful because they impose some stability on the optimization procedure. when using high learning rates, it is possible to encounter a positive feedback loop in which large weights induce large gradients which then induce a


. finally, explicit constraints with reprojection can be useful because they impose some stability on the optimization procedure. when using high learning rates, it is possible to encounter a positive feedback loop in which large weights induce large gradients which then induce a large update to the weights. if these updates consistently increase the size of the weights, then θ rapidly moves away from the origin until numerical overﬂow occurs. explicit constraints with reprojection prevent this feedback loop from continuing to increase the magnitude of the weights without bound. ( ) recommend using constraints combined with hinton et al. 2012c a high learning rate to allow rapid exploration of parameter space while maintaining some stability. in particular, hinton 2012c et al. ( ) recommend a strategy introduced by srebro and shraibman 2005 ( ) : constraining the norm of each column of the weight matrix 238


chapter 7. regularization for deep learning of a neural net layer, rather than constraining the frobenius norm of the entire weight matrix. constraining the norm of each column separately prevents any one hidden unit from having very large weights. if we converted this constraint into a penalty in a lagrange function, it would be similar to l2 weight decay but with a separate kkt multiplier for the weights of each hidden unit. each of these kkt multipliers would be dynamically updated separately to make each hidden unit obey the constraint. in practice, column norm limitation is always implemented as an explicit constraint with reprojection. 7. 3 regularization and under - constrained problems in some cases, regularization is necessary for machine learning problems to be prop - erly deﬁned. many linear models in machine learning, including linear regression and pca, depend on inverting the matrix x x. this is not possible whenever xx is singular. this matrix can be singular whenever the data generating distri - bution truly has no variance in some direction, or when no variance is observed in some direction because there are fewer examples ( rows of x ) than input features ( columns of x ). in this case, many forms of


generating distri - bution truly has no variance in some direction, or when no variance is observed in some direction because there are fewer examples ( rows of x ) than input features ( columns of x ). in this case, many forms of regularization correspond to inverting xx i + α instead. this regularized matrix is guaranteed to be invertible. these linear problems have closed form solutions when the relevant matrix is invertible. it is also possible for a problem with no closed form solution to be underdetermined. an example is logistic regression applied to a problem where the classes are linearly separable. if a weight vector w is able to achieve perfect classiﬁcation, then 2w will also achieve perfect classiﬁcation and higher likelihood. an iterative optimization procedure like stochastic gradient descent will continually increase the magnitude of w and, in theory, will never halt. in practice, a numerical implementation of gradient descent will eventually reach [UNK] large weights to cause numerical overﬂow, at which point its behavior will depend on how the programmer has decided to handle values that are not real numbers. most forms of regularization are able to guarantee the convergence of iterative methods applied to underdetermined problems. for example,


##ow, at which point its behavior will depend on how the programmer has decided to handle values that are not real numbers. most forms of regularization are able to guarantee the convergence of iterative methods applied to underdetermined problems. for example, weight decay will cause gradient descent to quit increasing the magnitude of the weights when the slope of the likelihood is equal to the weight decay [UNK]. the idea of using regularization to solve underdetermined problems extends beyond machine learning. the same idea is useful for several basic linear algebra problems. as we saw in section, we can solve underdetermined linear equations using 2. 9 239


chapter 7. regularization for deep learning the moore - penrose pseudoinverse. recall that one deﬁnition of the pseudoinverse x + of a matrix is x x + = lim α0 ( xx i + α ) −1x. ( 7. 29 ) we can now recognize equation as performing linear regression with weight 7. 29 decay. speciﬁcally, equation is the limit of equation as the regularization 7. 29 7. 17 [UNK] shrinks to zero. we can thus interpret the pseudoinverse as stabilizing underdetermined problems using regularization. 7. 4 dataset augmentation the best way to make a machine learning model generalize better is to train it on more data. of course, in practice, the amount of data we have is limited. one way to get around this problem is to create fake data and add it to the training set. for some machine learning tasks, it is reasonably straightforward to create new fake data. this approach is easiest for classiﬁcation. a classiﬁer needs to take a compli - cated, high dimensional input x and summarize it with a single category identity y. this means that the main task facing a classiﬁer is


classiﬁcation. a classiﬁer needs to take a compli - cated, high dimensional input x and summarize it with a single category identity y. this means that the main task facing a classiﬁer is to be invariant to a wide variety of transformations. we can generate new ( x, y ) pairs easily just by transforming the inputs in our training set. x this approach is not as readily applicable to many other tasks. for example, it is [UNK] to generate new fake data for a density estimation task unless we have already solved the density estimation problem. dataset augmentation has been a particularly [UNK] technique for a speciﬁc classiﬁcation problem : object recognition. images are high dimensional and include an enormous variety of factors of variation, many of which can be easily simulated. operations like translating the training images a few pixels in each direction can often greatly improve generalization, even if the model has already been designed to be partially translation invariant by using the convolution and pooling techniques described in chapter. many other operations such as rotating the image or scaling 9 the image have also proven quite [UNK]. one must be careful not to apply transformations that would change the correct class. for example, optical character recognition tasks


##tion and pooling techniques described in chapter. many other operations such as rotating the image or scaling 9 the image have also proven quite [UNK]. one must be careful not to apply transformations that would change the correct class. for example, optical character recognition tasks require recognizing the [UNK] between ‘ b ’ and ‘ d ’ and the [UNK] between ‘ 6 ’ and ‘ 9 ’, so horizontal ﬂips and [UNK] are not appropriate ways of augmenting datasets for these tasks. 240


chapter 7. regularization for deep learning there are also transformations that we would like our classiﬁers to be invariant to, but which are not easy to perform. for example, out - of - plane rotation can not be implemented as a simple geometric operation on the input pixels. dataset augmentation is [UNK] for speech recognition tasks as well ( jaitly and hinton 2013, ). injecting noise in the input to a neural network ( sietsma and dow 1991, ) can also be seen as a form of data augmentation. for many classiﬁcation and even some regression tasks, the task should still be possible to solve even if small random noise is added to the input. neural networks prove not to be very robust to noise, however ( tang and eliasmith 2010, ). one way to improve the robustness of neural networks is simply to train them with random noise applied to their inputs. input noise injection is part of some unsupervised learning algorithms such as the denoising autoencoder ( vincent 2008 et al., ). noise injection also works when the noise is applied to the hidden units, which can be seen as doing dataset augmentation at multiple levels of abstraction. poole 2014 et al


##oising autoencoder ( vincent 2008 et al., ). noise injection also works when the noise is applied to the hidden units, which can be seen as doing dataset augmentation at multiple levels of abstraction. poole 2014 et al. ( ) recently showed that this approach can be highly [UNK] provided that the magnitude of the noise is carefully tuned. dropout, a powerful regularization strategy that will be described in section, can be seen as a process of constructing new inputs by 7. 12 multiplying by noise. when comparing machine learning benchmark results, it is important to take the [UNK] of dataset augmentation into account. often, hand - designed dataset augmentation schemes can dramatically reduce the generalization error of a machine learning technique. to compare the performance of one machine learning algorithm to another, it is necessary to perform controlled experiments. when comparing machine learning algorithm a and machine learning algorithm b, it is necessary to make sure that both algorithms were evaluated using the same hand - designed dataset augmentation schemes. suppose that algorithm a performs poorly with no dataset augmentation and algorithm b performs well when combined with numerous synthetic transformations of the input. in such a case it is likely the synthetic transformations caused the improved performance, rather


##set augmentation schemes. suppose that algorithm a performs poorly with no dataset augmentation and algorithm b performs well when combined with numerous synthetic transformations of the input. in such a case it is likely the synthetic transformations caused the improved performance, rather than the use of machine learning algorithm b. sometimes deciding whether an experiment has been properly controlled requires subjective judgment. for example, machine learning algorithms that inject noise into the input are performing a form of dataset augmentation. usually, operations that are generally applicable ( such as adding gaussian noise to the input ) are considered part of the machine learning algorithm, while operations that are speciﬁc to one application domain ( such as randomly cropping an image ) are considered to be separate pre - processing steps. 241


chapter 7. regularization for deep learning 7. 5 noise robustness section has motivated the use of noise applied to the inputs as a dataset 7. 4 augmentation strategy. for some models, the addition of noise with inﬁnitesimal variance at the input of the model is equivalent to imposing a penalty on the norm of the weights (,, ). in the general case, it is important to bishop 1995a b remember that noise injection can be much more powerful than simply shrinking the parameters, especially when the noise is added to the hidden units. noise applied to the hidden units is such an important topic that it merit its own separate discussion ; the dropout algorithm described in section is the main development 7. 12 of that approach. another way that noise has been used in the service of regularizing models is by adding it to the weights. this technique has been used primarily in the context of recurrent neural networks (, ; jim et al. 1996 graves 2011, ). this can be interpreted as a stochastic implementation of bayesian inference over the weights. the bayesian treatment of learning would consider the model weights to be uncertain and representable via a probability distribution that reﬂects this uncertainty. adding noise to the weights is a practical


a stochastic implementation of bayesian inference over the weights. the bayesian treatment of learning would consider the model weights to be uncertain and representable via a probability distribution that reﬂects this uncertainty. adding noise to the weights is a practical, stochastic way to reﬂect this uncertainty. noise applied to the weights can also be interpreted as equivalent ( under some assumptions ) to a more traditional form of regularization, encouraging stability of the function to be learned. consider the regression setting, where we wish to train a function [UNK] ( x ) that maps a set of features x to a scalar using the least - squares cost function between the model predictions [UNK] ( ) x and the true values : y j = ep x, y ( ) ( [UNK] y ( ) x − ) 2. ( 7. 30 ) the training set consists of labeled examples m { ( x ( 1 ), y ( 1 ) ) (,..., x ( ) m, y ( ) m ) }. we now assume that with each input presentation we also include a random perturbation w [UNK] ( ; 0, ηi ) of the network weights. let us imagine that we have a standard l - layer mlp. we denote the perturbed model


now assume that with each input presentation we also include a random perturbation w [UNK] ( ; 0, ηi ) of the network weights. let us imagine that we have a standard l - layer mlp. we denote the perturbed model as [UNK] ( x ). despite the injection of noise, we are still interested in minimizing the squared error of the output of the network. the objective function thus becomes : [UNK] = ep, y, ( x w ) ( [UNK] ( ) ) x −y 2 ( 7. 31 ) = ep, y, ( x w ) [UNK] w ( ) 2 [UNK] x −yyw ( ) + x y2. ( 7. 32 ) for small η, the minimization of j with added weight noise ( with covariance ηi ) is equivalent to minimization of j with an additional regularization term : 242


chapter 7. regularization for deep learning ηep, y ( x ) ∇w [UNK] ( ) x 2. this form of regularization encourages the parameters to go to regions of parameter space where small perturbations of the weights have a relatively small inﬂuence on the output. in other words, it pushes the model into regions where the model is relatively insensitive to small variations in the weights, ﬁnding points that are not merely minima, but minima surrounded by ﬂat regions ( hochreiter and schmidhuber 1995, ). in the simpliﬁed case of linear regression ( where, for instance, [UNK] ( x ) = wx + b ), this regularization term collapses into ηep ( ) x x 2, which is not a function of parameters and therefore does not contribute to the gradient of [UNK] with respect to the model parameters. 7. 5. 1 injecting noise at the output targets most datasets have some amount of mistakes in the y labels. it can be harmful to maximize log p ( y | x ) when y is a mistake. one way to prevent this is to explicitly model the noise on the labels. for example, we can assume that for some small constant, the training


labels. it can be harmful to maximize log p ( y | x ) when y is a mistake. one way to prevent this is to explicitly model the noise on the labels. for example, we can assume that for some small constant, the training set label y is correct with probability 1−, and otherwise any of the other possible labels might be correct. this assumption is easy to incorporate into the cost function analytically, rather than by explicitly drawing noise samples. for example, label smoothing regularizes a model based on a softmax with k output values by replacing the hard and classiﬁcation targets 0 1 with targets of k−1 and 1−, respectively. the standard cross - entropy loss may then be used with these soft targets. maximum likelihood learning with a softmax classiﬁer and hard targets may actually never converge — the softmax can never predict a probability of exactly or exactly, so it will continue to learn larger 0 1 and larger weights, making more extreme predictions forever. it is possible to prevent this scenario using other regularization strategies like weight decay. label smoothing has the advantage of preventing the pursuit of hard probabilities without discouraging correct classiﬁcation. this strategy has been used since the 1980s and continues to be featured prominently in modern


scenario using other regularization strategies like weight decay. label smoothing has the advantage of preventing the pursuit of hard probabilities without discouraging correct classiﬁcation. this strategy has been used since the 1980s and continues to be featured prominently in modern neural networks ( szegedy et al., ). 2015 7. 6 semi - supervised learning in the paradigm of semi - supervised learning, both unlabeled examples from p ( x ) and labeled examples from p ( x y, ) are used to estimate p ( y x | ) or predict y from x. in the context of deep learning, semi - supervised learning usually refers to learning a representation h = f ( x ). the goal is to learn a representation so 243


chapter 7. regularization for deep learning that examples from the same class have similar representations. unsupervised learning can provide useful cues for how to group examples in representation space. examples that cluster tightly in the input space should be mapped to similar representations. a linear classiﬁer in the new space may achieve better generalization in many cases ( belkin and niyogi 2002 chapelle 2003, ; et al., ). a long - standing variant of this approach is the application of principal components analysis as a pre - processing step before applying a classiﬁer ( on the projected data ). instead of having separate unsupervised and supervised components in the model, one can construct models in which a generative model of either p ( x ) or p ( x y, ) shares parameters with a discriminative model of p ( y x | ). one can then trade - [UNK] supervised criterion −log p ( y x | ) with the unsupervised or generative one ( such as −log p ( x ) or −log p ( x y, ) ). the generative criterion then expresses a particular form of prior belief about the solution to the supervised learning problem (, ), namely that the structure of


one ( such as −log p ( x ) or −log p ( x y, ) ). the generative criterion then expresses a particular form of prior belief about the solution to the supervised learning problem (, ), namely that the structure of lasserre et al. 2006 p ( x ) is connected to the structure of p ( y x | ) in a way that is captured by the shared parametrization. by controlling how much of the generative criterion is included in the total criterion, one can ﬁnd a better trade - [UNK] with a purely generative or a purely discriminative training criterion (, ; lasserre et al. 2006 larochelle and bengio 2008, ). salakhutdinov and hinton 2008 ( ) describe a method for learning the kernel function of a kernel machine used for regression, in which the usage of unlabeled examples for modeling improves quite signiﬁcantly. p ( ) x p ( ) y x | see ( ) for more information about semi - supervised learning. chapelle et al. 2006 7. 7 multi - task learning multi - task learning (, ) is a way to improve generalization by pooling caruana 1993 the examples ( which can be seen as soft


information about semi - supervised learning. chapelle et al. 2006 7. 7 multi - task learning multi - task learning (, ) is a way to improve generalization by pooling caruana 1993 the examples ( which can be seen as soft constraints imposed on the parameters ) arising out of several tasks. in the same way that additional training examples put more pressure on the parameters of the model towards values that generalize well, when part of a model is shared across tasks, that part of the model is more constrained towards good values ( assuming the sharing is justiﬁed ), often yielding better generalization. figure illustrates a very common form of multi - task learning, in which 7. 2 [UNK] supervised tasks ( predicting y ( ) i given x ) share the same input x, as well as some intermediate - level representation h ( shared ) capturing a common pool of 244


chapter 7. regularization for deep learning factors. the model can generally be divided into two kinds of parts and associated parameters : 1. task - speciﬁc parameters ( which only beneﬁt from the examples of their task to achieve good generalization ). these are the upper layers of the neural network in ﬁgure. 7. 2 2. generic parameters, shared across all the tasks ( which beneﬁt from the pooled data of all the tasks ). these are the lower layers of the neural network in ﬁgure. 7. 2 h ( 1 ) h ( 1 ) h ( 2 ) h ( 2 ) h ( 3 ) h ( 3 ) y ( 1 ) y ( 1 ) y ( 2 ) y ( 2 ) h ( shared ) h ( shared ) x figure 7. 2 : multi - task learning can be cast in several ways in deep learning frameworks and this ﬁgure illustrates the common situation where the tasks share a common input but involve [UNK] target random variables. the lower layers of a deep network ( whether it is supervised and feedforward or includes a generative component with downward arrows ) can be shared across such tasks, while task - speciﬁc parameters ( associated respectively with the weights into and


. the lower layers of a deep network ( whether it is supervised and feedforward or includes a generative component with downward arrows ) can be shared across such tasks, while task - speciﬁc parameters ( associated respectively with the weights into and from h ( 1 ) and h ( 2 ) ) can be learned on top of those yielding a shared representation h ( shared ). the underlying assumption is that there exists a common pool of factors that explain the variations in the inputx, while each task is associated with a subset of these factors. in this example, it is additionally assumed that top - level hidden units h ( 1 ) and h ( 2 ) are specialized to each task ( respectively predicting y ( 1 ) and y ( 2 ) ) while some intermediate - level representationh ( shared ) is shared across all tasks. in the unsupervised learning context, it makes sense for some of the top - level factors to be associated with none of the output tasks ( h ( 3 ) ) : these are the factors that explain some of the input variations but are not relevant for predicting y ( 1 ) or y ( 2 ). improved generalization and generalization error bounds (, ) can be baxter 1995 achieved because of the shared parameters, for which statistical


the factors that explain some of the input variations but are not relevant for predicting y ( 1 ) or y ( 2 ). improved generalization and generalization error bounds (, ) can be baxter 1995 achieved because of the shared parameters, for which statistical strength can be 245


chapter 7. regularization for deep learning 0 50 100 150 200 250 time ( epochs ) 0 00. 0 05. 0 10. 0 15. 0 20. loss ( negative log - likelihood ) training set loss validation set loss figure 7. 3 : learning curves showing how the negative log - likelihood loss changes over time ( indicated as number of training iterations over the dataset, or epochs ). in this example, we train a maxout network on mnist. observe that the training objective decreases consistently over time, but the validation set average loss eventually begins to increase again, forming an asymmetric u - shaped curve. greatly improved ( in proportion with the increased number of examples for the shared parameters, compared to the scenario of single - task models ). of course this will happen only if some assumptions about the statistical relationship between the [UNK] tasks are valid, meaning that there is something shared across some of the tasks. from the point of view of deep learning, the underlying prior belief is the following : among the factors that explain the variations observed in the data associated with the [UNK] tasks, some are shared across two or more tasks. 7. 8 early stopping when training large models with [UNK] representational capacity to overﬁt the task, we often observe that


the factors that explain the variations observed in the data associated with the [UNK] tasks, some are shared across two or more tasks. 7. 8 early stopping when training large models with [UNK] representational capacity to overﬁt the task, we often observe that training error decreases steadily over time, but validation set error begins to rise again. see ﬁgure for an example of this 7. 3 behavior. this behavior occurs very reliably. this means we can obtain a model with better validation set error ( and thus, hopefully better test set error ) by returning to the parameter setting at the point in time with the lowest validation set error. every time the error on the validation set improves, we store a copy of the model parameters. when the training algorithm terminates, we return these parameters, rather than the latest parameters. the 246


chapter 7. regularization for deep learning algorithm terminates when no parameters have improved over the best recorded validation error for some pre - speciﬁed number of iterations. this procedure is speciﬁed more formally in algorithm. 7. 1 algorithm 7. 1 the early stopping meta - algorithm for determining the best amount of time to train. this meta - algorithm is a general strategy that works well with a variety of training algorithms and ways of quantifying error on the validation set. let be the number of steps between evaluations. n let p be the “ patience, ” the number of times to observe worsening validation set error before giving up. let θo be the initial parameters. θ θ ← o i ←0 j ←0 v ←∞ θ∗←θ i∗←i while do j < p update by running the training algorithm for steps. θ n i i n ← + v←validationseterror ( ) θ if v < v then j ←0 θ∗←θ i∗←i v v ← else j j ← + 1 end if end while best parameters are θ∗, best number of training steps is i∗ this strategy is known as early stopping. it is probably the most commonly used form of regularization in deep


##←i v v ← else j j ← + 1 end if end while best parameters are θ∗, best number of training steps is i∗ this strategy is known as early stopping. it is probably the most commonly used form of regularization in deep learning. its popularity is due both to its [UNK] and its simplicity. one way to think of early stopping is as a very [UNK] hyperparameter selection algorithm. in this view, the number of training steps is just another hyperparameter. we can see in ﬁgure that this hyperparameter has a u - shaped validation set 7. 3 247


chapter 7. regularization for deep learning performance curve. most hyperparameters that control model capacity have such a u - shaped validation set performance curve, as illustrated in ﬁgure. in the case of 5. 3 early stopping, we are controlling the [UNK] capacity of the model by determining how many steps it can take to ﬁt the training set. most hyperparameters must be chosen using an expensive guess and check process, where we set a hyperparameter at the start of training, then run training for several steps to see its [UNK]. the “ training time ” hyperparameter is unique in that by deﬁnition a single run of training tries out many values of the hyperparameter. the only signiﬁcant cost to choosing this hyperparameter automatically via early stopping is running the validation set evaluation periodically during training. ideally, this is done in parallel to the training process on a separate machine, separate cpu, or separate gpu from the main training process. if such resources are not available, then the cost of these periodic evaluations may be reduced by using a validation set that is small compared to the training set or by evaluating the validation set error less frequently and obtaining a lower resolution estimate of the optimal training time. an additional cost


available, then the cost of these periodic evaluations may be reduced by using a validation set that is small compared to the training set or by evaluating the validation set error less frequently and obtaining a lower resolution estimate of the optimal training time. an additional cost to early stopping is the need to maintain a copy of the best parameters. this cost is generally negligible, because it is acceptable to store these parameters in a slower and larger form of memory ( for example, training in gpu memory, but storing the optimal parameters in host memory or on a disk drive ). since the best parameters are written to infrequently and never read during training, these occasional slow writes have little [UNK] on the total training time. early stopping is a very unobtrusive form of regularization, in that it requires almost no change in the underlying training procedure, the objective function, or the set of allowable parameter values. this means that it is easy to use early stopping without damaging the learning dynamics. this is in contrast to weight decay, where one must be careful not to use too much weight decay and trap the network in a bad local minimum corresponding to a solution with pathologically small weights. early stopping may be used either alone or in conjunction with other regulariza -


weight decay, where one must be careful not to use too much weight decay and trap the network in a bad local minimum corresponding to a solution with pathologically small weights. early stopping may be used either alone or in conjunction with other regulariza - tion strategies. even when using regularization strategies that modify the objective function to encourage better generalization, it is rare for the best generalization to occur at a local minimum of the training objective. early stopping requires a validation set, which means some training data is not fed to the model. to best exploit this extra data, one can perform extra training after the initial training with early stopping has completed. in the second, extra training step, all of the training data is included. there are two basic strategies one can use for this second training procedure. one strategy ( algorithm ) is to initialize the model again and retrain on all 7. 2 248


chapter 7. regularization for deep learning of the data. in this second training pass, we train for the same number of steps as the early stopping procedure determined was optimal in the ﬁrst pass. there are some subtleties associated with this procedure. for example, there is not a good way of knowing whether to retrain for the same number of parameter updates or the same number of passes through the dataset. on the second round of training, each pass through the dataset will require more parameter updates because the training set is bigger. algorithm 7. 2 a meta - algorithm for using early stopping to determine how long to train, then retraining on all the data. let x ( ) train and y ( ) train be the training set. split x ( ) train and y ( ) train into ( x ( ) subtrain, x ( valid ) ) ( and y ( ) subtrain, y ( valid ) ) respectively. run early stopping ( algorithm ) starting from random 7. 1 θ using x ( ) subtrain and y ( ) subtrain for training data and x ( valid ) and y ( valid ) for validation data. this returns i∗, the optimal number of steps. set to random values again. θ train on x ( ) train and


subtrain and y ( ) subtrain for training data and x ( valid ) and y ( valid ) for validation data. this returns i∗, the optimal number of steps. set to random values again. θ train on x ( ) train and y ( ) train for i∗steps. another strategy for using all of the data is to keep the parameters obtained from the ﬁrst round of training and then continue training but now using all of the data. at this stage, we now no longer have a guide for when to stop in terms of a number of steps. instead, we can monitor the average loss function on the validation set, and continue training until it falls below the value of the training set objective at which the early stopping procedure halted. this strategy avoids the high cost of retraining the model from scratch, but is not as well - behaved. for example, there is not any guarantee that the objective on the validation set will ever reach the target value, so this strategy is not even guaranteed to terminate. this procedure is presented more formally in algorithm. 7. 3 early stopping is also useful because it reduces the computational cost of the training procedure. besides the obvious reduction in cost due to limiting the number of training iterations, it also has the ben


. this procedure is presented more formally in algorithm. 7. 3 early stopping is also useful because it reduces the computational cost of the training procedure. besides the obvious reduction in cost due to limiting the number of training iterations, it also has the beneﬁt of providing regularization without requiring the addition of penalty terms to the cost function or the computation of the gradients of such additional terms. how early stopping acts as a regularizer : so far we have stated that early stopping a regularization strategy, but we have supported this claim only by is showing learning curves where the validation set error has a u - shaped curve. what 249


chapter 7. regularization for deep learning algorithm 7. 3 meta - algorithm using early stopping to determine at what objec - tive value we start to overﬁt, then continue training until that value is reached. let x ( ) train and y ( ) train be the training set. split x ( ) train and y ( ) train into ( x ( ) subtrain, x ( valid ) ) ( and y ( ) subtrain, y ( valid ) ) respectively. run early stopping ( algorithm ) starting from random 7. 1 θ using x ( ) subtrain and y ( ) subtrain for training data and x ( valid ) and y ( valid ) for validation data. this updates. θ j, ← ( θ x ( ) subtrain, y ( ) subtrain ) while j, ( θ x ( valid ), y ( valid ) ) > do train on x ( ) train and y ( ) train for steps. n end while is the actual mechanism by which early stopping regularizes the model? bishop ( ) and ( ) argued that early stopping has the [UNK] of 1995a sjoberg and ljung 1995 restricting the optimization procedure to a relatively small volume of parameter space in the neighborhood of the initial parameter value θo, as


model? bishop ( ) and ( ) argued that early stopping has the [UNK] of 1995a sjoberg and ljung 1995 restricting the optimization procedure to a relatively small volume of parameter space in the neighborhood of the initial parameter value θo, as illustrated in ﬁgure. more speciﬁcally, imagine taking 7. 4 τ optimization steps ( corresponding to τ training iterations ) and with learning rate. we can view the product τ as a measure of [UNK] capacity. assuming the gradient is bounded, restricting both the number of iterations and the learning rate limits the volume of parameter space reachable from θo. in this sense, τ behaves as if it were the reciprocal of the [UNK] used for weight decay. indeed, we can show how — in the case of a simple linear model with a quadratic error function and simple gradient descent — early stopping is equivalent to l2 regularization. in order to compare with classical l2 regularization, we examine a simple setting where the only parameters are linear weights ( θ = w ). we can model the cost function j with a quadratic approximation in the neighborhood of the empirically optimal value of the weights w∗ : [UNK] j ( ) = θ ( w∗ ) + 1 2 ( w w −


( θ = w ). we can model the cost function j with a quadratic approximation in the neighborhood of the empirically optimal value of the weights w∗ : [UNK] j ( ) = θ ( w∗ ) + 1 2 ( w w − ∗ ) h w w ( − ∗ ), ( 7. 33 ) where h is the hessian matrix of j with respect to w evaluated at w∗. given the assumption that w∗is a minimum of j ( w ), we know that h is positive semideﬁnite. under a local taylor series approximation, the gradient is given by : ∇w [UNK] ( ) = ( w h w w − ∗ ). ( 7. 34 ) 250


chapter 7. regularization for deep learning w1 w2 w∗ [UNK] w w1 w2 w∗ [UNK] w figure 7. 4 : an illustration of the [UNK] of early stopping. ( left ) the solid contour lines indicate the contours of the negative log - likelihood. the dashed line indicates the trajectory taken by sgd beginning from the origin. rather than stopping at the point w∗that minimizes the cost, early stopping results in the trajectory stopping at an earlier point [UNK]. ( right ) an illustration of the [UNK] of l2 regularization for comparison. the dashed circles indicate the contours of the l2 penalty, which causes the minimum of the total cost to lie nearer the origin than the minimum of the unregularized cost. we are going to study the trajectory followed by the parameter vector during training. for simplicity, let us set the initial parameter vector to the origin, 3 that is w ( 0 ) = 0. let us study the approximate behavior of gradient descent on j by analyzing gradient descent on [UNK] : w ( ) τ = w ( 1 ) τ− −∇ w [UNK] ( w ( 1 ) τ− ) ( 7. 35 ) = w ( 1 ) τ− −h w ( ( 1 ) τ


by analyzing gradient descent on [UNK] : w ( ) τ = w ( 1 ) τ− −∇ w [UNK] ( w ( 1 ) τ− ) ( 7. 35 ) = w ( 1 ) τ− −h w ( ( 1 ) τ− −w ∗ ) ( 7. 36 ) w ( ) τ −w∗ = ( ) ( i h − w ( 1 ) τ− −w∗ ). ( 7. 37 ) let us now rewrite this expression in the space of the eigenvectors ofh, exploiting the eigendecomposition of h : h = q q λ, where λ is a diagonal matrix and q is an orthonormal basis of eigenvectors. w ( ) τ −w∗ = ( i q q − λ ) ( w ( 1 ) τ− −w∗ ) ( 7. 38 ) q ( w ( ) τ −w∗ ) = ( ) i −λ q ( w ( 1 ) τ− −w∗ ) ( 7. 39 ) 3for neural networks, to obtain symmetry breaking between hidden units, we cannot initialize all the parameters to 0, as discussed in section. however, the argument holds for any other 6. 2 initial value w ( 0


) ( 7. 39 ) 3for neural networks, to obtain symmetry breaking between hidden units, we cannot initialize all the parameters to 0, as discussed in section. however, the argument holds for any other 6. 2 initial value w ( 0 ). 251


chapter 7. regularization for deep learning assuming that w ( 0 ) = 0 and that is chosen to be small enough to guarantee | 1−λi | < 1, the parameter trajectory during training after τ parameter updates is as follows : qw ( ) τ = [ ( ) i −i −λ τ ] qw∗. ( 7. 40 ) now, the expression for [UNK] in equation for 7. 13 l2 regularization can be rear - ranged as : [UNK] i = ( + λ α ) −1λq w∗ ( 7. 41 ) [UNK] i i = [ − ( + λ α ) −1α ] qw∗ ( 7. 42 ) comparing equation and equation, we see that if the hyperparameters 7. 40 7. 42, α τ, and are chosen such that ( ) i −λ τ = ( + ) λ αi −1α, ( 7. 43 ) then l2 regularization and early stopping can be seen to be equivalent ( at least under the quadratic approximation of the objective function ). going even further, by taking logarithms and using the series expansion for log ( 1 + x ), we can conclude that if all λi are small ( that is, λi


under the quadratic approximation of the objective function ). going even further, by taking logarithms and using the series expansion for log ( 1 + x ), we can conclude that if all λi are small ( that is, λi 1 and λi / α 1 ) then τ ≈1 α, ( 7. 44 ) α ≈1 τ. ( 7. 45 ) that is, under these assumptions, the number of training iterations τ plays a role inversely proportional to the l2 regularization parameter, and the inverse of τ plays the role of the weight decay [UNK]. parameter values corresponding to directions of signiﬁcant curvature ( of the objective function ) are regularized less than directions of less curvature. of course, in the context of early stopping, this really means that parameters that correspond to directions of signiﬁcant curvature tend to learn early relative to parameters corresponding to directions of less curvature. the derivations in this section have shown that a trajectory of length τ ends at a point that corresponds to a minimum of the l2 - regularized objective. early stopping is of course more than the mere restriction of the trajectory length ; instead, early stopping typically involves monitoring the validation set error in order to stop the trajectory at a particularly


point that corresponds to a minimum of the l2 - regularized objective. early stopping is of course more than the mere restriction of the trajectory length ; instead, early stopping typically involves monitoring the validation set error in order to stop the trajectory at a particularly good point in space. early stopping therefore has the advantage over weight decay that early stopping automatically determines the correct amount of regularization while weight decay requires many training experiments with [UNK] values of its hyperparameter. 252


chapter 7. regularization for deep learning 7. 9 parameter tying and parameter sharing thus far, in this chapter, when we have discussed adding constraints or penalties to the parameters, we have always done so with respect to a ﬁxed region or point. for example, l2 regularization ( or weight decay ) penalizes model parameters for deviating from the ﬁxed value of zero. however, sometimes we may need other ways to express our prior knowledge about suitable values of the model parameters. sometimes we might not know precisely what values the parameters should take but we know, from knowledge of the domain and model architecture, that there should be some dependencies between the model parameters. a common type of dependency that we often want to express is that certain parameters should be close to one another. consider the following scenario : we have two models performing the same classiﬁcation task ( with the same set of classes ) but with somewhat [UNK] input distributions. formally, we have model a with parameters w ( ) a and model b with parameters w ( ) b. the two models map the input to two [UNK], but related outputs : [UNK] ( ) a = f ( w ( ) a, x ) and [UNK] ( ) b = ( g w ( ) b, x ). let


w ( ) b. the two models map the input to two [UNK], but related outputs : [UNK] ( ) a = f ( w ( ) a, x ) and [UNK] ( ) b = ( g w ( ) b, x ). let us imagine that the tasks are similar enough ( perhaps with similar input and output distributions ) that we believe the model parameters should be close to each other : [UNK], w ( ) a i should be close to w ( ) b i. we can leverage this information through regularization. speciﬁcally, we can use a parameter norm penalty of the form : ω ( w ( ) a, w ( ) b ) = w ( ) a −w ( ) b 2 2. here we used an l2 penalty, but other choices are also possible. this kind of approach was proposed by ( ), who regularized lasserre et al. 2006 the parameters of one model, trained as a classiﬁer in a supervised paradigm, to be close to the parameters of another model, trained in an unsupervised paradigm ( to capture the distribution of the observed input data ). the architectures were constructed such that many of the parameters in the classiﬁer model could be paired to corresponding parameters in


of another model, trained in an unsupervised paradigm ( to capture the distribution of the observed input data ). the architectures were constructed such that many of the parameters in the classiﬁer model could be paired to corresponding parameters in the unsupervised model. while a parameter norm penalty is one way to regularize parameters to be close to one another, the more popular way is to use constraints : to force sets of parameters to be equal. this method of regularization is often referred to as parameter sharing, because we interpret the various models or model components as sharing a unique set of parameters. a signiﬁcant advantage of parameter sharing over regularizing the parameters to be close ( via a norm penalty ) is that only a subset of the parameters ( the unique set ) need to be stored in memory. in certain models — such as the convolutional neural network — this can lead to signiﬁcant reduction in the memory footprint of the model. 253


chapter 7. regularization for deep learning convolutional neural networks by far the most popular and extensive use of parameter sharing occurs in convolutional neural networks ( cnns ) applied to computer vision. natural images have many statistical properties that are invariant to translation. for example, a photo of a cat remains a photo of a cat if it is translated one pixel to the right. cnns take this property into account by sharing parameters across multiple image locations. the same feature ( a hidden unit with the same weights ) is computed over [UNK] locations in the input. this means that we can ﬁnd a cat with the same cat detector whether the cat appears at column i or column i + 1 in the image. parameter sharing has allowed cnns to dramatically lower the number of unique model parameters and to signiﬁcantly increase network sizes without requiring a corresponding increase in training data. it remains one of the best examples of how to [UNK] incorporate domain knowledge into the network architecture. cnns will be discussed in more detail in chapter. 9 7. 10 sparse representations weight decay acts by placing a penalty directly on the model parameters. another strategy is to place a penalty on the activations of the units in a neural network, encouraging their activations to be sparse.


detail in chapter. 9 7. 10 sparse representations weight decay acts by placing a penalty directly on the model parameters. another strategy is to place a penalty on the activations of the units in a neural network, encouraging their activations to be sparse. this indirectly imposes a complicated penalty on the model parameters. we have already discussed ( in section ) how 7. 1. 2 l1 penalization induces a sparse parametrization — meaning that many of the parameters become zero ( or close to zero ). representational sparsity, on the other hand, describes a representation where many of the elements of the representation are zero ( or close to zero ). a simpliﬁed view of this distinction can be illustrated in the context of linear regression : 18 5 15 −9 −3 = 4 0 0 2 0 0 − 0 0 1 0 3 0 − 0 5 0 0 0 0 1 0 0 1 0 4 − − 1 0 0 0 5 0 − 2 3 −2 −5 1 4 y ∈rm a ∈r m n × x ∈r n ( 7. 46 ) 254


chapter 7. regularization for deep learning −14 1 19 2 23 = 3 1 2 5 4 1 − − 4 2 3 1 1 3 − − − − − 1 5 4 2 3 2 3 1 2 3 0 3 − − − − − − 5 4 2 2 5 1 0 2 0 0 −3 0 y ∈rm b ∈rm n × h ∈rn ( 7. 47 ) in the ﬁrst expression, we have an example of a sparsely parametrized linear regression model. in the second, we have linear regression with a sparse representa - tion h of the data x. that is, h is a function of x that, in some sense, represents the information present in, but does so with a sparse vector. x representational regularization is accomplished by the same sorts of mechanisms that we have used in parameter regularization. norm penalty regularization of representations is performed by adding to the loss function j a norm penalty on the representation. this penalty is denoted ω ( ) h. as before, we denote the regularized loss function by [UNK] : [UNK], j, α ( ; θ x y ) = ( ; θ x y ) + ω ( ) h ( 7. 48 ) where α ∈ [ 0, ∞ ) weights the


before, we denote the regularized loss function by [UNK] : [UNK], j, α ( ; θ x y ) = ( ; θ x y ) + ω ( ) h ( 7. 48 ) where α ∈ [ 0, ∞ ) weights the relative contribution of the norm penalty term, with larger values of corresponding to more regularization. α just as an l1 penalty on the parameters induces parameter sparsity, an l1 penalty on the elements of the representation induces representational sparsity : ω ( h ) = | | | | h 1 = i | hi |. of course, the l1 penalty is only one choice of penalty that can result in a sparse representation. others include the penalty derived from a student - t prior on the representation (, ;, ) olshausen and field 1996 bergstra 2011 and kl divergence penalties (, ) that are especially larochelle and bengio 2008 useful for representations with elements constrained to lie on the unit interval. lee 2008 goodfellow 2009 et al. ( ) and et al. ( ) both provide examples of strategies based on regularizing the average activation across several examples, 1 m i h ( ) i, to be near some target value, such as a vector with


##fellow 2009 et al. ( ) and et al. ( ) both provide examples of strategies based on regularizing the average activation across several examples, 1 m i h ( ) i, to be near some target value, such as a vector with. 01 for each entry. other approaches obtain representational sparsity with a hard constraint on the activation values. for example, orthogonal matching pursuit ( pati et al., 1993 ) encodes an input x with the representation h that solves the constrained optimization problem arg min h h, 0 < k − x w h 2, ( 7. 49 ) where h 0 is the number of non - zero entries of h. this problem can be solved [UNK] when w is constrained to be orthogonal. this method is often called 255


chapter 7. regularization for deep learning omp - k with the value of k speciﬁed to indicate the number of non - zero features allowed. ( ) demonstrated that omp - can be a very [UNK] coates and ng 2011 1 feature extractor for deep architectures. essentially any model that has hidden units can be made sparse. throughout this book, we will see many examples of sparsity regularization used in a variety of contexts. 7. 11 bagging and other ensemble methods bagging ( short for bootstrap aggregating ) is a technique for reducing gen - eralization error by combining several models (, ). the idea is to breiman 1994 train several [UNK] models separately, then have all of the models vote on the output for test examples. this is an example of a general strategy in machine learning called model averaging. techniques employing this strategy are known as ensemble methods. the reason that model averaging works is that [UNK] models will usually not make all the same errors on the test set. consider for example a set of k regression models. suppose that each model makes an error i on each example, with the errors drawn from a zero - mean multivariate normal distribution with variances e [ 2 i ] = v and covar


. consider for example a set of k regression models. suppose that each model makes an error i on each example, with the errors drawn from a zero - mean multivariate normal distribution with variances e [ 2 i ] = v and covariances e [ ij ] = c. then the error made by the average prediction of all the ensemble models is 1 k i i. the expected squared error of the ensemble predictor is e 1 k i i 2 = 1 k2 e i 2 i + j i = ij ( 7. 50 ) = 1 kv + k −1 k c. ( 7. 51 ) in the case where the errors are perfectly correlated and c = v, the mean squared error reduces to v, so the model averaging does not help at all. in the case where the errors are perfectly uncorrelated and c = 0, the expected squared error of the ensemble is only 1 kv. this means that the expected squared error of the ensemble decreases linearly with the ensemble size. in other words, on average, the ensemble will perform at least as well as any of its members, and if the members make independent errors, the ensemble will perform signiﬁcantly better than its members. [UNK] ensemble methods construct the


size. in other words, on average, the ensemble will perform at least as well as any of its members, and if the members make independent errors, the ensemble will perform signiﬁcantly better than its members. [UNK] ensemble methods construct the ensemble of models in [UNK] ways. for example, each member of the ensemble could be formed by training a completely 256


chapter 7. regularization for deep learning 8 8 first ensemble member second ensemble member original dataset first resampled dataset second resampled dataset figure 7. 5 : a cartoon depiction of how bagging works. suppose we train an 8 detector on the dataset depicted above, containing an 8, a 6 and a 9. suppose we make two [UNK] resampled datasets. the bagging training procedure is to construct each of these datasets by sampling with replacement. the ﬁrst dataset omits the 9 and repeats the 8. on this dataset, the detector learns that a loop on top of the digit corresponds to an 8. on the second dataset, we repeat the 9 and omit the 6. in this case, the detector learns that a loop on the bottom of the digit corresponds to an 8. each of these individual classiﬁcation rules is brittle, but if we average their output then the detector is robust, achieving maximal conﬁdence only when both loops of the 8 are present. [UNK] kind of model using a [UNK] algorithm or objective function. bagging is a method that allows the same kind of model, training algorithm and objective function to be reused several times. speciﬁcally, bagging involves


the 8 are present. [UNK] kind of model using a [UNK] algorithm or objective function. bagging is a method that allows the same kind of model, training algorithm and objective function to be reused several times. speciﬁcally, bagging involves constructing k [UNK] datasets. each dataset has the same number of examples as the original dataset, but each dataset is constructed by sampling with replacement from the original dataset. this means that, with high probability, each dataset is missing some of the examples from the original dataset and also contains several duplicate examples ( on average around 2 / 3 of the examples from the original dataset are found in the resulting training set, if it has the same size as the original ). model i is then trained on dataset i. the [UNK] between which examples are included in each dataset result in [UNK] between the trained models. see ﬁgure for an example. 7. 5 neural networks reach a wide enough variety of solution points that they can often beneﬁt from model averaging even if all of the models are trained on the same dataset. [UNK] in random initialization, random selection of minibatches, [UNK] in hyperparameters, or [UNK] outcomes of non - deterministic imple


##ﬁt from model averaging even if all of the models are trained on the same dataset. [UNK] in random initialization, random selection of minibatches, [UNK] in hyperparameters, or [UNK] outcomes of non - deterministic imple - mentations of neural networks are often enough to cause [UNK] members of the 257


chapter 7. regularization for deep learning ensemble to make partially independent errors. model averaging is an extremely powerful and reliable method for reducing generalization error. its use is usually discouraged when benchmarking algorithms for scientiﬁc papers, because any machine learning algorithm can beneﬁt substan - tially from model averaging at the price of increased computation and memory. for this reason, benchmark comparisons are usually made using a single model. machine learning contests are usually won by methods using model averag - ing over dozens of models. a recent prominent example is the netﬂix grand prize ( koren 2009, ). not all techniques for constructing ensembles are designed to make the ensemble more regularized than the individual models. for example, a technique called boosting ( freund and schapire 1996b a,, ) constructs an ensemble with higher capacity than the individual models. boosting has been applied to build ensembles of neural networks ( schwenk and bengio 1998, ) by incrementally adding neural networks to the ensemble. boosting has also been applied interpreting an individual neural network as an ensemble (, ), incrementally adding hidden bengio et al. 2006a units to the neural network. 7. 12 drop


##entally adding neural networks to the ensemble. boosting has also been applied interpreting an individual neural network as an ensemble (, ), incrementally adding hidden bengio et al. 2006a units to the neural network. 7. 12 dropout dropout ( srivastava 2014 et al., ) provides a computationally inexpensive but powerful method of regularizing a broad family of models. to a ﬁrst approximation, dropout can be thought of as a method of making bagging practical for ensembles of very many large neural networks. bagging involves training multiple models, and evaluating multiple models on each test example. this seems impractical when each model is a large neural network, since training and evaluating such networks is costly in terms of runtime and memory. it is common to use ensembles of ﬁve to ten neural networks — ( ) used six to win the ilsvrc — szegedy et al. 2014a but more than this rapidly becomes unwieldy. dropout provides an inexpensive approximation to training and evaluating a bagged ensemble of exponentially many neural networks. speciﬁcally, dropout trains the ensemble consisting of all sub - networks that can be formed by removing non - output units from an underlying base network,


inexpensive approximation to training and evaluating a bagged ensemble of exponentially many neural networks. speciﬁcally, dropout trains the ensemble consisting of all sub - networks that can be formed by removing non - output units from an underlying base network, as illustrated in ﬁgure. in most modern neural networks, based on a series of 7. 6 [UNK] transformations and nonlinearities, we can [UNK] remove a unit from a network by multiplying its output value by zero. this procedure requires some slight modiﬁcation for models such as radial basis function networks, which take 258


chapter 7. regularization for deep learning the [UNK] between the unit ’ s state and some reference value. here, we present the dropout algorithm in terms of multiplication by zero for simplicity, but it can be trivially modiﬁed to work with other operations that remove a unit from the network. recall that to learn with bagging, we deﬁne k [UNK] models, construct k [UNK] datasets by sampling from the training set with replacement, and then train model i on dataset i. dropout aims to approximate this process, but with an exponentially large number of neural networks. speciﬁcally, to train with dropout, we use a minibatch - based learning algorithm that makes small steps, such as stochastic gradient descent. each time we load an example into a minibatch, we randomly sample a [UNK] binary mask to apply to all of the input and hidden units in the network. the mask for each unit is sampled independently from all of the others. the probability of sampling a mask value of one ( causing a unit to be included ) is a hyperparameter ﬁxed before training begins. it is not a function of the current value of the model parameters or the input example. typically, an input unit is included


sampling a mask value of one ( causing a unit to be included ) is a hyperparameter ﬁxed before training begins. it is not a function of the current value of the model parameters or the input example. typically, an input unit is included with probability 0. 8 and a hidden unit is included with probability 0. 5. we then run forward propagation, back - propagation, and the learning update as usual. figure illustrates how to run forward propagation 7. 7 with dropout. more formally, suppose that a mask vector µ speciﬁes which units to include, and j ( θ µ, ) deﬁnes the cost of the model deﬁned by parameters θ and mask µ. then dropout training consists in minimizing eµj ( θ µ, ). the expectation contains exponentially many terms but we can obtain an unbiased estimate of its gradient by sampling values of. µ dropout training is not quite the same as bagging training. in the case of bagging, the models are all independent. in the case of dropout, the models share parameters, with each model inheriting a [UNK] subset of parameters from the parent neural network. this parameter sharing makes it possible to represent an exponential number of models with a tractable amount


are all independent. in the case of dropout, the models share parameters, with each model inheriting a [UNK] subset of parameters from the parent neural network. this parameter sharing makes it possible to represent an exponential number of models with a tractable amount of memory. in the case of bagging, each model is trained to convergence on its respective training set. in the case of dropout, typically most models are not explicitly trained at all — usually, the model is large enough that it would be infeasible to sample all possible sub - networks within the lifetime of the universe. instead, a tiny fraction of the possible sub - networks are each trained for a single step, and the parameter sharing causes the remaining sub - networks to arrive at good settings of the parameters. these are the only [UNK]. beyond these, dropout follows the bagging algorithm. for example, the training set encountered by each sub - network is indeed a subset of the original training set sampled with replacement. 259


chapter 7. regularization for deep learning y h1 h1 h2 h2 x1 x1 x2 x2 y h1 h1 h2 h2 x1 x1 x2 x2 y h1 h1 h2 h2 x2 x2 y h1 h1 h2 h2 x1 x1 y h2 h2 x1 x1 x2 x2 y h1 h1 x1 x1 x2 x2 y h1 h1 h2 h2 y x1 x1 x2 x2 y h2 h2 x2 x2 y h1 h1 x1 x1 y h1 h1 x2 x2 y h2 h2 x1 x1 y x1 x1 y x2 x2 y h2 h2 y h1 h1 y base network ensemble of subnetworks figure 7. 6 : dropout trains an ensemble consisting of all sub - networks that can be constructed by removing non - output units from an underlying base network. here, we begin with a base network with two visible units and two hidden units. there are sixteen possible subsets of these four units. we show all sixteen subnetworks that may be formed by dropping out [UNK] subsets of units from


network. here, we begin with a base network with two visible units and two hidden units. there are sixteen possible subsets of these four units. we show all sixteen subnetworks that may be formed by dropping out [UNK] subsets of units from the original network. in this small example, a large proportion of the resulting networks have no input units or no path connecting the input to the output. this problem becomes insigniﬁcant for networks with wider layers, where the probability of dropping all possible paths from inputs to outputs becomes smaller. 260


chapter 7. regularization for deep learning [UNK] µx1 µx1 x1 x1 [UNK] x2 x2 µx2 µx2 h1 h1 h2 h2 µh1 µh1 µh2 µh2 [UNK] [UNK] h1 [UNK] y y h1 h1 h2 h2 x1 x1 x2 x2 figure 7. 7 : an example of forward propagation through a feedforward network using dropout. ( top ) in this example, we use a feedforward network with two input units, one hidden layer with two hidden units, and one output unit. to perform forward ( bottom ) propagation with dropout, we randomly sample a vector µ with one entry for each input or hidden unit in the network. the entries of µ are binary and are sampled independently from each other. the probability of each entry being is a hyperparameter, usually 1 0. 5 for the hidden layers and 0. 8 for the input. each unit in the network is multiplied by the corresponding mask, and then forward propagation continues through the rest of the network as usual. this is equivalent to randomly selecting one of the sub - networks from ﬁgure and running forward propagation through it. 7. 6 261


chapter 7. regularization for deep learning to make a prediction, a bagged ensemble must accumulate votes from all of its members. we refer to this process as inference in this context. so far, our description of bagging and dropout has not required that the model be explicitly probabilistic. now, we assume that the model ’ s role is to output a probability distribution. in the case of bagging, each model iproduces a probability distribution p ( ) i ( y | x ). the prediction of the ensemble is given by the arithmetic mean of all of these distributions, 1 k k i = 1 p ( ) i ( ) y | x. ( 7. 52 ) in the case of dropout, each sub - model deﬁned by mask vector µ deﬁnes a prob - ability distribution p ( y, | x µ ). the arithmetic mean over all masks is given by µ p p y, ( ) µ ( | x µ ) ( 7. 53 ) where p ( µ ) is the probability distribution that was used to sample µ at training time. because this sum includes an exponential number of terms, it is intractable to evaluate except in cases where the structure of the model permits some form of simpli


p ( µ ) is the probability distribution that was used to sample µ at training time. because this sum includes an exponential number of terms, it is intractable to evaluate except in cases where the structure of the model permits some form of simpliﬁcation. so far, deep neural nets are not known to permit any tractable simpliﬁcation. instead, we can approximate the inference with sampling, by averaging together the output from many masks. even 10 - 20 masks are often [UNK] to obtain good performance. however, there is an even better approach, that allows us to obtain a good approximation to the predictions of the entire ensemble, at the cost of only one forward propagation. to do so, we change to using the geometric mean rather than the arithmetic mean of the ensemble members ’ predicted distributions. warde - farley 2014 et al. ( ) present arguments and empirical evidence that the geometric mean performs comparably to the arithmetic mean in this context. the geometric mean of multiple probability distributions is not guaranteed to be a probability distribution. to guarantee that the result is a probability distribution, we impose the requirement that none of the sub - models assigns probability 0 to any event, and we renormalize the resulting distribution. the unnormalized probability distribution


be a probability distribution. to guarantee that the result is a probability distribution, we impose the requirement that none of the sub - models assigns probability 0 to any event, and we renormalize the resulting distribution. the unnormalized probability distribution deﬁned directly by the geometric mean is given by [UNK] ( ) = y | x 2d µ p y, ( | x µ ) ( 7. 54 ) where d is the number of units that may be dropped. here we use a uniform distribution over µ to simplify the presentation, but non - uniform distributions are 262


chapter 7. regularization for deep learning also possible. to make predictions we must re - normalize the ensemble : pensemble ( ) = y | x [UNK] ( ) y | x y [UNK] ( y | x ). ( 7. 55 ) a key insight (, ) involved in dropout is that we can approxi - hinton et al. 2012c mate pensemble by evaluating p ( y | x ) in one model : the model with all units, but with the weights going out of unit i multiplied by the probability of including unit i. the motivation for this modiﬁcation is to capture the right expected value of the output from that unit. we call this approach the weight scaling inference rule. there is not yet any theoretical argument for the accuracy of this approximate inference rule in deep nonlinear networks, but empirically it performs very well. because we usually use an inclusion probability of 1 2, the weight scaling rule usually amounts to dividing the weights by at the end of training, and then using 2 the model as usual. another way to achieve the same result is to multiply the states of the units by during training. either way, the goal is to make sure that 2 the expected total input to a unit at test time is roughly the


2 the model as usual. another way to achieve the same result is to multiply the states of the units by during training. either way, the goal is to make sure that 2 the expected total input to a unit at test time is roughly the same as the expected total input to that unit at train time, even though half the units at train time are missing on average. for many classes of models that do not have nonlinear hidden units, the weight scaling inference rule is exact. for a simple example, consider a softmax regression classiﬁer with input variables represented by the vector : n v p y ( = y | v ) = softmax w v + b y. ( 7. 56 ) we can index into the family of sub - models by element - wise multiplication of the input with a binary vector : d p y ( = y | v ; ) = d softmax w ( ) + d v b y. ( 7. 57 ) the ensemble predictor is deﬁned by re - normalizing the geometric mean over all ensemble members ’ predictions : pensemble ( = ) = y y | v [UNK] ( = ) y y | v [UNK] ( = y y | v ) ( 7. 58 ) where [UNK] ( = ) = y


##izing the geometric mean over all ensemble members ’ predictions : pensemble ( = ) = y y | v [UNK] ( = ) y y | v [UNK] ( = y y | v ) ( 7. 58 ) where [UNK] ( = ) = y y | v 2n d∈ { } 0 1, n p y. ( = y | v ; ) d ( 7. 59 ) 263


chapter 7. regularization for deep learning to see that the weight scaling rule is exact, we can simplify [UNK] : [UNK] ( = ) = y y | v 2n d∈ { } 0 1, n p y ( = y | v ; ) d ( 7. 60 ) = 2n d∈ { } 0 1, n softmax ( w ( ) + ) d v b y ( 7. 61 ) = 2n d∈ { } 0 1, n exp w y, : ( ) + d v by yexp w y, : ( ) + d v by ( 7. 62 ) = 2n d∈ { } 0 1, n exp w y, : ( ) + d v b y 2n d∈ { } 0 1, n yexp w y, : ( ) + d v by ( 7. 63 ) because [UNK] will be normalized, we can safely ignore multiplication by factors that are constant with respect to : y [UNK] ( = ) y y | v [UNK] 2n d∈ { } 0 1, n exp w y, : ( ) + d v by ( 7. 64 ) = exp 1 2 n d∈ { } 0 1, n w y, : (


y | v [UNK] 2n d∈ { } 0 1, n exp w y, : ( ) + d v by ( 7. 64 ) = exp 1 2 n d∈ { } 0 1, n w y, : ( ) + d v by ( 7. 65 ) = exp 1 2w y, : v + by. ( 7. 66 ) substituting this back into equation we obtain a softmax classiﬁer with weights 7. 58 1 2w. the weight scaling rule is also exact in other settings, including regression networks with conditionally normal outputs, and deep networks that have hidden layers without nonlinearities. however, the weight scaling rule is only an approxi - mation for deep models that have nonlinearities. though the approximation has not been theoretically characterized, it often works well, empirically. goodfellow et al. ( ) found experimentally that the weight scaling approximation can work 2013a better ( in terms of classiﬁcation accuracy ) than monte carlo approximations to the ensemble predictor. this held true even when the monte carlo approximation was allowed to sample up to 1, 000 sub - networks. ( ) found gal and ghahramani 2015 that some models obtain better classiﬁcation accuracy


approximations to the ensemble predictor. this held true even when the monte carlo approximation was allowed to sample up to 1, 000 sub - networks. ( ) found gal and ghahramani 2015 that some models obtain better classiﬁcation accuracy using twenty samples and 264


chapter 7. regularization for deep learning the monte carlo approximation. it appears that the optimal choice of inference approximation is problem - dependent. srivastava 2014 et al. ( ) showed that dropout is more [UNK] than other standard computationally inexpensive regularizers, such as weight decay, ﬁlter norm constraints and sparse activity regularization. dropout may also be combined with other forms of regularization to yield a further improvement. one advantage of dropout is that it is very computationally cheap. using dropout during training requires only o ( n ) computation per example per update, to generate n random binary numbers and multiply them by the state. depending on the implementation, it may also require o ( n ) memory to store these binary numbers until the back - propagation stage. running inference in the trained model has the same cost per - example as if dropout were not used, though we must pay the cost of dividing the weights by 2 once before beginning to run inference on examples. another signiﬁcant advantage of dropout is that it does not signiﬁcantly limit the type of model or training procedure that can be used. it works well with nearly any model that uses a distributed representation and can be trained with stochastic gradient descent.


advantage of dropout is that it does not signiﬁcantly limit the type of model or training procedure that can be used. it works well with nearly any model that uses a distributed representation and can be trained with stochastic gradient descent. this includes feedforward neural networks, probabilistic models such as restricted boltzmann machines ( srivastava 2014 et al., ), and recurrent neural networks ( bayer and osendorfer 2014 pascanu 2014a, ; et al., ). many other regularization strategies of comparable power impose more severe restrictions on the architecture of the model. though the cost per - step of applying dropout to a speciﬁc model is negligible, the cost of using dropout in a complete system can be signiﬁcant. because dropout is a regularization technique, it reduces the [UNK] capacity of a model. to [UNK] this [UNK], we must increase the size of the model. typically the optimal validation set error is much lower when using dropout, but this comes at the cost of a much larger model and many more iterations of the training algorithm. for very large datasets, regularization confers little reduction in generalization error. in these cases,


much lower when using dropout, but this comes at the cost of a much larger model and many more iterations of the training algorithm. for very large datasets, regularization confers little reduction in generalization error. in these cases, the computational cost of using dropout and larger models may outweigh the beneﬁt of regularization. when extremely few labeled training examples are available, dropout is less [UNK]. bayesian neural networks (, ) outperform dropout on the neal 1996 alternative splicing dataset (, ) where fewer than 5, 000 examples xiong et al. 2011 are available ( srivastava 2014 et al., ). when additional unlabeled data is available, unsupervised feature learning can gain an advantage over dropout. wager 2013 et al. ( ) showed that, when applied to linear regression, dropout is equivalent to l2 weight decay, with a [UNK] weight decay [UNK] for 265


chapter 7. regularization for deep learning each input feature. the magnitude of each feature ’ s weight decay [UNK] is determined by its variance. similar results hold for other linear models. for deep models, dropout is not equivalent to weight decay. the stochasticity used while training with dropout is not necessary for the approach ’ s success. it is just a means of approximating the sum over all sub - models. wang and manning 2013 ( ) derived analytical approximations to this marginalization. their approximation, known as fast dropout resulted in faster convergence time due to the reduced stochasticity in the computation of the gradient. this method can also be applied at test time, as a more principled ( but also more computationally expensive ) approximation to the average over all sub - networks than the weight scaling approximation. fast dropout has been used to nearly match the performance of standard dropout on small neural network problems, but has not yet yielded a signiﬁcant improvement or been applied to a large problem. just as stochasticity is not necessary to achieve the regularizing [UNK] of dropout, it is also not [UNK]. to demonstrate this, warde - farley 2014 et al. ( ) designed control experiments using a method called drop


problem. just as stochasticity is not necessary to achieve the regularizing [UNK] of dropout, it is also not [UNK]. to demonstrate this, warde - farley 2014 et al. ( ) designed control experiments using a method called dropout boosting that they designed to use exactly the same mask noise as traditional dropout but lack its regularizing [UNK]. dropout boosting trains the entire ensemble to jointly maximize the log - likelihood on the training set. in the same sense that traditional dropout is analogous to bagging, this approach is analogous to boosting. as intended, experiments with dropout boosting show almost no regularization [UNK] compared to training the entire network as a single model. this demonstrates that the interpretation of dropout as bagging has value beyond the interpretation of dropout as robustness to noise. the regularization [UNK] of the bagged ensemble is only achieved when the stochastically sampled ensemble members are trained to perform well independently of each other. dropout has inspired other stochastic approaches to training exponentially large ensembles of models that share weights. dropconnect is a special case of dropout where each product between a single scalar weight and a single hidden unit state is considered a unit that can be dropped ( wan 2013


approaches to training exponentially large ensembles of models that share weights. dropconnect is a special case of dropout where each product between a single scalar weight and a single hidden unit state is considered a unit that can be dropped ( wan 2013 et al., ). stochastic pooling is a form of randomized pooling ( see section ) for building ensembles 9. 3 of convolutional networks with each convolutional network attending to [UNK] spatial locations of each feature map. so far, dropout remains the most widely used implicit ensemble method. one of the key insights of dropout is that training a network with stochastic behavior and making predictions by averaging over multiple stochastic decisions implements a form of bagging with parameter sharing. earlier, we described 266


chapter 7. regularization for deep learning dropout as bagging an ensemble of models formed by including or excluding units. however, there is no need for this model averaging strategy to be based on inclusion and exclusion. in principle, any kind of random modiﬁcation is admissible. in practice, we must choose modiﬁcation families that neural networks are able to learn to resist. ideally, we should also use model families that allow a fast approximate inference rule. we can think of any form of modiﬁcation parametrized by a vector µ as training an ensemble consisting of p ( y, | x µ ) for all possible values of µ. there is no requirement that µ have a ﬁnite number of values. for example, µ can be real - valued. srivastava 2014 et al. ( ) showed that multiplying the weights by µ [UNK] ( 1, i ) can outperform dropout based on binary masks. because e [ µ ] = 1 the standard network automatically implements approximate inference in the ensemble, without needing any weight scaling. so far we have described dropout purely as a means of performing [UNK], approximate bagging. however, there is another view of dropout that goes further than this. dropout trains


implements approximate inference in the ensemble, without needing any weight scaling. so far we have described dropout purely as a means of performing [UNK], approximate bagging. however, there is another view of dropout that goes further than this. dropout trains not just a bagged ensemble of models, but an ensemble of models that share hidden units. this means each hidden unit must be able to perform well regardless of which other hidden units are in the model. hidden units must be prepared to be swapped and interchanged between models. hinton et al. ( ) were inspired by an idea from biology : sexual reproduction, which involves 2012c swapping genes between two [UNK] organisms, creates evolutionary pressure for genes to become not just good, but to become readily swapped between [UNK] organisms. such genes and such features are very robust to changes in their environment because they are not able to incorrectly adapt to unusual features of any one organism or model. dropout thus regularizes each hidden unit to be not merely a good feature but a feature that is good in many contexts. warde - farley 2014 et al. ( ) compared dropout training to training of large ensembles and concluded that dropout [UNK] additional improvements to generalization error beyond those obtained by ensembles of independent models. it is important to