learn to implement a totally [UNK] function altogether. the maxout layer will of course be parametrized [UNK] from any of these other layer types, so the learning dynamics will be [UNK] even in the cases where maxout learns to implement the same function of as one of the other layer types. x each maxout unit is now parametrized by k weight vectors instead of just one, so maxout units typically need more regularization than rectiﬁed linear units. they can work well without regularization if the training set is large and the number of pieces per unit is kept low (, ). cai et al. 2013 maxout units have a few other beneﬁts. in some cases, one can gain some sta - tistical and computational advantages by requiring fewer parameters. speciﬁcally, if the features captured by n [UNK] linear ﬁlters can be summarized without losing information by taking the max over each group of k features, then the next layer can get by with times fewer weights. k because each unit is driven by multiple ﬁlters, maxout units have some redun - dancy that helps them to resist a phenomenon called catastrophic forgetting in which neural networks forget how to perform tasks that they were trained on in the past (,


because each unit is driven by multiple ﬁlters, maxout units have some redun - dancy that helps them to resist a phenomenon called catastrophic forgetting in which neural networks forget how to perform tasks that they were trained on in the past (, ). goodfellow et al. 2014a rectiﬁed linear units and all of these generalizations of them are based on the principle that models are easier to optimize if their behavior is closer to linear. this same general principle of using linear behavior to obtain easier optimization also applies in other contexts besides deep linear networks. recurrent networks can learn from sequences and produce a sequence of states and outputs. when training them, one needs to propagate information through several time steps, which is much easier when some linear computations ( with some directional derivatives being of magnitude near 1 ) are involved. one of the best - performing recurrent network 194

/home/ricoiban/GEMMA/mnlp_chatsplaining/RAG/Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville (z-lib.org).pdf


chapter 6. deep feedforward networks architectures, the lstm, propagates information through time via summation — a particular straightforward kind of such linear activation. this is discussed further in section. 10. 10 6. 3. 2 logistic sigmoid and hyperbolic tangent prior to the introduction of rectiﬁed linear units, most neural networks used the logistic sigmoid activation function g z σ z ( ) = ( ) ( 6. 38 ) or the hyperbolic tangent activation function g z z. ( ) = tanh ( ) ( 6. 39 ) these activation functions are closely related because. tanh ( ) = 2 ( 2 ) 1 z σ z − we have already seen sigmoid units as output units, used to predict the probability that a binary variable is. unlike piecewise linear units, sigmoidal 1 units saturate across most of their domain — they saturate to a high value when z is very positive, saturate to a low value when z is very negative, and are only strongly sensitive to their input when z is near 0. the widespread saturation of sigmoidal units can make gradient - based learning very [UNK]. for this reason, their use as hidden units in


value when z is very negative, and are only strongly sensitive to their input when z is near 0. the widespread saturation of sigmoidal units can make gradient - based learning very [UNK]. for this reason, their use as hidden units in feedforward networks is now discouraged. their use as output units is compatible with the use of gradient - based learning when an appropriate cost function can undo the saturation of the sigmoid in the output layer. when a sigmoidal activation function must be used, the hyperbolic tangent activation function typically performs better than the logistic sigmoid. it resembles the identity function more closely, in the sense that tanh ( 0 ) = 0 while σ ( 0 ) = 1 2. because tanh is similar to the identity function near, training a deep neural 0 network [UNK] = wtanh ( u tanh ( v x ) ) resembles training a linear model [UNK] = wuv x so long as the activations of the network can be kept small. this makes training the network easier. tanh sigmoidal activation functions are more common in settings other than feed - forward networks. recurrent networks, many probabilistic models, and some autoencoders have additional requirements that rule out the use


training the network easier. tanh sigmoidal activation functions are more common in settings other than feed - forward networks. recurrent networks, many probabilistic models, and some autoencoders have additional requirements that rule out the use of piecewise linear activation functions and make sigmoidal units more appealing despite the drawbacks of saturation. 195


chapter 6. deep feedforward networks 6. 3. 3 other hidden units many other types of hidden units are possible, but are used less frequently. in general, a wide variety of [UNK] functions perform perfectly well. many unpublished activation functions perform just as well as the popular ones. to provide a concrete example, the authors tested a feedforward network using h = cos ( wx + b ) on the mnist dataset and obtained an error rate of less than 1 %, which is competitive with results obtained using more conventional activation functions. during research and development of new techniques, it is common to test many [UNK] activation functions and ﬁnd that several variations on standard practice perform comparably. this means that usually new hidden unit types are published only if they are clearly demonstrated to provide a signiﬁcant improvement. new hidden unit types that perform roughly comparably to known types are so common as to be uninteresting. it would be impractical to list all of the hidden unit types that have appeared in the literature. we highlight a few especially useful and distinctive ones. one possibility is to not have an activation g ( z ) at all. one can also think of this as using the identity function as the activation function. we have


that have appeared in the literature. we highlight a few especially useful and distinctive ones. one possibility is to not have an activation g ( z ) at all. one can also think of this as using the identity function as the activation function. we have already seen that a linear unit can be useful as the output of a neural network. it may also be used as a hidden unit. if every layer of the neural network consists of only linear transformations, then the network as a whole will be linear. however, it is acceptable for some layers of the neural network to be purely linear. consider a neural network layer with n inputs and p outputs, h = g ( w x + b ). we may replace this with two layers, with one layer using weight matrix u and the other using weight matrix v. if the ﬁrst layer has no activation function, then we have essentially factored the weight matrix of the original layer based on w. the factored approach is to compute h = g ( v ux + b ). if u produces q outputs, then u and v together contain only ( n + p ) q parameters, while w contains np parameters. for small q, this can be a considerable saving in parameters. it comes at the cost of constraining


. if u produces q outputs, then u and v together contain only ( n + p ) q parameters, while w contains np parameters. for small q, this can be a considerable saving in parameters. it comes at the cost of constraining the linear transformation to be low - rank, but these low - rank relationships are often [UNK]. linear hidden units thus [UNK] an [UNK] way of reducing the number of parameters in a network. softmax units are another kind of unit that is usually used as an output ( as described in section ) but may sometimes be used as a hidden unit. softmax 6. 2. 2. 3 units naturally represent a probability distribution over a discrete variable with k possible values, so they may be used as a kind of switch. these kinds of hidden units are usually only used in more advanced architectures that explicitly learn to manipulate memory, described in section. 10. 12 196


chapter 6. deep feedforward networks a few other reasonably common hidden unit types include : • radial basis function or rbf unit : hi = exp −1 σ 2 i | | w :, i − | | x 2. this function becomes more active as x approaches a template w :, i. because it saturates to for most, it can be [UNK] to optimize. 0 x • softplus : g ( a ) = ζ ( a ) = log ( 1 + ea ). this is a smooth version of the rectiﬁer, introduced by ( ) for function approximation and by dugas et al. 2001 nair and hinton 2010 ( ) for the conditional distributions of undirected probabilistic models. ( ) compared the softplus and rectiﬁer and found glorot et al. 2011a better results with the latter. the use of the softplus is generally discouraged. the softplus demonstrates that the performance of hidden unit types can be very counterintuitive — one might expect it to have an advantage over the rectiﬁer due to being [UNK] everywhere or due to saturating less completely, but empirically it does not. • hard tanh : this is shaped similarly


very counterintuitive — one might expect it to have an advantage over the rectiﬁer due to being [UNK] everywhere or due to saturating less completely, but empirically it does not. • hard tanh : this is shaped similarly to the tanh and the rectiﬁer but unlike the latter, it is bounded, g ( a ) = max ( −1, min ( 1, a ) ). it was introduced by ( ). collobert 2004 hidden unit design remains an active area of research and many useful hidden unit types remain to be discovered. 6. 4 architecture design another key design consideration for neural networks is determining the architecture. the word architecture refers to the overall structure of the network : how many units it should have and how these units should be connected to each other. most neural networks are organized into groups of units called layers. most neural network architectures arrange these layers in a chain structure, with each layer being a function of the layer that preceded it. in this structure, the ﬁrst layer is given by h ( 1 ) = g ( 1 ) w ( 1 ) x b + ( 1 ), ( 6. 40 ) the second layer is given by h ( 2 ) = g ( 2 ) w ( 2


the ﬁrst layer is given by h ( 1 ) = g ( 1 ) w ( 1 ) x b + ( 1 ), ( 6. 40 ) the second layer is given by h ( 2 ) = g ( 2 ) w ( 2 ) h ( 1 ) + b ( 2 ), ( 6. 41 ) and so on. 197


chapter 6. deep feedforward networks in these chain - based architectures, the main architectural considerations are to choose the depth of the network and the width of each layer. as we will see, a network with even one hidden layer is [UNK] to ﬁt the training set. deeper networks often are able to use far fewer units per layer and far fewer parameters and often generalize to the test set, but are also often harder to optimize. the ideal network architecture for a task must be found via experimentation guided by monitoring the validation set error. 6. 4. 1 universal approximation properties and depth a linear model, mapping from features to outputs via matrix multiplication, can by deﬁnition represent only linear functions. it has the advantage of being easy to train because many loss functions result in convex optimization problems when applied to linear models. unfortunately, we often want to learn nonlinear functions. at ﬁrst glance, we might presume that learning a nonlinear function requires designing a specialized model family for the kind of nonlinearity we want to learn. fortunately, feedforward networks with hidden layers provide a universal approxi - mation framework. speciﬁcally, the universal approximation theorem ( hornik et al., ;, ) states that a feedforward network


want to learn. fortunately, feedforward networks with hidden layers provide a universal approxi - mation framework. speciﬁcally, the universal approximation theorem ( hornik et al., ;, ) states that a feedforward network with a linear output 1989 cybenko 1989 layer and at least one hidden layer with any “ squashing ” activation function ( such as the logistic sigmoid activation function ) can approximate any borel measurable function from one ﬁnite - dimensional space to another with any desired non - zero amount of error, provided that the network is given enough hidden units. the derivatives of the feedforward network can also approximate the derivatives of the function arbitrarily well (, ). the concept of borel measurability hornik et al. 1990 is beyond the scope of this book ; for our purposes it [UNK] to say that any continuous function on a closed and bounded subset of rn is borel measurable and therefore may be approximated by a neural network. a neural network may also approximate any function mapping from any ﬁnite dimensional discrete space to another. while the original theorems were ﬁrst stated in terms of units with activation functions that saturate both for very negative and for very positive arguments


. a neural network may also approximate any function mapping from any ﬁnite dimensional discrete space to another. while the original theorems were ﬁrst stated in terms of units with activation functions that saturate both for very negative and for very positive arguments, universal approximation theorems have also been proved for a wider class of activation functions, which includes the now commonly used rectiﬁed linear unit (, ). leshno et al. 1993 the universal approximation theorem means that regardless of what function we are trying to learn, we know that a large mlp will be able to represent this function. however, we are not guaranteed that the training algorithm will be able to learn that function. even if the mlp is able to represent the function, learning can fail for two [UNK] reasons. first, the optimization algorithm used for training 198


chapter 6. deep feedforward networks may not be able to ﬁnd the value of the parameters that corresponds to the desired function. second, the training algorithm might choose the wrong function due to overﬁtting. recall from section that the “ no free lunch ” theorem shows that 5. 2. 1 there is no universally superior machine learning algorithm. feedforward networks provide a universal system for representing functions, in the sense that, given a function, there exists a feedforward network that approximates the function. there is no universal procedure for examining a training set of speciﬁc examples and choosing a function that will generalize to points not in the training set. the universal approximation theorem says that there exists a network large enough to achieve any degree of accuracy we desire, but the theorem does not say how large this network will be. ( ) provides some bounds on the barron 1993 size of a single - layer network needed to approximate a broad class of functions. unfortunately, in the worse case, an exponential number of hidden units ( possibly with one hidden unit corresponding to each input conﬁguration that needs to be distinguished ) may be required. this is easiest to see in the binary case : the number of possible binary functions on vectors v ∈ { 0, 1


hidden units ( possibly with one hidden unit corresponding to each input conﬁguration that needs to be distinguished ) may be required. this is easiest to see in the binary case : the number of possible binary functions on vectors v ∈ { 0, 1 } n is 22 n and selecting one such function requires 2n bits, which will in general require o ( 2n ) degrees of freedom. in summary, a feedforward network with a single layer is [UNK] to represent any function, but the layer may be infeasibly large and may fail to learn and generalize correctly. in many circumstances, using deeper models can reduce the number of units required to represent the desired function and can reduce the amount of generalization error. there exist families of functions which can be approximated [UNK] by an architecture with depth greater than some valued, but which require a much larger model if depth is restricted to be less than or equal to d. in many cases, the number of hidden units required by the shallow model is exponential in n. such results were ﬁrst proved for models that do not resemble the continuous, [UNK] neural networks used for machine learning, but have since been extended to these models. the ﬁrst results were for circuits of logic gates (, ). later hastad


. such results were ﬁrst proved for models that do not resemble the continuous, [UNK] neural networks used for machine learning, but have since been extended to these models. the ﬁrst results were for circuits of logic gates (, ). later hastad 1986 work extended these results to linear threshold units with non - negative weights (, ;, ), and then to networks with hastad and goldmann 1991 hajnal et al. 1993 continuous - valued activations (, ;, ). many modern maass 1992 maass et al. 1994 neural networks use rectiﬁed linear units. ( ) demonstrated leshno et al. 1993 that shallow networks with a broad family of non - polynomial activation functions, including rectiﬁed linear units, have universal approximation properties, but these results do not address the questions of depth or [UNK] — they specify only that a [UNK] wide rectiﬁer network could represent any function. montufar et al. 199


chapter 6. deep feedforward networks ( ) showed that functions representable with a deep rectiﬁer net can require 2014 an exponential number of hidden units with a shallow ( one hidden layer ) network. more precisely, they showed that piecewise linear networks ( which can be obtained from rectiﬁer nonlinearities or maxout units ) can represent functions with a number of regions that is exponential in the depth of the network. figure illustrates how 6. 5 a network with absolute value rectiﬁcation creates mirror images of the function computed on top of some hidden unit, with respect to the input of that hidden unit. each hidden unit speciﬁes where to fold the input space in order to create mirror responses ( on both sides of the absolute value nonlinearity ). by composing these folding operations, we obtain an exponentially large number of piecewise linear regions which can capture all kinds of regular ( e. g., repeating ) patterns. figure 6. 5 : an intuitive, geometric explanation of the exponential advantage of deeper rectiﬁer networks formally by ( ). montufar et al. 2014 ( left ) an absolute value rectiﬁcation unit has the same output for every pair of mirror points in its input. the mirror axis of symmetry is


of deeper rectiﬁer networks formally by ( ). montufar et al. 2014 ( left ) an absolute value rectiﬁcation unit has the same output for every pair of mirror points in its input. the mirror axis of symmetry is given by the hyperplane deﬁned by the weights and bias of the unit. a function computed on top of that unit ( the green decision surface ) will be a mirror image of a simpler pattern across that axis of symmetry. the function can be obtained ( center ) by folding the space around the axis of symmetry. another repeating pattern can ( right ) be folded on top of the ﬁrst ( by another downstream unit ) to obtain another symmetry ( which is now repeated four times, with two hidden layers ). figure reproduced with permission from ( ). montufar et al. 2014 more precisely, the main theorem in ( ) states that the montufar et al. 2014 number of linear regions carved out by a deep rectiﬁer network with d inputs, depth, and units per hidden layer, is l n o n d d l ( −1 ) nd, ( 6. 42 ) i. e., exponential in the depth. in the case of maxout networks with ﬁlters per l


, and units per hidden layer, is l n o n d d l ( −1 ) nd, ( 6. 42 ) i. e., exponential in the depth. in the case of maxout networks with ﬁlters per l k unit, the number of linear regions is o k ( 1 ) + l− d. ( 6. 43 ) 200


chapter 6. deep feedforward networks of course, there is no guarantee that the kinds of functions we want to learn in applications of machine learning ( and in particular for ai ) share such a property. we may also want to choose a deep model for statistical reasons. any time we choose a speciﬁc machine learning algorithm, we are implicitly stating some set of prior beliefs we have about what kind of function the algorithm should learn. choosing a deep model encodes a very general belief that the function we want to learn should involve composition of several simpler functions. this can be interpreted from a representation learning point of view as saying that we believe the learning problem consists of discovering a set of underlying factors of variation that can in turn be described in terms of other, simpler underlying factors of variation. alternately, we can interpret the use of a deep architecture as expressing a belief that the function we want to learn is a computer program consisting of multiple steps, where each step makes use of the previous step ’ s output. these intermediate outputs are not necessarily factors of variation, but can instead be analogous to counters or pointers that the network uses to organize its internal processing. empirically, greater depth does seem to result in better generalization for a wide variety of tasks (, ;


outputs are not necessarily factors of variation, but can instead be analogous to counters or pointers that the network uses to organize its internal processing. empirically, greater depth does seem to result in better generalization for a wide variety of tasks (, ;, ;, ; bengio et al. 2007 erhan et al. 2009 bengio 2009 mesnil 2011 ciresan 2012 krizhevsky 2012 sermanet et al., ; et al., ; et al., ; et al., 2013 farabet 2013 couprie 2013 kahou 2013 goodfellow ; et al., ; et al., ; et al., ; et al. et al., ; 2014d szegedy, ). see ﬁgure and ﬁgure for examples of 2014a 6. 6 6. 7 some of these empirical results. this suggests that using deep architectures does indeed express a useful prior over the space of functions the model learns. 6. 4. 2 other architectural considerations so far we have described neural networks as being simple chains of layers, with the main considerations being the depth of the network and the width of each layer. in practice, neural networks show considerably more diversity. many neural network architectures have been


architectural considerations so far we have described neural networks as being simple chains of layers, with the main considerations being the depth of the network and the width of each layer. in practice, neural networks show considerably more diversity. many neural network architectures have been developed for speciﬁc tasks. specialized architectures for computer vision called convolutional networks are described in chapter. feedforward networks may also be generalized to the 9 recurrent neural networks for sequence processing, described in chapter, which 10 have their own architectural considerations. in general, the layers need not be connected in a chain, even though this is the most common practice. many architectures build a main chain but then add extra architectural features to it, such as skip connections going from layer i to layer i + 2 or higher. these skip connections make it easier for the gradient to ﬂow from output layers to layers nearer the input. 201


chapter 6. deep feedforward networks 3 4 5 6 7 8 9 10 11 92 0. 92 5. 93 0. 93 5. 94 0. 94 5. 95 0. 95 5. 96 0. 96 5. test accuracy ( percent ) figure 6. 6 : empirical results showing that deeper networks generalize better when used to transcribe multi - digit numbers from photographs of addresses. data from goodfellow et al. ( ). the test set accuracy consistently increases with increasing depth. see 2014d ﬁgure for a control experiment demonstrating that other increases to the model size 6. 7 do not yield the same [UNK]. another key consideration of architecture design is exactly how to connect a pair of layers to each other. in the default neural network layer described by a linear transformation via a matrix w, every input unit is connected to every output unit. many specialized networks in the chapters ahead have fewer connections, so that each unit in the input layer is connected to only a small subset of units in the output layer. these strategies for reducing the number of connections reduce the number of parameters and the amount of computation required to evaluate the network, but are often highly problem - dependent. for example, convolutional networks, described in chapter, use specialized


layer. these strategies for reducing the number of connections reduce the number of parameters and the amount of computation required to evaluate the network, but are often highly problem - dependent. for example, convolutional networks, described in chapter, use specialized patterns of sparse connections 9 that are very [UNK] for computer vision problems. in this chapter, it is [UNK] to give much more speciﬁc advice concerning the architecture of a generic neural network. subsequent chapters develop the particular architectural strategies that have been found to work well for [UNK] application domains. 202


chapter 6. deep feedforward networks 0 0 0 2 0 4 0 6 0 8 1 0...... number of parameters ×108 91 92 93 94 95 96 97 test accuracy ( percent ) 3, convolutional 3, fully connected 11, convolutional figure 6. 7 : deeper models tend to perform better. this is not merely because the model is larger. this experiment from goodfellow 2014d et al. ( ) shows that increasing the number of parameters in layers of convolutional networks without increasing their depth is not nearly as [UNK] at increasing test set performance. the legend indicates the depth of network used to make each curve and whether the curve represents variation in the size of the convolutional or the fully connected layers. we observe that shallow models in this context overﬁt at around 20 million parameters while deep ones can beneﬁt from having over 60 million. this suggests that using a deep model expresses a useful preference over the space of functions the model can learn. speciﬁcally, it expresses a belief that the function should consist of many simpler functions composed together. this could result either in learning a representation that is composed in turn of simpler representations ( e. g.


of functions the model can learn. speciﬁcally, it expresses a belief that the function should consist of many simpler functions composed together. this could result either in learning a representation that is composed in turn of simpler representations ( e. g., corners deﬁned in terms of edges ) or in learning a program with sequentially dependent steps ( e. g., ﬁrst locate a set of objects, then segment them from each other, then recognize them ). 203


chapter 6. deep feedforward networks 6. 5 back - propagation and other [UNK] algo - rithms when we use a feedforward neural network to accept an input x and produce an output [UNK], information ﬂows forward through the network. the inputs x provide the initial information that then propagates up to the hidden units at each layer and ﬁnally produces [UNK]. this is called forward propagation. during training, forward propagation can continue onward until it produces a scalar cost j ( θ ). the back - propagation algorithm (, ), often simply called rumelhart et al. 1986a backprop, allows the information from the cost to then ﬂow backwards through the network, in order to compute the gradient. computing an analytical expression for the gradient is straightforward, but numerically evaluating such an expression can be computationally expensive. the back - propagation algorithm does so using a simple and inexpensive procedure. the term back - propagation is often misunderstood as meaning the whole learning algorithm for multi - layer neural networks. actually, back - propagation refers only to the method for computing the gradient, while another algorithm, such as stochastic gradient descent, is used to perform learning using this gradient. furthermore, back - propagation is often misunderstood as being speciﬁ


. actually, back - propagation refers only to the method for computing the gradient, while another algorithm, such as stochastic gradient descent, is used to perform learning using this gradient. furthermore, back - propagation is often misunderstood as being speciﬁc to multi - layer neural networks, but in principle it can compute derivatives of any function ( for some functions, the correct response is to report that the derivative of the function is undeﬁned ). speciﬁcally, we will describe how to compute the gradient ∇xf ( x y, ) for an arbitrary function f, wherex is a set of variables whose derivatives are desired, and y is an additional set of variables that are inputs to the function but whose derivatives are not required. in learning algorithms, the gradient we most often require is the gradient of the cost function with respect to the parameters, ∇θj ( θ ). many machine learning tasks involve computing other derivatives, either as part of the learning process, or to analyze the learned model. the back - propagation algorithm can be applied to these tasks as well, and is not restricted to computing the gradient of the cost function with respect to the parameters. the idea of computing derivatives by propagating information through a network is very general,


. the back - propagation algorithm can be applied to these tasks as well, and is not restricted to computing the gradient of the cost function with respect to the parameters. the idea of computing derivatives by propagating information through a network is very general, and can be used to compute values such as the jacobian of a function f with multiple outputs. we restrict our description here to the most commonly used case where has a single output. f 204


chapter 6. deep feedforward networks 6. 5. 1 computational graphs so far we have discussed neural networks with a relatively informal graph language. to describe the back - propagation algorithm more precisely, it is helpful to have a more precise language. computational graph many ways of formalizing computation as graphs are possible. here, we use each node in the graph to indicate a variable. the variable may be a scalar, vector, matrix, tensor, or even a variable of another type. to formalize our graphs, we also need to introduce the idea of an operation. an operation is a simple function of one or more variables. our graph language is accompanied by a set of allowable operations. functions more complicated than the operations in this set may be described by composing many operations together. without loss of generality, we deﬁne an operation to return only a single output variable. this does not lose generality because the output variable can have multiple entries, such as a vector. software implementations of back - propagation usually support operations with multiple outputs, but we avoid this case in our description because it introduces many extra details that are not important to conceptual understanding. if a variable y is computed by applying an operation to a variable x, then we draw a directed edge from x


operations with multiple outputs, but we avoid this case in our description because it introduces many extra details that are not important to conceptual understanding. if a variable y is computed by applying an operation to a variable x, then we draw a directed edge from x to y. we sometimes annotate the output node with the name of the operation applied, and other times omit this label when the operation is clear from context. examples of computational graphs are shown in ﬁgure. 6. 8 6. 5. 2 chain rule of calculus the chain rule of calculus ( not to be confused with the chain rule of probability ) is used to compute the derivatives of functions formed by composing other functions whose derivatives are known. back - propagation is an algorithm that computes the chain rule, with a speciﬁc order of operations that is highly [UNK]. let x be a real number, and let f and g both be functions mapping from a real number to a real number. suppose that y = g ( x ) and z = f ( g ( x ) ) = f ( y ). then the chain rule states that dz dx = dz dy dy dx. ( 6. 44 ) we can generalize this beyond the scalar case. suppose that


( g ( x ) ) = f ( y ). then the chain rule states that dz dx = dz dy dy dx. ( 6. 44 ) we can generalize this beyond the scalar case. suppose that x ∈rm, y ∈rn, 205


chapter 6. deep feedforward networks z x y ( a ) × x w ( b ) u ( 1 ) u ( 1 ) dot b u ( 2 ) u ( 2 ) + [UNK] σ ( c ) x w u ( 1 ) u ( 1 ) matmul b u ( 2 ) u ( 2 ) + h relu x w ( d ) [UNK] dot λ u ( 1 ) u ( 1 ) sqr u ( 2 ) u ( 2 ) sum u ( 3 ) u ( 3 ) × figure 6. 8 : examples of computational graphs. the graph using the ( a ) × operation to compute z = xy. the graph for the logistic regression prediction ( b ) [UNK] = σ xw + b. some of the intermediate expressions do not have names in the algebraic expression but need names in the graph. we simply name the i - th such variable u ( ) i. the ( c ) computational graph for the expression h = max { 0, xw + b }, which computes a design matrix of rectiﬁed linear unit activations h given a design matrix containing a minibatch of inputs x. examples a – c applied at most one operation to each variable, but it ( d ) is possible to


which computes a design matrix of rectiﬁed linear unit activations h given a design matrix containing a minibatch of inputs x. examples a – c applied at most one operation to each variable, but it ( d ) is possible to apply more than one operation. here we show a computation graph that applies more than one operation to the weights w of a linear regression model. the weights are used to make both the prediction [UNK] and the weight decay penalty λ i w2 i. 206


chapter 6. deep feedforward networks g maps from rm to rn, and f maps from rn to r. if y = g ( x ) and z = f ( y ), then ∂z ∂xi = j ∂z ∂yj ∂yj ∂xi. ( 6. 45 ) in vector notation, this may be equivalently written as ∇xz = ∂y ∂x ∇y z, ( 6. 46 ) where ∂y ∂x is the jacobian matrix of. n m × g from this we see that the gradient of a variable x can be obtained by multiplying a jacobian matrix ∂y ∂x by a gradient ∇yz. the back - propagation algorithm consists of performing such a jacobian - gradient product for each operation in the graph. usually we do not apply the back - propagation algorithm merely to vectors, but rather to tensors of arbitrary dimensionality. conceptually, this is exactly the same as back - propagation with vectors. the only [UNK] is how the numbers are arranged in a grid to form a tensor. we could imagine ﬂattening each tensor into a vector before we run back - propagation, computing a vector - valued gradient, and then reshaping the gradient back into a tensor. in this rear


are arranged in a grid to form a tensor. we could imagine ﬂattening each tensor into a vector before we run back - propagation, computing a vector - valued gradient, and then reshaping the gradient back into a tensor. in this rearranged view, back - propagation is still just multiplying jacobians by gradients. to denote the gradient of a value z with respect to a tensor x, we write ∇xz, just as if x were a vector. the indices into x now have multiple coordinates — for example, a 3 - d tensor is indexed by three coordinates. we can abstract this away by using a single variable i to represent the complete tuple of indices. for all possible index tuples i, ( ∇xz ) i gives ∂z ∂xi. this is exactly the same as how for all possible integer indices i into a vector, ( ∇x z ) i gives ∂z ∂xi. using this notation, we can write the chain rule as it applies to tensors. if and, then y x = ( g ) z f = ( ) y ∇x z = j ( ∇xyj ) ∂z ∂yj. ( 6. 47 ) 6. 5. 3 recursively applying the chain rule


. if and, then y x = ( g ) z f = ( ) y ∇x z = j ( ∇xyj ) ∂z ∂yj. ( 6. 47 ) 6. 5. 3 recursively applying the chain rule to obtain backprop using the chain rule, it is straightforward to write down an algebraic expression for the gradient of a scalar with respect to any node in the computational graph that produced that scalar. however, actually evaluating that expression in a computer introduces some extra considerations. speciﬁcally, many subexpressions may be repeated several times within the overall expression for the gradient. any procedure that computes the gradient 207


chapter 6. deep feedforward networks will need to choose whether to store these subexpressions or to recompute them several times. an example of how these repeated subexpressions arise is given in ﬁgure. in some cases, computing the same subexpression twice would simply 6. 9 be wasteful. for complicated graphs, there can be exponentially many of these wasted computations, making a naive implementation of the chain rule infeasible. in other cases, computing the same subexpression twice could be a valid way to reduce memory consumption at the cost of higher runtime. we ﬁrst begin by a version of the back - propagation algorithm that speciﬁes the actual gradient computation directly ( algorithm along with algorithm for the 6. 2 6. 1 associated forward computation ), in the order it will actually be done and according to the recursive application of chain rule. one could either directly perform these computations or view the description of the algorithm as a symbolic speciﬁcation of the computational graph for computing the back - propagation. however, this formulation does not make explicit the manipulation and the construction of the symbolic graph that performs the gradient computation. such a formulation is presented below in section, with algorithm, where


##iﬁcation of the computational graph for computing the back - propagation. however, this formulation does not make explicit the manipulation and the construction of the symbolic graph that performs the gradient computation. such a formulation is presented below in section, with algorithm, where we also generalize to 6. 5. 6 6. 5 nodes that contain arbitrary tensors. first consider a computational graph describing how to compute a single scalar u ( ) n ( say the loss on a training example ). this scalar is the quantity whose gradient we want to obtain, with respect to the ni input nodes u ( 1 ) to u ( ni ). in other words we wish to compute ∂u ( ) n ∂u ( ) i for all i ∈ { 1, 2,..., ni }. in the application of back - propagation to computing gradients for gradient descent over parameters, u ( ) n will be the cost associated with an example or a minibatch, while u ( 1 ) to u ( ni ) correspond to the parameters of the model. we will assume that the nodes of the graph have been ordered in such a way that we can compute their output one after the other, starting at u ( ni + 1 ) and going up to u ( ) n


to the parameters of the model. we will assume that the nodes of the graph have been ordered in such a way that we can compute their output one after the other, starting at u ( ni + 1 ) and going up to u ( ) n. as deﬁned in algorithm, each node 6. 1 u ( ) i is associated with an operation f ( ) i and is computed by evaluating the function u ( ) i = ( f a ( ) i ) ( 6. 48 ) where a ( ) i is the set of all nodes that are parents of u ( ) i. that algorithm speciﬁes the forward propagation computation, which we could put in a graph g. in order to perform back - propagation, we can construct a computational graph that depends ong and adds to it an extra set of nodes. these form a subgraph b with one node per node of g. computation in b proceeds in exactly the reverse of the order of computation in g, and each node of b computes the derivative ∂u ( ) n ∂u ( ) i associated with the forward graph node u ( ) i. this is done 208


chapter 6. deep feedforward networks algorithm 6. 1 a procedure that performs the computations mapping ni inputs u ( 1 ) to u ( ni ) to an output u ( ) n. this deﬁnes a computational graph where each node computes numerical value u ( ) i by applying a function f ( ) i to the set of arguments a ( ) i that comprises the values of previous nodes u ( ) j, j < i, with j pa ∈ ( u ( ) i ). the input to the computational graph is the vector x, and is set into the ﬁrst ni nodes u ( 1 ) to u ( ni ). the output of the computational graph is read [UNK] last ( output ) node u ( ) n. for i,..., n = 1 i do u ( ) i ←xi end for for i n = i + 1,..., n do a ( ) i ← { u ( ) j | ∈ j pa u ( ( ) i ) } u ( ) i ←f ( ) i ( a ( ) i ) end for return u ( ) n using the chain rule with respect to scalar output u ( ) n : ∂u ( ) n ∂u ( ) j = i j p


( ) i ←f ( ) i ( a ( ) i ) end for return u ( ) n using the chain rule with respect to scalar output u ( ) n : ∂u ( ) n ∂u ( ) j = i j p a u : ∈ ( ( ) i ) ∂u ( ) n ∂u ( ) i ∂u ( ) i ∂u ( ) j ( 6. 49 ) as speciﬁed by algorithm. the subgraph 6. 2 b contains exactly one edge for each edge from node u ( ) j to node u ( ) i of g. the edge from u ( ) j to u ( ) i is associated with the computation of ∂u ( ) i ∂u ( ) j. in addition, a dot product is performed for each node, between the gradient already computed with respect to nodes u ( ) i that are children of u ( ) j and the vector containing the partial derivatives ∂u ( ) i ∂u ( ) j for the same children nodes u ( ) i. to summarize, the amount of computation required for performing the back - propagation scales linearly with the number of edges in g, where the computation for each edge corresponds to computing a partial derivative ( of one node with respect to


( ) i. to summarize, the amount of computation required for performing the back - propagation scales linearly with the number of edges in g, where the computation for each edge corresponds to computing a partial derivative ( of one node with respect to one of its parents ) as well as performing one multiplication and one addition. below, we generalize this analysis to tensor - valued nodes, which is just a way to group multiple scalar values in the same node and enable more [UNK] implementations. the back - propagation algorithm is designed to reduce the number of common subexpressions without regard to memory. speciﬁcally, it performs on the order of one jacobian product per node in the graph. this can be seen from the fact that backprop ( algorithm ) visits each edge from node 6. 2 u ( ) j to node u ( ) i of the graph exactly once in order to obtain the associated partial derivative ∂u ( ) i ∂u ( ) j. 209


chapter 6. deep feedforward networks algorithm 6. 2 simpliﬁed version of the back - propagation algorithm for computing the derivatives of u ( ) n with respect to the variables in the graph. this example is intended to further understanding by showing a simpliﬁed case where all variables are scalars, and we wish to compute the derivatives with respect to u ( 1 ),..., u ( ni ). this simpliﬁed version computes the derivatives of all nodes in the graph. the computational cost of this algorithm is proportional to the number of edges in the graph, assuming that the partial derivative associated with each edge requires a constant time. this is of the same order as the number of computations for the forward propagation. each ∂u ( ) i ∂u ( ) j is a function of the parents u ( ) j of u ( ) i, thus linking the nodes of the forward graph to those added for the back - propagation graph. run forward propagation ( algorithm for this example ) to obtain the activa - 6. 1 tions of the network initialize grad _ table, a data structure that will store the derivatives that have been computed. the entry grad table _ [ u ( ) i ] will


this example ) to obtain the activa - 6. 1 tions of the network initialize grad _ table, a data structure that will store the derivatives that have been computed. the entry grad table _ [ u ( ) i ] will store the computed value of ∂u ( ) n ∂u ( ) i. grad table _ [ u ( ) n ] 1 ← for do j n = −1 down to 1 the next line computes ∂u ( ) n ∂u ( ) j = i j p a u : ∈ ( ( ) i ) ∂u ( ) n ∂u ( ) i ∂u ( ) i ∂u ( ) j using stored values : grad table _ [ u ( ) j ] ← i j p a u : ∈ ( ( ) i ) grad table _ [ u ( ) i ] ∂u ( ) i ∂u ( ) j end for return { grad table _ [ u ( ) i ] = 1 | i,..., ni } back - propagation thus avoids the exponential explosion in repeated subexpressions. however, other algorithms may be able to avoid more subexpressions by performing simpliﬁcations on the computational graph, or may be able to conserve


ni } back - propagation thus avoids the exponential explosion in repeated subexpressions. however, other algorithms may be able to avoid more subexpressions by performing simpliﬁcations on the computational graph, or may be able to conserve memory by recomputing rather than storing some subexpressions. we will revisit these ideas after describing the back - propagation algorithm itself. 6. 5. 4 back - propagation computation in fully - connected mlp to clarify the above deﬁnition of the back - propagation computation, let us consider the speciﬁc graph associated with a fully - connected multi - layer mlp. algorithm ﬁrst shows the forward propagation, which maps parameters to 6. 3 the supervised loss l ( [UNK] y, ) associated with a single ( input, target ) training example ( ) x y,, with [UNK] the output of the neural network when is provided in input. x algorithm then shows the corresponding computation to be done for 6. 4 210


chapter 6. deep feedforward networks z x y w f f f figure 6. 9 : a computational graph that results in repeated subexpressions when computing the gradient. let w ∈r be the input to the graph. we use the same function f : r r → as the operation that we apply at every step of a chain : x = f ( w ), y = f ( x ), z = f ( y ). to compute ∂z ∂w, we apply equation and obtain : 6. 44 ∂z ∂w ( 6. 50 ) = ∂z ∂y ∂y ∂x ∂x ∂w ( 6. 51 ) = f ( ) y f ( ) x f ( ) w ( 6. 52 ) = f ( ( ( ) ) ) f f w f ( ( ) ) f w f ( ) w ( 6. 53 ) equation suggests an implementation in which we compute the value of 6. 52 f ( w ) only once and store it in the variable x. this is the approach taken by the back - propagation algorithm. an alternative approach is suggested by equation, where the subexpression 6. 53 f ( w ) appears more than once. in the alternative approach, f ( w ) is rec


. this is the approach taken by the back - propagation algorithm. an alternative approach is suggested by equation, where the subexpression 6. 53 f ( w ) appears more than once. in the alternative approach, f ( w ) is recomputed each time it is needed. when the memory required to store the value of these expressions is low, the back - propagation approach of equation is clearly preferable because of its reduced 6. 52 runtime. however, equation is also a valid implementation of the chain rule, and is 6. 53 useful when memory is limited. 211


chapter 6. deep feedforward networks applying the back - propagation algorithm to this graph. algorithms and are demonstrations that are chosen to be simple and 6. 3 6. 4 straightforward to understand. however, they are specialized to one speciﬁc problem. modern software implementations are based on the generalized form of back - propagation described in section below, which can accommodate any compu - 6. 5. 6 tational graph by explicitly manipulating a data structure for representing symbolic computation. algorithm 6. 3 forward propagation through a typical deep neural network and the computation of the cost function. the loss l ( [UNK] y, ) depends on the output [UNK] and on the target y ( see section for examples of loss functions ). to 6. 2. 1. 1 obtain the total cost j, the loss may be added to a regularizer ω ( θ ), where θ contains all the parameters ( weights and biases ). algorithm shows how to 6. 4 compute gradients of j with respect to parameters w and b. for simplicity, this demonstration uses only a single input example x. practical applications should use a minibatch. see section for a more realistic demonstration. 6. 5. 7 require : network depth, l require : w ( ) i, i,


for simplicity, this demonstration uses only a single input example x. practical applications should use a minibatch. see section for a more realistic demonstration. 6. 5. 7 require : network depth, l require : w ( ) i, i,..., l, ∈ { 1 } the weight matrices of the model require : b ( ) i, i,..., l, ∈ { 1 } the bias parameters of the model require : x, the input to process require : y, the target output h ( 0 ) = x for do k,..., l = 1 a ( ) k = b ( ) k + w ( ) k h ( 1 ) k− h ( ) k = ( f a ( ) k ) end for [UNK] h = ( ) l j l = ( [UNK] y, ) + ω ( ) λ θ 6. 5. 5 symbol - to - symbol derivatives algebraic expressions and computational graphs both operate on symbols, or variables that do not have speciﬁc values. these algebraic and graph - based representations are called symbolic representations. when we actually use or train a neural network, we must assign speciﬁc values to these symbols. we replace a symbolic input to the network x with a spec


##ﬁc values. these algebraic and graph - based representations are called symbolic representations. when we actually use or train a neural network, we must assign speciﬁc values to these symbols. we replace a symbolic input to the network x with a speciﬁc numeric value, such as [ 1 2 3 765 1 8 ].,., −.. 212


chapter 6. deep feedforward networks algorithm 6. 4 backward computation for the deep neural network of algo - rithm, which uses in addition to the input 6. 3 x a target y. this computation yields the gradients on the activations a ( ) k for each layer k, starting from the output layer and going backwards to the ﬁrst hidden layer. from these gradients, which can be interpreted as an indication of how each layer ’ s output should change to reduce error, one can obtain the gradient on the parameters of each layer. the gradients on weights and biases can be immediately used as part of a stochas - tic gradient update ( performing the update right after the gradients have been computed ) or used with other gradient - based optimization methods. after the forward computation, compute the gradient on the output layer : g [UNK] = [UNK] ( [UNK] y, ) for do k l, l,..., = −1 1 convert the gradient on the layer ’ s output into a gradient into the pre - nonlinearity activation ( element - wise multiplication if is element - wise ) : f g ←∇a ( ) k j f = g ( a ( ) k ) compute gradients on weights and biases ( including


into a gradient into the pre - nonlinearity activation ( element - wise multiplication if is element - wise ) : f g ←∇a ( ) k j f = g ( a ( ) k ) compute gradients on weights and biases ( including the regularization term, where needed ) : ∇b ( ) k j λ = + g ∇b ( ) k ω ( ) θ ∇w ( ) k j = g h ( 1 ) k− + λ∇w ( ) k ω ( ) θ propagate the gradients w. r. t. the next lower - level hidden layer ’ s activations : g ←∇h ( 1 ) k−j = w ( ) k g end for 213


chapter 6. deep feedforward networks z x y w f f f z x y w f f f dz dy dz dy f dy dx dy dx f dz dx dz dx × dx dw dx dw f dz dw dz dw × figure 6. 10 : an example of the symbol - to - symbol approach to computing derivatives. in this approach, the back - propagation algorithm does not need to ever access any actual speciﬁc numeric values. instead, it adds nodes to a computational graph describing how to compute these derivatives. a generic graph evaluation engine can later compute the derivatives for any speciﬁc numeric values. ( left ) in this example, we begin with a graph representing z = f ( f ( f ( w ) ) ). we run the back - propagation algorithm, instructing ( right ) it to construct the graph for the expression corresponding to dz dw. in this example, we do not explain how the back - propagation algorithm works. the purpose is only to illustrate what the desired result is : a computational graph with a symbolic description of the derivative. some approaches to back - propagation take a computational graph and a set of numerical values


do not explain how the back - propagation algorithm works. the purpose is only to illustrate what the desired result is : a computational graph with a symbolic description of the derivative. some approaches to back - propagation take a computational graph and a set of numerical values for the inputs to the graph, then return a set of numerical values describing the gradient at those input values. we call this approach “ symbol - to - number ” [UNK]. this is the approach used by libraries such as torch (, ) and [UNK] (, ). collobert et al. 2011b jia 2013 another approach is to take a computational graph and add additional nodes to the graph that provide a symbolic description of the desired derivatives. this is the approach taken by theano (, ;, ) bergstra et al. 2010 bastien et al. 2012 and tensorflow (, ). an example of how this approach works abadi et al. 2015 is illustrated in ﬁgure. the primary advantage of this approach is that 6. 10 the derivatives are described in the same language as the original expression. because the derivatives are just another computational graph, it is possible to run back - propagation again, [UNK] the derivatives in order to obtain higher derivatives. computation of higher - order derivatives is described in section.


described in the same language as the original expression. because the derivatives are just another computational graph, it is possible to run back - propagation again, [UNK] the derivatives in order to obtain higher derivatives. computation of higher - order derivatives is described in section. 6. 5. 10 we will use the latter approach and describe the back - propagation algorithm in 214


chapter 6. deep feedforward networks terms of constructing a computational graph for the derivatives. any subset of the graph may then be evaluated using speciﬁc numerical values at a later time. this allows us to avoid specifying exactly when each operation should be computed. instead, a generic graph evaluation engine can evaluate every node as soon as its parents ’ values are available. the description of the symbol - to - symbol based approach subsumes the symbol - to - number approach. the symbol - to - number approach can be understood as performing exactly the same computations as are done in the graph built by the symbol - to - symbol approach. the key [UNK] is that the symbol - to - number approach does not expose the graph. 6. 5. 6 general back - propagation the back - propagation algorithm is very simple. to compute the gradient of some scalar z with respect to one of its ancestors x in the graph, we begin by observing that the gradient with respect to z is given by dz dz = 1. we can then compute the gradient with respect to each parent of z in the graph by multiplying the current gradient by the jacobian of the operation that produced z. we continue multiplying by jacobians traveling backwards through the graph in


1. we can then compute the gradient with respect to each parent of z in the graph by multiplying the current gradient by the jacobian of the operation that produced z. we continue multiplying by jacobians traveling backwards through the graph in this way until we reach x. for any node that may be reached by going backwards from z through two or more paths, we simply sum the gradients arriving from [UNK] paths at that node. more formally, each node in the graph g corresponds to a variable. to achieve maximum generality, we describe this variable as being a tensor v. tensor can in general have any number of dimensions. they subsume scalars, vectors, and matrices. we assume that each variable is associated with the following subroutines : v • get operation _ ( v ) : this returns the operation that computes v, repre - sented by the edges coming into v in the computational graph. for example, there may be a python or c + + class representing the matrix multiplication operation, and the get _ operation function. suppose we have a variable that is created by matrix multiplication, c = ab. then get operation _ ( v ) returns a pointer to an instance of the corresponding c + + class. • get consumers _


operation, and the get _ operation function. suppose we have a variable that is created by matrix multiplication, c = ab. then get operation _ ( v ) returns a pointer to an instance of the corresponding c + + class. • get consumers _ ( v, g ) : this returns the list of variables that are children of v in the computational graph. g • g get inputs _ ( v, ) : this returns the list of variables that are parents of v in the computational graph. g 215


chapter 6. deep feedforward networks each operation op is also associated with a bprop operation. this bprop operation can compute a jacobian - vector product as described by equation. 6. 47 this is how the back - propagation algorithm is able to achieve great generality. each operation is responsible for knowing how to back - propagate through the edges in the graph that it participates in. for example, we might use a matrix multiplication operation to create a variable c = ab. suppose that the gradient of a scalar z with respect to c is given by g. the matrix multiplication operation is responsible for deﬁning two back - propagation rules, one for each of its input arguments. if we call the bprop method to request the gradient with respect to a given that the gradient on the output is g, then the bprop method of the matrix multiplication operation must state that the gradient with respect to a is given by gb. likewise, if we call the bprop method to request the gradient with respect to b, then the matrix operation is responsible for implementing the bprop method and specifying that the desired gradient is given by ag. the back - propagation algorithm itself does not need to know any [UNK] rules. it only needs to call each operation ’ s bp


, then the matrix operation is responsible for implementing the bprop method and specifying that the desired gradient is given by ag. the back - propagation algorithm itself does not need to know any [UNK] rules. it only needs to call each operation ’ s bprop rules with the right arguments. formally, op bprop inputs. (,, x g ) must return i ( ∇xop f inputs. ( ) i ) gi, ( 6. 54 ) which is just an implementation of the chain rule as expressed in equation. 6. 47 here, inputs is a list of inputs that are supplied to the operation, op. f is the mathematical function that the operation implements, x is the input whose gradient we wish to compute, and is the gradient on the output of the operation. g the op. bprop method should always pretend that all of its inputs are distinct from each other, even if they are not. for example, if the mul operator is passed two copies of x to compute x2, the op. bprop method should still return x as the derivative with respect to both inputs. the back - propagation algorithm will later add both of these arguments together to obtain 2x, which is the correct total derivative on. x software implementations of back - propagation usually


bprop method should still return x as the derivative with respect to both inputs. the back - propagation algorithm will later add both of these arguments together to obtain 2x, which is the correct total derivative on. x software implementations of back - propagation usually provide both the opera - tions and their bprop methods, so that users of deep learning software libraries are able to back - propagate through graphs built using common operations like matrix multiplication, exponents, logarithms, and so on. software engineers who build a new implementation of back - propagation or advanced users who need to add their own operation to an existing library must usually derive the op. bprop method for any new operations manually. the back - propagation algorithm is formally described in algorithm. 6. 5 216


chapter 6. deep feedforward networks algorithm 6. 5 the outermost skeleton of the back - propagation algorithm. this portion does simple setup and cleanup work. most of the important work happens in the subroutine of algorithm build _ grad 6. 6. require : t, the target set of variables whose gradients must be computed. require : g, the computational graph require : z, the variable to be [UNK] let gbe g pruned to contain only nodes that are ancestors of z and descendents of nodes in. t initialize, a data structure associating tensors to their gradients grad _ table grad table _ [ ] 1 z ← for do v in t build grad _ ( v,, g g, grad table _ ) end for return restricted to grad _ table t in section, we explained that back - propagation was developed in order to 6. 5. 2 avoid computing the same subexpression in the chain rule multiple times. the naive algorithm could have exponential runtime due to these repeated subexpressions. now that we have speciﬁed the back - propagation algorithm, we can understand its computational cost. if we assume that each operation evaluation has roughly the same cost, then we may


have exponential runtime due to these repeated subexpressions. now that we have speciﬁed the back - propagation algorithm, we can understand its computational cost. if we assume that each operation evaluation has roughly the same cost, then we may analyze the computational cost in terms of the number of operations executed. keep in mind here that we refer to an operation as the fundamental unit of our computational graph, which might actually consist of very many arithmetic operations ( for example, we might have a graph that treats matrix multiplication as a single operation ). computing a gradient in a graph with n nodes will never execute more than o ( n2 ) operations or store the output of more than o ( n2 ) operations. here we are counting operations in the computational graph, not individual operations executed by the underlying hardware, so it is important to remember that the runtime of each operation may be highly variable. for example, multiplying two matrices that each contain millions of entries might correspond to a single operation in the graph. we can see that computing the gradient requires as most o ( n2 ) operations because the forward propagation stage will at worst execute all n nodes in the original graph ( depending on which values we want to compute, we may not need to execute the entire graph


can see that computing the gradient requires as most o ( n2 ) operations because the forward propagation stage will at worst execute all n nodes in the original graph ( depending on which values we want to compute, we may not need to execute the entire graph ). the back - propagation algorithm adds one jacobian - vector product, which should be expressed with o ( 1 ) nodes, per edge in the original graph. because the computational graph is a directed acyclic graph it has at most o ( n2 ) edges. for the kinds of graphs that are commonly used in practice, the situation is even better. most neural network cost functions are 217


chapter 6. deep feedforward networks algorithm 6. 6 the inner loop subroutine build grad _ ( v,, g g, grad table _ ) of the back - propagation algorithm, called by the back - propagation algorithm deﬁned in algorithm. 6. 5 require : v, the variable whose gradient should be added to and. g grad _ table require : g, the graph to modify. require : g, the restriction of to nodes that participate in the gradient. g require : grad _ table, a data structure mapping nodes to their gradients if then v is in grad _ table return _ grad table [ ] v end if i ←1 for c v in _ get consumers (, g ) do op get operation ← _ ( ) c d c ←build grad _ (,, g g, grad table _ ) g ( ) i ← g op bprop get inputs. ( _ ( c, ) ),, v d i i ← + 1 end for g ← ig ( ) i grad table _ [ ] = v g insert and the operations creating it into g g return g roughly chain - structured, causing back - propagation to have o ( n ) cost. this is far


1 end for g ← ig ( ) i grad table _ [ ] = v g insert and the operations creating it into g g return g roughly chain - structured, causing back - propagation to have o ( n ) cost. this is far better than the naive approach, which might need to execute exponentially many nodes. this potentially exponential cost can be seen by expanding and rewriting the recursive chain rule ( equation ) non - recursively : 6. 49 ∂u ( ) n ∂u ( ) j = path ( u ( π1 ), u ( π2 ),..., u ( πt ) ), from π1 = to j πt = n t k = 2 ∂u ( πk ) ∂u ( πk−1 ). ( 6. 55 ) since the number of paths from node j to node n can grow exponentially in the length of these paths, the number of terms in the above sum, which is the number of such paths, can grow exponentially with the depth of the forward propagation graph. this large cost would be incurred because the same computation for ∂u ( ) i ∂u ( ) j would be redone many times. to avoid such recomputation, we can think of


exponentially with the depth of the forward propagation graph. this large cost would be incurred because the same computation for ∂u ( ) i ∂u ( ) j would be redone many times. to avoid such recomputation, we can think of back - propagation as a table - ﬁlling algorithm that takes advantage of storing intermediate results ∂u ( ) n ∂u ( ) i. each node in the graph has a corresponding slot in a table to store the gradient for that node. by ﬁlling in these table entries in order, 218


chapter 6. deep feedforward networks back - propagation avoids repeating many common subexpressions. this table - ﬁlling strategy is sometimes called. dynamic programming 6. 5. 7 example : back - propagation for mlp training as an example, we walk through the back - propagation algorithm as it is used to train a multilayer perceptron. here we develop a very simple multilayer perception with a single hidden layer. to train this model, we will use minibatch stochastic gradient descent. the back - propagation algorithm is used to compute the gradient of the cost on a single minibatch. speciﬁcally, we use a minibatch of examples from the training set formatted as a design matrix x and a vector of associated class labels y. the network computes a layer of hidden features h = max { 0, xw ( 1 ) }. to simplify the presentation we do not use biases in this model. we assume that our graph language includes a relu operation that can compute max { 0, z } element - wise. the predictions of the unnormalized log probabilities over classes are then given by hw ( 2 ). we assume that our graph language includes a cross _ entropy operation


operation that can compute max { 0, z } element - wise. the predictions of the unnormalized log probabilities over classes are then given by hw ( 2 ). we assume that our graph language includes a cross _ entropy operation that computes the cross - entropy between the targets y and the probability distribution deﬁned by these unnormalized log probabilities. the resulting cross - entropy deﬁnes the cost jmle. minimizing this cross - entropy performs maximum likelihood estimation of the classiﬁer. however, to make this example more realistic, we also include a regularization term. the total cost j j = mle + λ i, j w ( 1 ) i, j 2 + i, j w ( 2 ) i, j 2 ( 6. 56 ) consists of the cross - entropy and a weight decay term with [UNK] λ. the computational graph is illustrated in ﬁgure. 6. 11 the computational graph for the gradient of this example is large enough that it would be tedious to draw or to read. this demonstrates one of the beneﬁts of the back - propagation algorithm, which is that it can automatically generate gradients that would be straightforward but tedious for a software engineer to derive


that it would be tedious to draw or to read. this demonstrates one of the beneﬁts of the back - propagation algorithm, which is that it can automatically generate gradients that would be straightforward but tedious for a software engineer to derive manually. we can roughly trace out the behavior of the back - propagation algorithm by looking at the forward propagation graph in ﬁgure. to train, we wish 6. 11 to compute both ∇w ( 1 ) j and ∇w ( 2 ) j. there are two [UNK] paths leading backward from j to the weights : one through the cross - entropy cost, and one through the weight decay cost. the weight decay cost is relatively simple ; it will always contribute 2λw ( ) i to the gradient on w ( ) i. 219


chapter 6. deep feedforward networks x w ( 1 ) w ( 1 ) u ( 1 ) u ( 1 ) matmul h relu u ( 3 ) u ( 3 ) sqr u ( 4 ) u ( 4 ) sum λ u ( 7 ) u ( 7 ) w ( 2 ) w ( 2 ) u ( 2 ) u ( 2 ) matmul y j mle j mle cross _ entropy u ( 5 ) u ( 5 ) sqr u ( 6 ) u ( 6 ) sum u ( 8 ) u ( 8 ) j + × + figure 6. 11 : the computational graph used to compute the cost used to train our example of a single - layer mlp using the cross - entropy loss and weight decay. the other path through the cross - entropy cost is slightly more complicated. let g be the gradient on the unnormalized log probabilities u ( 2 ) provided by the cross _ entropy operation. the back - propagation algorithm now needs to explore two [UNK] branches. on the shorter branch, it adds h g to the gradient on w ( 2 ), using the back - propagation rule for the second argument to the matrix multiplication operation. the other branch corresponds to the longer chain descending further along the network.


branches. on the shorter branch, it adds h g to the gradient on w ( 2 ), using the back - propagation rule for the second argument to the matrix multiplication operation. the other branch corresponds to the longer chain descending further along the network. first, the back - propagation algorithm computes ∇hj = gw ( 2 ) using the back - propagation rule for the ﬁrst argument to the matrix multiplication operation. next, the relu operation uses its back - propagation rule to zero out components of the gradient corresponding to entries of u ( 1 ) that were less than. let the result be called 0 g. the last step of the back - propagation algorithm is to use the back - propagation rule for the second argument of the operation to add matmul xgto the gradient on w ( 1 ). after these gradients have been computed, it is the responsibility of the gradient descent algorithm, or another optimization algorithm, to use these gradients to update the parameters. for the mlp, the computational cost is dominated by the cost of matrix multiplication. during the forward propagation stage, we multiply by each weight 220


chapter 6. deep feedforward networks matrix, resulting in o ( w ) multiply - adds, where w is the number of weights. during the backward propagation stage, we multiply by the transpose of each weight matrix, which has the same computational cost. the main memory cost of the algorithm is that we need to store the input to the nonlinearity of the hidden layer. this value is stored from the time it is computed until the backward pass has returned to the same point. the memory cost is thus o ( mnh ), where m is the number of examples in the minibatch and nh is the number of hidden units. 6. 5. 8 complications our description of the back - propagation algorithm here is simpler than the imple - mentations actually used in practice. as noted above, we have restricted the deﬁnition of an operation to be a function that returns a single tensor. most software implementations need to support operations that can return more than one tensor. for example, if we wish to compute both the maximum value in a tensor and the index of that value, it is best to compute both in a single pass through memory, so it is most [UNK] to implement this procedure as a single operation with two outputs. we have not


wish to compute both the maximum value in a tensor and the index of that value, it is best to compute both in a single pass through memory, so it is most [UNK] to implement this procedure as a single operation with two outputs. we have not described how to control the memory consumption of back - propagation. back - propagation often involves summation of many tensors together. in the naive approach, each of these tensors would be computed separately, then all of them would be added in a second step. the naive approach has an overly high memory bottleneck that can be avoided by maintaining a single [UNK] and adding each value to that [UNK] as it is computed. real - world implementations of back - propagation also need to handle various data types, such as 32 - bit ﬂoating point, 64 - bit ﬂoating point, and integer values. the policy for handling each of these types takes special care to design. some operations have undeﬁned gradients, and it is important to track these cases and determine whether the gradient requested by the user is undeﬁned. various other technicalities make real - world [UNK] more complicated. these technicalities are not insurmountable, and this chapter has described the key intellectual tools needed to compute derivatives, but it


the gradient requested by the user is undeﬁned. various other technicalities make real - world [UNK] more complicated. these technicalities are not insurmountable, and this chapter has described the key intellectual tools needed to compute derivatives, but it is important to be aware that many more subtleties exist. 6. 5. 9 [UNK] outside the deep learning community the deep learning community has been somewhat isolated from the broader computer science community and has largely developed its own cultural attitudes 221


chapter 6. deep feedforward networks concerning how to perform [UNK]. more generally, the ﬁeld of automatic [UNK] is concerned with how to compute derivatives algorithmically. the back - propagation algorithm described here is only one approach to automatic [UNK]. it is a special case of a broader class of techniques called reverse mode accumulation. other approaches evaluate the subexpressions of the chain rule in [UNK] orders. in general, determining the order of evaluation that results in the lowest computational cost is a [UNK] problem. finding the optimal sequence of operations to compute the gradient is np - complete (, ), naumann 2008 in the sense that it may require simplifying algebraic expressions into their least expensive form. for example, suppose we have variables p1, p2,..., pn representing probabilities and variables z1, z2,..., zn representing unnormalized log probabilities. suppose we deﬁne qi = exp ( zi ) i exp ( zi ), ( 6. 57 ) where we build the softmax function out of exponentiation, summation and division operations, and construct a cross - entropy loss j = − i pilog qi. a human mathematician can observe that the derivative of


), ( 6. 57 ) where we build the softmax function out of exponentiation, summation and division operations, and construct a cross - entropy loss j = − i pilog qi. a human mathematician can observe that the derivative of j with respect to zi takes a very simple form : qi −pi. the back - propagation algorithm is not capable of simplifying the gradient this way, and will instead explicitly propagate gradients through all of the logarithm and exponentiation operations in the original graph. some software libraries such as theano (, ;, ) are able to bergstra et al. 2010 bastien et al. 2012 perform some kinds of algebraic substitution to improve over the graph proposed by the pure back - propagation algorithm. when the forward graph g has a single output node and each partial derivative ∂u ( ) i ∂u ( ) j can be computed with a constant amount of computation, back - propagation guarantees that the number of computations for the gradient computation is of the same order as the number of computations for the forward computation : this can be seen in algorithm because each local partial derivative 6. 2 ∂u ( ) i ∂u ( ) j needs to be computed only once along with an associated


is of the same order as the number of computations for the forward computation : this can be seen in algorithm because each local partial derivative 6. 2 ∂u ( ) i ∂u ( ) j needs to be computed only once along with an associated multiplication and addition for the recursive chain - rule formulation ( equation ). the overall computation is 6. 49 therefore o ( # edges ). however, it can potentially be reduced by simplifying the computational graph constructed by back - propagation, and this is an np - complete task. implementations such as theano and tensorflow use heuristics based on matching known simpliﬁcation patterns in order to iteratively attempt to simplify the graph. we deﬁned back - propagation only for the computation of a gradient of a scalar output but back - propagation can be extended to compute a jacobian ( either of k [UNK] scalar nodes in the graph, or of a tensor - valued node containing k values ). a naive implementation may then need k times more computation : for 222


chapter 6. deep feedforward networks each scalar internal node in the original forward graph, the naive implementation computes k gradients instead of a single gradient. when the number of outputs of the graph is larger than the number of inputs, it is sometimes preferable to use another form of automatic [UNK] called forward mode accumulation. forward mode computation has been proposed for obtaining real - time computation of gradients in recurrent networks, for example (, ). this williams and zipser 1989 also avoids the need to store the values and gradients for the whole graph, trading [UNK] [UNK] for memory. the relationship between forward mode and backward mode is analogous to the relationship between left - multiplying versus right - multiplying a sequence of matrices, such as abcd, ( 6. 58 ) where the matrices can be thought of as jacobian matrices. for example, if d is a column vector while a has many rows, this corresponds to a graph with a single output and many inputs, and starting the multiplications from the end and going backwards only requires matrix - vector products. this corresponds to the backward mode. instead, starting to multiply from the left would involve a series of matrix - matrix products, which makes the whole computation much more expensive. however, if


the end and going backwards only requires matrix - vector products. this corresponds to the backward mode. instead, starting to multiply from the left would involve a series of matrix - matrix products, which makes the whole computation much more expensive. however, if a has fewer rows than d has columns, it is cheaper to run the multiplications left - to - right, corresponding to the forward mode. in many communities outside of machine learning, it is more common to im - plement [UNK] software that acts directly on traditional programming language code, such as python or c code, and automatically generates programs that [UNK] functions written in these languages. in the deep learning com - munity, computational graphs are usually represented by explicit data structures created by specialized libraries. the specialized approach has the drawback of requiring the library developer to deﬁne the bprop methods for every operation and limiting the user of the library to only those operations that have been deﬁned. however, the specialized approach also has the beneﬁt of allowing customized back - propagation rules to be developed for each operation, allowing the developer to improve speed or stability in non - obvious ways that an automatic procedure would presumably be unable to replicate. back - propagation is therefore not the only way or the optimal way of computing the


- propagation rules to be developed for each operation, allowing the developer to improve speed or stability in non - obvious ways that an automatic procedure would presumably be unable to replicate. back - propagation is therefore not the only way or the optimal way of computing the gradient, but it is a very practical method that continues to serve the deep learning community very well. in the future, [UNK] technology for deep networks may improve as deep learning practitioners become more aware of advances in the broader ﬁeld of automatic [UNK]. 223


chapter 6. deep feedforward networks 6. 5. 10 higher - order derivatives some software frameworks support the use of higher - order derivatives. among the deep learning software frameworks, this includes at least theano and tensorflow. these libraries use the same kind of data structure to describe the expressions for derivatives as they use to describe the original function being [UNK]. this means that the symbolic [UNK] machinery can be applied to derivatives. in the context of deep learning, it is rare to compute a single second derivative of a scalar function. instead, we are usually interested in properties of the hessian matrix. if we have a function f : rn →r, then the hessian matrix is of size n n ×. in typical deep learning applications, n will be the number of parameters in the model, which could easily number in the billions. the entire hessian matrix is thus infeasible to even represent. instead of explicitly computing the hessian, the typical deep learning approach is to use krylov methods. krylov methods are a set of iterative techniques for performing various operations like approximately inverting a matrix or ﬁnding approximations to its eigenvectors or eigenvalues, without using any operation other than matrix - vector products


. krylov methods are a set of iterative techniques for performing various operations like approximately inverting a matrix or ﬁnding approximations to its eigenvectors or eigenvalues, without using any operation other than matrix - vector products. in order to use krylov methods on the hessian, we only need to be able to compute the product between the hessian matrix h and an arbitrary vector v. a straightforward technique (, ) for doing so is to compute christianson 1992 hv = ∇x ( ∇xf x ( ) ) v. ( 6. 59 ) both of the gradient computations in this expression may be computed automati - cally by the appropriate software library. note that the outer gradient expression takes the gradient of a function of the inner gradient expression. if v is itself a vector produced by a computational graph, it is important to specify that the automatic [UNK] software should not [UNK] through the graph that produced. v while computing the hessian is usually not advisable, it is possible to do with hessian vector products. one simply computes he ( ) i for all i = 1,..., n, where e ( ) i is the one - hot vector with e ( ) i i =


, it is possible to do with hessian vector products. one simply computes he ( ) i for all i = 1,..., n, where e ( ) i is the one - hot vector with e ( ) i i = 1 and all other entries equal to 0. 6. 6 historical notes feedforward networks can be seen as [UNK] nonlinear function approximators based on using gradient descent to minimize the error in a function approximation. 224


chapter 6. deep feedforward networks from this point of view, the modern feedforward network is the culmination of centuries of progress on the general function approximation task. the chain rule that underlies the back - propagation algorithm was invented in the 17th century (, ;, ). calculus and algebra have leibniz 1676 l ’ hopital 1696 long been used to solve optimization problems in closed form, but gradient descent was not introduced as a technique for iteratively approximating the solution to optimization problems until the 19th century ( cauchy 1847, ). beginning in the 1940s, these function approximation techniques were used to motivate machine learning models such as the perceptron. however, the earliest models were based on linear models. critics including marvin minsky pointed out several of the ﬂaws of the linear model family, such as its inability to learn the xor function, which led to a backlash against the entire neural network approach. learning nonlinear functions required the development of a multilayer per - ceptron and a means of computing the gradient through such a model. [UNK] applications of the chain rule based on dynamic programming began to appear in the 1960s and 1970s, mostly for control applications (, ; kelley 1960 bryson and denham 1961


per - ceptron and a means of computing the gradient through such a model. [UNK] applications of the chain rule based on dynamic programming began to appear in the 1960s and 1970s, mostly for control applications (, ; kelley 1960 bryson and denham 1961 dreyfus 1962 bryson and ho 1969 dreyfus 1973, ;, ;, ;, ) but also for sensitivity analysis (, ). linnainmaa 1976 werbos 1981 ( ) proposed applying these techniques to training artiﬁcial neural networks. the idea was ﬁnally developed in practice after being independently rediscovered in [UNK] ways (, ; lecun 1985 parker 1985 rumelhart 1986a, ; et al., ). the book parallel distributed pro - cessing presented the results of some of the ﬁrst successful experiments with back - propagation in a chapter (, ) that contributed greatly rumelhart et al. 1986b to the popularization of back - propagation and initiated a very active period of research in multi - layer neural networks. however, the ideas put forward by the authors of that book and in particular by rumelhart and hinton go much beyond back - propagation. they include crucial ideas about the possible computational implementation of several central aspects of cognition and learning, which came under the name


the ideas put forward by the authors of that book and in particular by rumelhart and hinton go much beyond back - propagation. they include crucial ideas about the possible computational implementation of several central aspects of cognition and learning, which came under the name of “ connectionism ” because of the importance this school of thought places on the connections between neurons as the locus of learning and memory. in particular, these ideas include the notion of distributed representation ( hinton et al., ). 1986 following the success of back - propagation, neural network research gained pop - ularity and reached a peak in the early 1990s. afterwards, other machine learning techniques became more popular until the modern deep learning renaissance that began in 2006. the core ideas behind modern feedforward networks have not changed sub - stantially since the 1980s. the same back - propagation algorithm and the same 225


chapter 6. deep feedforward networks approaches to gradient descent are still in use. most of the improvement in neural network performance from 1986 to 2015 can be attributed to two factors. first, larger datasets have reduced the degree to which statistical generalization is a challenge for neural networks. second, neural networks have become much larger, due to more powerful computers, and better software infrastructure. however, a small number of algorithmic changes have improved the performance of neural networks noticeably. one of these algorithmic changes was the replacement of mean squared error with the cross - entropy family of loss functions. mean squared error was popular in the 1980s and 1990s, but was gradually replaced by cross - entropy losses and the principle of maximum likelihood as ideas spread between the statistics community and the machine learning community. the use of cross - entropy losses greatly improved the performance of models with sigmoid and softmax outputs, which had previously [UNK] from saturation and slow learning when using the mean squared error loss. the other major algorithmic change that has greatly improved the performance of feedforward networks was the replacement of sigmoid hidden units with piecewise linear hidden units, such as rectiﬁed linear units. rectiﬁcation using the max { 0, z } function was introduced in


greatly improved the performance of feedforward networks was the replacement of sigmoid hidden units with piecewise linear hidden units, such as rectiﬁed linear units. rectiﬁcation using the max { 0, z } function was introduced in early neural network models and dates back at least as far as the cognitron and neocognitron ( fukushima 1975 1980,, ). these early models did not use rectiﬁed linear units, but instead applied rectiﬁcation to nonlinear functions. despite the early popularity of rectiﬁcation, rectiﬁcation was largely replaced by sigmoids in the 1980s, perhaps because sigmoids perform better when neural networks are very small. as of the early 2000s, rectiﬁed linear units were avoided due to a somewhat superstitious belief that activation functions with non - [UNK] points must be avoided. this began to change in about 2009. jarrett 2009 et al. ( ) observed that “ using a rectifying nonlinearity is the single most important factor in improving the performance of a recognition system ” among several [UNK] factors of neural network architecture design. for small datasets, ( ) observed that using rectifying non - jarrett et al. 2009 linearities is even more important than learning


important factor in improving the performance of a recognition system ” among several [UNK] factors of neural network architecture design. for small datasets, ( ) observed that using rectifying non - jarrett et al. 2009 linearities is even more important than learning the weights of the hidden layers. random weights are [UNK] to propagate useful information through a rectiﬁed linear network, allowing the classiﬁer layer at the top to learn how to map [UNK] feature vectors to class identities. when more data is available, learning begins to extract enough useful knowledge to exceed the performance of randomly chosen parameters. ( ) glorot et al. 2011a showed that learning is far easier in deep rectiﬁed linear networks than in deep networks that have curvature or two - sided saturation in their activation functions. 226


chapter 6. deep feedforward networks rectiﬁed linear units are also of historical interest because they show that neuroscience has continued to have an inﬂuence on the development of deep learning algorithms. ( ) motivate rectiﬁed linear units from glorot et al. 2011a biological considerations. the half - rectifying nonlinearity was intended to capture these properties of biological neurons : 1 ) for some inputs, biological neurons are completely inactive. 2 ) for some inputs, a biological neuron ’ s output is proportional to its input. 3 ) most of the time, biological neurons operate in the regime where they are inactive ( i. e., they should have sparse activations ). when the modern resurgence of deep learning began in 2006, feedforward networks continued to have a bad reputation. from about 2006 - 2012, it was widely believed that feedforward networks would not perform well unless they were assisted by other models, such as probabilistic models. today, it is now known that with the right resources and engineering practices, feedforward networks perform very well. today, gradient - based learning in feedforward networks is used as a tool to develop probabilistic models, such as the variational autoencode


known that with the right resources and engineering practices, feedforward networks perform very well. today, gradient - based learning in feedforward networks is used as a tool to develop probabilistic models, such as the variational autoencoder and generative adversarial networks, described in chapter. rather than being viewed as an unreliable 20 technology that must be supported by other techniques, gradient - based learning in feedforward networks has been viewed since 2012 as a powerful technology that may be applied to many other machine learning tasks. in 2006, the community used unsupervised learning to support supervised learning, and now, ironically, it is more common to use supervised learning to support unsupervised learning. feedforward networks continue to have unfulﬁlled potential. in the future, we expect they will be applied to many more tasks, and that advances in optimization algorithms and model design will improve their performance even further. this chapter has primarily described the neural network family of models. in the subsequent chapters, we turn to how to use these models — how to regularize and train them. 227


chapter 7 regularization for deep learning a central problem in machine learning is how to make an algorithm that will perform well not just on the training data, but also on new inputs. many strategies used in machine learning are explicitly designed to reduce the test error, possibly at the expense of increased training error. these strategies are known collectively as regularization. as we will see there are a great many forms of regularization available to the deep learning practitioner. in fact, developing more [UNK] regularization strategies has been one of the major research [UNK] in the ﬁeld. chapter introduced the basic concepts of generalization, underﬁtting, overﬁt - 5 ting, bias, variance and regularization. if you are not already familiar with these notions, please refer to that chapter before continuing with this one. in this chapter, we describe regularization in more detail, focusing on regular - ization strategies for deep models or models that may be used as building blocks to form deep models. some sections of this chapter deal with standard concepts in machine learning. if you are already familiar with these concepts, feel free to skip the relevant sections. however, most of this chapter is concerned with the extension of these basic concepts to the particular case of neural networks. in section, we deﬁned regular


learning. if you are already familiar with these concepts, feel free to skip the relevant sections. however, most of this chapter is concerned with the extension of these basic concepts to the particular case of neural networks. in section, we deﬁned regularization as “ any modiﬁcation we make to 5. 2. 2 a learning algorithm that is intended to reduce its generalization error but not its training error. ” there are many regularization strategies. some put extra constraints on a machine learning model, such as adding restrictions on the parameter values. some add extra terms in the objective function that can be thought of as corresponding to a soft constraint on the parameter values. if chosen carefully, these extra constraints and penalties can lead to improved performance 228


chapter 7. regularization for deep learning on the test set. sometimes these constraints and penalties are designed to encode speciﬁc kinds of prior knowledge. other times, these constraints and penalties are designed to express a generic preference for a simpler model class in order to promote generalization. sometimes penalties and constraints are necessary to make an underdetermined problem determined. other forms of regularization, known as ensemble methods, combine multiple hypotheses that explain the training data. in the context of deep learning, most regularization strategies are based on regularizing estimators. regularization of an estimator works by trading increased bias for reduced variance. an [UNK] regularizer is one that makes a proﬁtable trade, reducing variance signiﬁcantly while not overly increasing the bias. when we discussed generalization and overﬁtting in chapter, we focused on three situations, 5 where the model family being trained either ( 1 ) excluded the true data generating process — corresponding to underﬁtting and inducing bias, or ( 2 ) matched the true data generating process, or ( 3 ) included the generating process but also many other possible generating processes — the overﬁtting regime where variance rather than bias dominates the estimation error. the goal of regularization is to take


or ( 2 ) matched the true data generating process, or ( 3 ) included the generating process but also many other possible generating processes — the overﬁtting regime where variance rather than bias dominates the estimation error. the goal of regularization is to take a model from the third regime into the second regime. in practice, an overly complex model family does not necessarily include the target function or the true data generating process, or even a close approximation of either. we almost never have access to the true data generating process so we can never know for sure if the model family being estimated includes the generating process or not. however, most applications of deep learning algorithms are to domains where the true data generating process is almost certainly outside the model family. deep learning algorithms are typically applied to extremely complicated domains such as images, audio sequences and text, for which the true generation process essentially involves simulating the entire universe. to some extent, we are always trying to ﬁt a square peg ( the data generating process ) into a round hole ( our model family ). what this means is that controlling the complexity of the model is not a simple matter of ﬁnding the model of the right size, with the right number of parameters. instead, we might ﬁnd — and indeed in practical deep learning


model family ). what this means is that controlling the complexity of the model is not a simple matter of ﬁnding the model of the right size, with the right number of parameters. instead, we might ﬁnd — and indeed in practical deep learning scenarios, we almost always do ﬁnd — that the best ﬁtting model ( in the sense of minimizing generalization error ) is a large model that has been regularized appropriately. we now review several strategies for how to create such a large, deep, regularized model. 229


chapter 7. regularization for deep learning 7. 1 parameter norm penalties regularization has been used for decades prior to the advent of deep learning. linear models such as linear regression and logistic regression allow simple, straightforward, and [UNK] regularization strategies. many regularization approaches are based on limiting the capacity of models, such as neural networks, linear regression, or logistic regression, by adding a pa - rameter norm penalty ω ( θ ) to the objective function j. we denote the regularized objective function by [UNK] : [UNK] j, j, α ( ; θ x y ) = ( ; θ x y ) + ω ( ) θ ( 7. 1 ) where α ∈ [ 0, ∞ ) is a hyperparameter that weights the relative contribution of the norm penalty term,, relative to the standard objective function ω j. setting α to 0 results in no regularization. larger values of α correspond to more regularization. when our training algorithm minimizes the regularized objective function [UNK] it will decrease both the original objective j on the training data and some measure of the size of the parameters θ ( or some subset of the parameters ). [UNK] choices for the parameter norm can result in [UNK] solutions being preferred. ω in this section, we discuss the [UNK] of the


objective j on the training data and some measure of the size of the parameters θ ( or some subset of the parameters ). [UNK] choices for the parameter norm can result in [UNK] solutions being preferred. ω in this section, we discuss the [UNK] of the various norms when used as penalties on the model parameters. before delving into the regularization behavior of [UNK] norms, we note that for neural networks, we typically choose to use a parameter norm penalty that ω penalizes of the [UNK] transformation at each layer and leaves only the weights the biases unregularized. the biases typically require less data to ﬁt accurately than the weights. each weight speciﬁes how two variables interact. fitting the weight well requires observing both variables in a variety of conditions. each bias controls only a single variable. this means that we do not induce too much variance by leaving the biases unregularized. also, regularizing the bias parameters can introduce a signiﬁcant amount of underﬁtting. we therefore use the vector w to indicate all of the weights that should be [UNK] by a norm penalty, while the vector θ denotes all of the parameters, including both w and the unregularized parameters. in the context of neural networks, it is