chapter 5. machine learning basics hand - designed, it can usually be understood as using a special - case optimizer. some models such as decision trees or k - means require special - case optimizers because their cost functions have ﬂat regions that make them inappropriate for minimization by gradient - based optimizers. recognizing that most machine learning algorithms can be described using this recipe helps to see the [UNK] algorithms as part of a taxonomy of methods for doing related tasks that work for similar reasons, rather than as a long list of algorithms that each have separate justiﬁcations. 5. 11 challenges motivating deep learning the simple machine learning algorithms described in this chapter work very well on a wide variety of important problems. however, they have not succeeded in solving the central problems in ai, such as recognizing speech or recognizing objects. the development of deep learning was motivated in part by the failure of traditional algorithms to generalize well on such ai tasks. this section is about how the challenge of generalizing to new examples becomes exponentially more [UNK] when working with high - dimensional data, and how the mechanisms used to achieve generalization in traditional machine learning are [UNK] to learn complicated functions in high - dimensional spaces. such spaces also often impose high computational costs. deep learning


examples becomes exponentially more [UNK] when working with high - dimensional data, and how the mechanisms used to achieve generalization in traditional machine learning are [UNK] to learn complicated functions in high - dimensional spaces. such spaces also often impose high computational costs. deep learning was designed to overcome these and other obstacles. 5. 11. 1 the curse of dimensionality many machine learning problems become exceedingly [UNK] when the number of dimensions in the data is high. this phenomenon is known as the curse of dimensionality. of particular concern is that the number of possible distinct conﬁgurations of a set of variables increases exponentially as the number of variables increases. 155

/home/ricoiban/GEMMA/mnlp_chatsplaining/RAG/Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville (z-lib.org).pdf


chapter 5. machine learning basics figure 5. 9 : as the number of relevant dimensions of the data increases ( from left to right ), the number of conﬁgurations of interest may grow exponentially. ( left ) in this one - dimensional example, we have one variable for which we only care to distinguish 10 regions of interest. with enough examples falling within each of these regions ( each region corresponds to a cell in the illustration ), learning algorithms can easily generalize correctly. a straightforward way to generalize is to estimate the value of the target function within each region ( and possibly interpolate between neighboring regions ). with 2 ( center ) dimensions it is more [UNK] to distinguish 10 [UNK] values of each variable. we need to keep track of up to 10×10 = 100 regions, and we need at least that many examples to cover all those regions. with 3 dimensions this grows to ( right ) 103 = 1000 regions and at least that many examples. for d dimensions and v values to be distinguished along each axis, we seem to need o ( v d ) regions and examples. this is an instance of the curse of dimensionality. figure graciously provided by nicolas chapados. the curse of dimensionality arises in many places in computer science, and


, we seem to need o ( v d ) regions and examples. this is an instance of the curse of dimensionality. figure graciously provided by nicolas chapados. the curse of dimensionality arises in many places in computer science, and especially so in machine learning. one challenge posed by the curse of dimensionality is a statistical challenge. as illustrated in ﬁgure, a statistical challenge arises because the number of 5. 9 possible conﬁgurations of x is much larger than the number of training examples. to understand the issue, let us consider that the input space is organized into a grid, like in the ﬁgure. we can describe low - dimensional space with a low number of grid cells that are mostly occupied by the data. when generalizing to a new data point, we can usually tell what to do simply by inspecting the training examples that lie in the same cell as the new input. for example, if estimating the probability density at some point x, we can just return the number of training examples in the same unit volume cell as x, divided by the total number of training examples. if we wish to classify an example, we can return the most common class of training examples in the same cell. if we are doing regression we can


training examples in the same unit volume cell as x, divided by the total number of training examples. if we wish to classify an example, we can return the most common class of training examples in the same cell. if we are doing regression we can average the target values observed over the examples in that cell. but what about the cells for which we have seen no example? because in high - dimensional spaces the number of conﬁgurations is huge, much larger than our number of examples, a typical grid cell has no training example associated with it. how could we possibly say something 156


chapter 5. machine learning basics meaningful about these new conﬁgurations? many traditional machine learning algorithms simply assume that the output at a new point should be approximately the same as the output at the nearest training point. 5. 11. 2 local constancy and smoothness regularization in order to generalize well, machine learning algorithms need to be guided by prior beliefs about what kind of function they should learn. previously, we have seen these priors incorporated as explicit beliefs in the form of probability distributions over parameters of the model. more informally, we may also discuss prior beliefs as directly inﬂuencing the itself and only indirectly acting on the parameters function via their [UNK] on the function. additionally, we informally discuss prior beliefs as being expressed implicitly, by choosing algorithms that are biased toward choosing some class of functions over another, even though these biases may not be expressed ( or even possible to express ) in terms of a probability distribution representing our degree of belief in various functions. among the most widely used of these implicit “ priors ” is the smoothness prior or local constancy prior. this prior states that the function we learn should not change very much within a small region. many simpler algorithms rely exclusively on this prior to generalize well, and as a


“ priors ” is the smoothness prior or local constancy prior. this prior states that the function we learn should not change very much within a small region. many simpler algorithms rely exclusively on this prior to generalize well, and as a result they fail to scale to the statistical challenges involved in solving ai - level tasks. throughout this book, we will describe how deep learning introduces additional ( explicit and implicit ) priors in order to reduce the generalization error on sophisticated tasks. here, we explain why the smoothness prior alone is [UNK] for these tasks. there are many [UNK] ways to implicitly or explicitly express a prior belief that the learned function should be smooth or locally constant. all of these [UNK] methods are designed to encourage the learning process to learn a function f∗that satisﬁes the condition f∗ ( ) x ≈f ∗ ( + ) x ( 5. 103 ) for most conﬁgurations x and small change. in other words, if we know a good answer for an input x ( for example, if x is a labeled training example ) then that answer is probably good in the neighborhood of x. if we have several good answers in some neighborhood we would combine them ( by some form of averaging or interpolation ) to


x ( for example, if x is a labeled training example ) then that answer is probably good in the neighborhood of x. if we have several good answers in some neighborhood we would combine them ( by some form of averaging or interpolation ) to produce an answer that agrees with as many of them as much as possible. an extreme example of the local constancy approach is the k - nearest neighbors family of learning algorithms. these predictors are literally constant over each 157


chapter 5. machine learning basics region containing all the points x that have the same set of k nearest neighbors in the training set. for k = 1, the number of distinguishable regions cannot be more than the number of training examples. while the k - nearest neighbors algorithm copies the output from nearby training examples, most kernel machines interpolate between training set outputs associated with nearby training examples. an important class of kernels is the family oflocal kernels where k ( u v, ) is large when u = v and decreases as u and v grow farther apart from each other. a local kernel can be thought of as a similarity function that performs template matching, by measuring how closely a test example x resembles each training example x ( ) i. much of the modern motivation for deep learning is derived from studying the limitations of local template matching and how deep models are able to succeed in cases where local template matching fails (, ). bengio et al. 2006b decision trees also [UNK] from the limitations of exclusively smoothness - based learning because they break the input space into as many regions as there are leaves and use a separate parameter ( or sometimes many parameters for extensions of decision trees ) in each region. if the target function requires a tree with at least n leaves to


- based learning because they break the input space into as many regions as there are leaves and use a separate parameter ( or sometimes many parameters for extensions of decision trees ) in each region. if the target function requires a tree with at least n leaves to be represented accurately, then at least n training examples are required to ﬁt the tree. a multiple of n is needed to achieve some level of statistical conﬁdence in the predicted output. in general, to distinguish o ( k ) regions in input space, all of these methods require o ( k ) examples. typically there are o ( k ) parameters, with o ( 1 ) parameters associated with each of the o ( k ) regions. the case of a nearest neighbor scenario, where each training example can be used to deﬁne at most one region, is illustrated in ﬁgure. 5. 10 is there a way to represent a complex function that has many more regions to be distinguished than the number of training examples? clearly, assuming only smoothness of the underlying function will not allow a learner to do that. for example, imagine that the target function is a kind of checkerboard. a checkerboard contains many variations but there is a simple structure to them. imagine what happens when the number


underlying function will not allow a learner to do that. for example, imagine that the target function is a kind of checkerboard. a checkerboard contains many variations but there is a simple structure to them. imagine what happens when the number of training examples is substantially smaller than the number of black and white squares on the checkerboard. based on only local generalization and the smoothness or local constancy prior, we would be guaranteed to correctly guess the color of a new point if it lies within the same checkerboard square as a training example. there is no guarantee that the learner could correctly extend the checkerboard pattern to points lying in squares that do not contain training examples. with this prior alone, the only information that an example tells us is the color of its square, and the only way to get the colors of the 158


chapter 5. machine learning basics figure 5. 10 : illustration of how the nearest neighbor algorithm breaks up the input space into regions. an example ( represented here by a circle ) within each region deﬁnes the region boundary ( represented here by the lines ). they value associated with each example deﬁnes what the output should be for all points within the corresponding region. the regions deﬁned by nearest neighbor matching form a geometric pattern called a voronoi diagram. the number of these contiguous regions cannot grow faster than the number of training examples. while this ﬁgure illustrates the behavior of the nearest neighbor algorithm speciﬁcally, other machine learning algorithms that rely exclusively on the local smoothness prior for generalization exhibit similar behaviors : each training example only informs the learner about how to generalize in some neighborhood immediately surrounding that example. 159


chapter 5. machine learning basics entire checkerboard right is to cover each of its cells with at least one example. the smoothness assumption and the associated non - parametric learning algo - rithms work extremely well so long as there are enough examples for the learning algorithm to observe high points on most peaks and low points on most valleys of the true underlying function to be learned. this is generally true when the function to be learned is smooth enough and varies in few enough dimensions. in high dimensions, even a very smooth function can change smoothly but in a [UNK] way along each dimension. if the function additionally behaves [UNK] in [UNK] regions, it can become extremely complicated to describe with a set of training examples. if the function is complicated ( we want to distinguish a huge number of regions compared to the number of examples ), is there any hope to generalize well? the answer to both of these questions — whether it is possible to represent a complicated function [UNK], and whether it is possible for the estimated function to generalize well to new inputs — is yes. the key insight is that a very large number of regions, e. g., o ( 2k ), can be deﬁned with o ( k ) examples, so long as we introduce some


##ize well to new inputs — is yes. the key insight is that a very large number of regions, e. g., o ( 2k ), can be deﬁned with o ( k ) examples, so long as we introduce some dependencies between the regions via additional assumptions about the underlying data generating distribution. in this way, we can actually generalize non - locally (, ;, ). many bengio and monperrus 2005 bengio et al. 2006c [UNK] deep learning algorithms provide implicit or explicit assumptions that are reasonable for a broad range of ai tasks in order to capture these advantages. other approaches to machine learning often make stronger, task - speciﬁc as - sumptions. for example, we could easily solve the checkerboard task by providing the assumption that the target function is periodic. usually we do not include such strong, task - speciﬁc assumptions into neural networks so that they can generalize to a much wider variety of structures. ai tasks have structure that is much too complex to be limited to simple, manually speciﬁed properties such as periodicity, so we want learning algorithms that embody more general - purpose assumptions. the core idea in deep learning is that we assume that the data was generated by


much too complex to be limited to simple, manually speciﬁed properties such as periodicity, so we want learning algorithms that embody more general - purpose assumptions. the core idea in deep learning is that we assume that the data was generated by the composition of factors or features, potentially at multiple levels in a hierarchy. many other similarly generic assumptions can further improve deep learning al - gorithms. these apparently mild assumptions allow an exponential gain in the relationship between the number of examples and the number of regions that can be distinguished. these exponential gains are described more precisely in sections 6. 4. 1 15. 4 15. 5, and. the exponential advantages conferred by the use of deep, distributed representations counter the exponential challenges posed by the curse of dimensionality. 160


chapter 5. machine learning basics 5. 11. 3 manifold learning an important concept underlying many ideas in machine learning is that of a manifold. a manifold is a connected region. mathematically, it is a set of points, associated with a neighborhood around each point. from any given point, the manifold locally appears to be a euclidean space. in everyday life, we experience the surface of the world as a 2 - d plane, but it is in fact a spherical manifold in 3 - d space. the deﬁnition of a neighborhood surrounding each point implies the existence of transformations that can be applied to move on the manifold from one position to a neighboring one. in the example of the world ’ s surface as a manifold, one can walk north, south, east, or west. although there is a formal mathematical meaning to the term “ manifold, ” in machine learning it tends to be used more loosely to designate a connected set of points that can be approximated well by considering only a small number of degrees of freedom, or dimensions, embedded in a higher - dimensional space. each dimension corresponds to a local direction of variation. see ﬁgure for an 5. 11 example of training data lying near a one - dimensional manifold embedded in two - dimensional space. in the


or dimensions, embedded in a higher - dimensional space. each dimension corresponds to a local direction of variation. see ﬁgure for an 5. 11 example of training data lying near a one - dimensional manifold embedded in two - dimensional space. in the context of machine learning, we allow the dimensionality of the manifold to vary from one point to another. this often happens when a manifold intersects itself. for example, a ﬁgure eight is a manifold that has a single dimension in most places but two dimensions at the intersection at the center. 0 5 1 0 1 5 2 0 2 5 3 0 3 5 4 0........ −1 0. −0 5. 0 0. 0 5. 1 0. 1 5. 2 0. 2 5. figure 5. 11 : data sampled from a distribution in a two - dimensional space that is actually concentrated near a one - dimensional manifold, like a twisted string. the solid line indicates the underlying manifold that the learner should infer. 161


chapter 5. machine learning basics many machine learning problems seem hopeless if we expect the machine learning algorithm to learn functions with interesting variations across all of rn. manifold learning algorithms surmount this obstacle by assuming that most of rn consists of invalid inputs, and that interesting inputs occur only along a collection of manifolds containing a small subset of points, with interesting variations in the output of the learned function occurring only along directions that lie on the manifold, or with interesting variations happening only when we move from one manifold to another. manifold learning was introduced in the case of continuous - valued data and the unsupervised learning setting, although this probability concentration idea can be generalized to both discrete data and the supervised learning setting : the key assumption remains that probability mass is highly concentrated. the assumption that the data lies along a low - dimensional manifold may not always be correct or useful. we argue that in the context of ai tasks, such as those that involve processing images, sounds, or text, the manifold assumption is at least approximately correct. the evidence in favor of this assumption consists of two categories of observations. the ﬁrst observation in favor of the manifold hypothesis is that the proba - bility distribution over images, text strings, and sounds that occur in real life is highly concentrated. uniform


evidence in favor of this assumption consists of two categories of observations. the ﬁrst observation in favor of the manifold hypothesis is that the proba - bility distribution over images, text strings, and sounds that occur in real life is highly concentrated. uniform noise essentially never resembles structured inputs from these domains. figure shows how, instead, uniformly sampled points 5. 12 look like the patterns of static that appear on analog television sets when no signal is available. similarly, if you generate a document by picking letters uniformly at random, what is the probability that you will get a meaningful english - language text? almost zero, again, because most of the long sequences of letters do not correspond to a natural language sequence : the distribution of natural language sequences occupies a very small volume in the total space of sequences of letters. 162


chapter 5. machine learning basics figure 5. 12 : sampling images uniformly at random ( by randomly picking each pixel according to a uniform distribution ) gives rise to noisy images. although there is a non - zero probability to generate an image of a face or any other object frequently encountered in ai applications, we never actually observe this happening in practice. this suggests that the images encountered in ai applications occupy a negligible proportion of the volume of image space. of course, concentrated probability distributions are not [UNK] to show that the data lies on a reasonably small number of manifolds. we must also establish that the examples we encounter are connected to each other by other 163


chapter 5. machine learning basics examples, with each example surrounded by other highly similar examples that may be reached by applying transformations to traverse the manifold. the second argument in favor of the manifold hypothesis is that we can also imagine such neighborhoods and transformations, at least informally. in the case of images, we can certainly think of many possible transformations that allow us to trace out a manifold in image space : we can gradually dim or brighten the lights, gradually move or rotate objects in the image, gradually alter the colors on the surfaces of objects, etc. it remains likely that there are multiple manifolds involved in most applications. for example, the manifold of images of human faces may not be connected to the manifold of images of cat faces. these thought experiments supporting the manifold hypotheses convey some in - tuitive reasons supporting it. more rigorous experiments ( cayton 2005 narayanan, ; and mitter 2010 scholkopf 1998 roweis and saul 2000 tenenbaum, ; et al., ;, ; et al., 2000 brand 2003 belkin and niyogi 2003 donoho and grimes 2003 weinberger ;, ;, ;, ; and saul 2004, ) clearly support the hypothesis for a large class of datasets of interest in ai.


al., 2000 brand 2003 belkin and niyogi 2003 donoho and grimes 2003 weinberger ;, ;, ;, ; and saul 2004, ) clearly support the hypothesis for a large class of datasets of interest in ai. when the data lies on a low - dimensional manifold, it can be most natural for machine learning algorithms to represent the data in terms of coordinates on the manifold, rather than in terms of coordinates in rn. in everyday life, we can think of roads as 1 - d manifolds embedded in 3 - d space. we give directions to speciﬁc addresses in terms of address numbers along these 1 - d roads, not in terms of coordinates in 3 - d space. extracting these manifold coordinates is challenging, but holds the promise to improve many machine learning algorithms. this general principle is applied in many contexts. figure shows the manifold structure of 5. 13 a dataset consisting of faces. by the end of this book, we will have developed the methods necessary to learn such a manifold structure. in ﬁgure, we will see 20. 6 how a machine learning algorithm can successfully accomplish this goal. this concludes part, which has provided the basic concepts in mathematics i and machine learning which are employed throughout the remaining parts of


a manifold structure. in ﬁgure, we will see 20. 6 how a machine learning algorithm can successfully accomplish this goal. this concludes part, which has provided the basic concepts in mathematics i and machine learning which are employed throughout the remaining parts of the book. you are now prepared to embark upon your study of deep learning. 164


chapter 5. machine learning basics figure 5. 13 : training examples from the qmul multiview face dataset (, ) gong et al. 2000 for which the subjects were asked to move in such a way as to cover the two - dimensional manifold corresponding to two angles of rotation. we would like learning algorithms to be able to discover and disentangle such manifold coordinates. figure illustrates such a 20. 6 feat. 165


part ii deep networks : modern practices 166


this part of the book summarizes the state of modern deep learning as it is used to solve practical applications. deep learning has a long history and many aspirations. several approaches have been proposed that have yet to entirely bear fruit. several ambitious goals have yet to be realized. these less - developed branches of deep learning appear in the ﬁnal part of the book. this part focuses only on those approaches that are essentially working tech - nologies that are already used heavily in industry. modern deep learning provides a very powerful framework for supervised learning. by adding more layers and more units within a layer, a deep network can represent functions of increasing complexity. most tasks that consist of mapping an input vector to an output vector, and that are easy for a person to do rapidly, can be accomplished via deep learning, given [UNK] large models and [UNK] large datasets of labeled training examples. other tasks, that can not be described as associating one vector to another, or that are [UNK] enough that a person would require time to think and reﬂect in order to accomplish the task, remain beyond the scope of deep learning for now. this part of the book describes the core parametric function approximation technology that is behind nearly all modern practical applications of deep learning. we begin by describing


and reﬂect in order to accomplish the task, remain beyond the scope of deep learning for now. this part of the book describes the core parametric function approximation technology that is behind nearly all modern practical applications of deep learning. we begin by describing the feedforward deep network model that is used to represent these functions. next, we present advanced techniques for regularization and optimization of such models. scaling these models to large inputs such as high resolution images or long temporal sequences requires specialization. we introduce the convolutional network for scaling to large images and the recurrent neural network for processing temporal sequences. finally, we present general guidelines for the practical methodology involved in designing, building, and conﬁguring an application involving deep learning, and review some of the applications of deep learning. these chapters are the most important for a practitioner — someone who wants to begin implementing and using deep learning algorithms to solve real - world problems today. 167


chapter 6 deep feedforward networks deep feedforward networks, also often called feedforward neural networks, or multilayer perceptrons ( mlps ), are the quintessential deep learning models. the goal of a feedforward network is to approximate some functionf ∗. for example, for a classiﬁer, y = f ∗ ( x ) maps an input x to a category y. a feedforward network deﬁnes a mapping y = f ( x ; θ ) and learns the value of the parameters θ that result in the best function approximation. these models are called feedforward because information ﬂows through the function being evaluated from x, through the intermediate computations used to deﬁne f, and ﬁnally to the output y. there are no feedback connections in which outputs of the model are fed back into itself. when feedforward neural networks are extended to include feedback connections, they are called recurrent neural networks, presented in chapter. 10 feedforward networks are of extreme importance to machine learning practi - tioners. they form the basis of many important commercial applications. for example, the convolutional networks used for object recognition from photos are a specialized kind of feedforward network


are of extreme importance to machine learning practi - tioners. they form the basis of many important commercial applications. for example, the convolutional networks used for object recognition from photos are a specialized kind of feedforward network. feedforward networks are a conceptual stepping stone on the path to recurrent networks, which power many natural language applications. feedforward neural networks are called networks because they are typically represented by composing together many [UNK] functions. the model is asso - ciated with a directed acyclic graph describing how the functions are composed together. for example, we might have three functions f ( 1 ), f ( 2 ), and f ( 3 ) connected in a chain, to form f ( x ) = f ( 3 ) ( f ( 2 ) ( f ( 1 ) ( x ) ) ). these chain structures are the most commonly used structures of neural networks. in this case, f ( 1 ) is called the ﬁrst layer of the network, f ( 2 ) is called the second layer, and so on. the overall 168


chapter 6. deep feedforward networks length of the chain gives the depth of the model. it is from this terminology that the name “ deep learning ” arises. the ﬁnal layer of a feedforward network is called the output layer. during neural network training, we drive f ( x ) to match f∗ ( x ). the training data provides us with noisy, approximate examples of f ∗ ( x ) evaluated at [UNK] training points. each example x is accompanied by a label y f ≈ ∗ ( x ). the training examples specify directly what the output layer must do at each point x ; it must produce a value that is close to y. the behavior of the other layers is not directly speciﬁed by the training data. the learning algorithm must decide how to use those layers to produce the desired output, but the training data does not say what each individual layer should do. instead, the learning algorithm must decide how to use these layers to best implement an approximation of f∗. because the training data does not show the desired output for each of these layers, these layers are called hidden layers. finally, these networks are called neural because they are loosely inspired by neuroscience. each hidden layer of the network is typically vector - valued. the dimensionality of


does not show the desired output for each of these layers, these layers are called hidden layers. finally, these networks are called neural because they are loosely inspired by neuroscience. each hidden layer of the network is typically vector - valued. the dimensionality of these hidden layers determines the width of the model. each element of the vector may be interpreted as playing a role analogous to a neuron. rather than thinking of the layer as representing a single vector - to - vector function, we can also think of the layer as consisting of many units that act in parallel, each representing a vector - to - scalar function. each unit resembles a neuron in the sense that it receives input from many other units and computes its own activation value. the idea of using many layers of vector - valued representation is drawn from neuroscience. the choice of the functions f ( ) i ( x ) used to compute these representations is also loosely guided by neuroscientiﬁc observations about the functions that biological neurons compute. however, modern neural network research is guided by many mathematical and engineering disciplines, and the goal of neural networks is not to perfectly model the brain. it is best to think of feedforward networks as function approximation machines that are designed to achieve statistical generalization, occasionally drawing


network research is guided by many mathematical and engineering disciplines, and the goal of neural networks is not to perfectly model the brain. it is best to think of feedforward networks as function approximation machines that are designed to achieve statistical generalization, occasionally drawing some insights from what we know about the brain, rather than as models of brain function. one way to understand feedforward networks is to begin with linear models and consider how to overcome their limitations. linear models, such as logistic regression and linear regression, are appealing because they may be ﬁt [UNK] and reliably, either in closed form or with convex optimization. linear models also have the obvious defect that the model capacity is limited to linear functions, so the model cannot understand the interaction between any two input variables. to extend linear models to represent nonlinear functions of x, we can apply the linear model not to x itself but to a transformed input φ ( x ), where φ is a 169


chapter 6. deep feedforward networks nonlinear transformation. equivalently, we can apply the kernel trick described in section, to obtain a nonlinear learning algorithm based on implicitly applying 5. 7. 2 the φ mapping. we can think of φ as providing a set of features describing x, or as providing a new representation for. x the question is then how to choose the mapping. φ 1. one option is to use a very generic φ, such as the inﬁnite - dimensional φ that is implicitly used by kernel machines based on the rbf kernel. if φ ( x ) is of high enough dimension, we can always have enough capacity to ﬁt the training set, but generalization to the test set often remains poor. very generic feature mappings are usually based only on the principle of local smoothness and do not encode enough prior information to solve advanced problems. 2. another option is to manually engineer φ. until the advent of deep learning, this was the dominant approach. this approach requires decades of human [UNK] for each separate task, with practitioners specializing in [UNK] domains such as speech recognition or computer vision, and with little transfer between domains. 3. the strategy of deep learning is to learn φ. in this approach, we have a model y


of human [UNK] for each separate task, with practitioners specializing in [UNK] domains such as speech recognition or computer vision, and with little transfer between domains. 3. the strategy of deep learning is to learn φ. in this approach, we have a model y = f ( x ; θ w, ) = φ ( x ; θ ) w. we now have parameters θ that we use to learn φ from a broad class of functions, and parameters w that map from φ ( x ) to the desired output. this is an example of a deep feedforward network, with φ deﬁning a hidden layer. this approach is the only one of the three that gives up on the convexity of the training problem, but the beneﬁts outweigh the harms. in this approach, we parametrize the representation as φ ( x ; θ ) and use the optimization algorithm to ﬁnd the θ that corresponds to a good representation. if we wish, this approach can capture the beneﬁt of the ﬁrst approach by being highly generic — we do so by using a very broad family φ ( x ; θ ). this approach can also capture the beneﬁt of the second approach. human practitioners can encode their knowledge to help generalization by


##rst approach by being highly generic — we do so by using a very broad family φ ( x ; θ ). this approach can also capture the beneﬁt of the second approach. human practitioners can encode their knowledge to help generalization by designing families φ ( x ; θ ) that they expect will perform well. the advantage is that the human designer only needs to ﬁnd the right general function family rather than ﬁnding precisely the right function. this general principle of improving models by learning features extends beyond the feedforward networks described in this chapter. it is a recurring theme of deep learning that applies to all of the kinds of models described throughout this book. feedforward networks are the application of this principle to learning deterministic 170


chapter 6. deep feedforward networks mappings from x to y that lack feedback connections. other models presented later will apply these principles to learning stochastic mappings, learning functions with feedback, and learning probability distributions over a single vector. we begin this chapter with a simple example of a feedforward network. next, we address each of the design decisions needed to deploy a feedforward network. first, training a feedforward network requires making many of the same design decisions as are necessary for a linear model : choosing the optimizer, the cost function, and the form of the output units. we review these basics of gradient - based learning, then proceed to confront some of the design decisions that are unique to feedforward networks. feedforward networks have introduced the concept of a hidden layer, and this requires us to choose the activation functions that will be used to compute the hidden layer values. we must also design the architecture of the network, including how many layers the network should contain, how these layers should be connected to each other, and how many units should be in each layer. learning in deep neural networks requires computing the gradients of complicated functions. we present the back - propagation algorithm and its modern generalizations, which can be used to


layers should be connected to each other, and how many units should be in each layer. learning in deep neural networks requires computing the gradients of complicated functions. we present the back - propagation algorithm and its modern generalizations, which can be used to [UNK] compute these gradients. finally, we close with some historical perspective. 6. 1 example : learning xor to make the idea of a feedforward network more concrete, we begin with an example of a fully functioning feedforward network on a very simple task : learning the xor function. the xor function ( “ exclusive or ” ) is an operation on two binary values, x1 and x2. when exactly one of these binary values is equal to, the xor function 1 returns. otherwise, it returns 0. the xor function provides the target function 1 y = f∗ ( x ) that we want to learn. our model provides a function y = f ( x ; θ ) and our learning algorithm will adapt the parameters θ to make f as similar as possible to f∗. in this simple example, we will not be concerned with statistical generalization. we want our network to perform correctly on the four points x = { [ 0, 0 ], [ 0, 1 ], [


as similar as possible to f∗. in this simple example, we will not be concerned with statistical generalization. we want our network to perform correctly on the four points x = { [ 0, 0 ], [ 0, 1 ], [ 1, 0 ], and [ 1, 1 ] }. we will train the network on all four of these points. the only challenge is to ﬁt the training set. we can treat this problem as a regression problem and use a mean squared error loss function. we choose this loss function to simplify the math for this example as much as possible. in practical applications, mse is usually not an 171


chapter 6. deep feedforward networks appropriate cost function for modeling binary data. more appropriate approaches are described in section. 6. 2. 2. 2 evaluated on our whole training set, the mse loss function is j ( ) = θ 1 4 x∈x ( f∗ ( ) ( ; ) ) x −f x θ 2. ( 6. 1 ) now we must choose the form of our model, f ( x ; θ ). suppose that we choose a linear model, with consisting of and. our model is deﬁned to be θ w b f, b ( ; x w ) = xw + b. ( 6. 2 ) we can minimize j ( θ ) in closed form with respect to w and b using the normal equations. after solving the normal equations, we obtain w = 0 and b = 1 2. the linear model simply outputs 0. 5 everywhere. why does this happen? figure shows 6. 1 how a linear model is not able to represent the xor function. one way to solve this problem is to use a model that learns a [UNK] feature space in which a linear model is able to represent the solution. speciﬁcally, we will introduce a very simple feedforward network with one hidden layer


. one way to solve this problem is to use a model that learns a [UNK] feature space in which a linear model is able to represent the solution. speciﬁcally, we will introduce a very simple feedforward network with one hidden layer containing two hidden units. see ﬁgure for an illustration of 6. 2 this model. this feedforward network has a vector of hidden units h that are computed by a function f ( 1 ) ( x ; w c, ). the values of these hidden units are then used as the input for a second layer. the second layer is the output layer of the network. the output layer is still just a linear regression model, but now it is applied to h rather than to x. the network now contains two functions chained together : h = f ( 1 ) ( x ; w c, ) and y = f ( 2 ) ( h ; w, b ), with the complete model being f,,, b f ( ; x w c w ) = ( 2 ) ( f ( 1 ) ( ) ) x. what function should f ( 1 ) compute? linear models have served us well so far, and it may be tempting to make f ( 1 ) be linear as well. unfortunately, if


( 2 ) ( f ( 1 ) ( ) ) x. what function should f ( 1 ) compute? linear models have served us well so far, and it may be tempting to make f ( 1 ) be linear as well. unfortunately, if f ( 1 ) were linear, then the feedforward network as a whole would remain a linear function of its input. ignoring the intercept terms for the moment, suppose f ( 1 ) ( x ) = wx and f ( 2 ) ( h ) = hw. then f ( x ) = ww x. we could represent this function as f ( ) = x xwwhere w = w w. clearly, we must use a nonlinear function to describe the features. most neural networks do so using an [UNK] transformation controlled by learned parameters, followed by a ﬁxed, nonlinear function called an activation function. we use that strategy here, by deﬁning h = g ( w x + c ), where w provides the weights of a linear transformation and c the biases. previously, to describe a linear regression 172


chapter 6. deep feedforward networks 0 1 x1 0 1 x2 original space x 0 1 2 h1 0 1 h2 learned space h figure 6. 1 : solving the xor problem by learning a representation. the bold numbers printed on the plot indicate the value that the learned function must output at each point. ( left ) a linear model applied directly to the original input cannot implement the xor function. when x1 = 0, the model ’ s output must increase as x2 increases. when x1 = 1, the model ’ s output must decrease as x2 increases. a linear model must apply a ﬁxed [UNK] w2 to x2. the linear model therefore cannot use the value of x1 to change the [UNK] on x2 and cannot solve this problem. ( right ) in the transformed space represented by the features extracted by a neural network, a linear model can now solve the problem. in our example solution, the two points that must have output have been 1 collapsed into a single point in feature space. in other words, the nonlinear features have mapped both x = [ 1, 0 ] and x = [ 0, 1 ] to a single point in feature space, h = [ 1, 0 ]. the linear model can now


in feature space. in other words, the nonlinear features have mapped both x = [ 1, 0 ] and x = [ 0, 1 ] to a single point in feature space, h = [ 1, 0 ]. the linear model can now describe the function as increasing in h1 and decreasing in h2. in this example, the motivation for learning the feature space is only to make the model capacity greater so that it can ﬁt the training set. in more realistic applications, learned representations can also help the model to generalize. 173


chapter 6. deep feedforward networks y h x w w y h1 h1 x1 x1 h2 h2 x2 x2 figure 6. 2 : an example of a feedforward network, drawn in two [UNK] styles. speciﬁcally, this is the feedforward network we use to solve the xor example. it has a single hidden layer containing two units. ( left ) in this style, we draw every unit as a node in the graph. this style is very explicit and unambiguous but for networks larger than this example it can consume too much space. in this style, we draw a node in the graph for ( right ) each entire vector representing a layer ’ s activations. this style is much more compact. sometimes we annotate the edges in this graph with the name of the parameters that describe the relationship between two layers. here, we indicate that a matrixw describes the mapping from x to h, and a vector w describes the mapping from h to y. we typically omit the intercept parameters associated with each layer when labeling this kind of drawing. model, we used a vector of weights and a scalar bias parameter to describe an [UNK] transformation from an input vector to an output scalar.


to y. we typically omit the intercept parameters associated with each layer when labeling this kind of drawing. model, we used a vector of weights and a scalar bias parameter to describe an [UNK] transformation from an input vector to an output scalar. now, we describe an [UNK] transformation from a vector x to a vector h, so an entire vector of bias parameters is needed. the activation function g is typically chosen to be a function that is applied element - wise, with hi = g ( xw :, i + c i ). in modern neural networks, the default recommendation is to use the rectiﬁed linear unit or relu ( jarrett et al. et al., ;, ; 2009 nair and hinton 2010 glorot, ) deﬁned by the activation 2011a function depicted in ﬁgure. g z, z ( ) = max 0 { } 6. 3 we can now specify our complete network as f,,, b ( ; x w c w ) = wmax 0 {, w x c + } + b. ( 6. 3 ) we can now specify a solution to the xor problem. let w = 1 1 1 1, ( 6. 4 ) c = 0 −1, ( 6. 5


{, w x c + } + b. ( 6. 3 ) we can now specify a solution to the xor problem. let w = 1 1 1 1, ( 6. 4 ) c = 0 −1, ( 6. 5 ) 174


chapter 6. deep feedforward networks 0 z 0 g z ( ) = max 0 {, z } figure 6. 3 : the rectiﬁed linear activation function. this activation function is the default activation function recommended for use with most feedforward neural networks. applying this function to the output of a linear transformation yields a nonlinear transformation. however, the function remains very close to linear, in the sense that is a piecewise linear function with two linear pieces. because rectiﬁed linear units are nearly linear, they preserve many of the properties that make linear models easy to optimize with gradient - based methods. they also preserve many of the properties that make linear models generalize well. a common principle throughout computer science is that we can build complicated systems from minimal components. much as a turing machine ’ s memory needs only to be able to store 0 or 1 states, we can build a universal function approximator from rectiﬁed linear functions. 175


chapter 6. deep feedforward networks w = 1 −2, ( 6. 6 ) and. b = 0 we can now walk through the way that the model processes a batch of inputs. let x be the design matrix containing all four points in the binary input space, with one example per row : x = 0 0 0 1 1 0 1 1. ( 6. 7 ) the ﬁrst step in the neural network is to multiply the input matrix by the ﬁrst layer ’ s weight matrix : xw = 0 0 1 1 1 1 2 2. ( 6. 8 ) next, we add the bias vector, to obtain c 0 1 − 1 0 1 0 2 1. ( 6. 9 ) in this space, all of the examples lie along a line with slope. as we move along 1 this line, the output needs to begin at, then rise to, then drop back down to. 0 1 0 a linear model cannot implement such a function. to ﬁnish computing the value of for each example, we apply the rectiﬁed linear transformation : h 0 0 1 0 1 0 2 1. ( 6. 10 ) this transformation has changed the relationship between the examples. they no longer lie on a single line. as shown in


each example, we apply the rectiﬁed linear transformation : h 0 0 1 0 1 0 2 1. ( 6. 10 ) this transformation has changed the relationship between the examples. they no longer lie on a single line. as shown in ﬁgure, they now lie in a space where a 6. 1 linear model can solve the problem. we ﬁnish by multiplying by the weight vector : w 0 1 1 0. ( 6. 11 ) 176


chapter 6. deep feedforward networks the neural network has obtained the correct answer for every example in the batch. in this example, we simply speciﬁed the solution, then showed that it obtained zero error. in a real situation, there might be billions of model parameters and billions of training examples, so one cannot simply guess the solution as we did here. instead, a gradient - based optimization algorithm can ﬁnd parameters that produce very little error. the solution we described to the xor problem is at a global minimum of the loss function, so gradient descent could converge to this point. there are other equivalent solutions to the xor problem that gradient descent could also ﬁnd. the convergence point of gradient descent depends on the initial values of the parameters. in practice, gradient descent would usually not ﬁnd clean, easily understood, integer - valued solutions like the one we presented here. 6. 2 gradient - based learning designing and training a neural network is not much [UNK] from training any other machine learning model with gradient descent. in section, we described 5. 10 how to build a machine learning algorithm by specifying an optimization procedure, a cost function, and a model family. the largest [UNK] between the linear models we have seen so far and neural networks is that the


. in section, we described 5. 10 how to build a machine learning algorithm by specifying an optimization procedure, a cost function, and a model family. the largest [UNK] between the linear models we have seen so far and neural networks is that the nonlinearity of a neural network causes most interesting loss functions to become non - convex. this means that neural networks are usually trained by using iterative, gradient - based optimizers that merely drive the cost function to a very low value, rather than the linear equation solvers used to train linear regression models or the convex optimization algorithms with global conver - gence guarantees used to train logistic regression or svms. convex optimization converges starting from any initial parameters ( in theory — in practice it is very robust but can encounter numerical problems ). stochastic gradient descent applied to non - convex loss functions has no such convergence guarantee, and is sensitive to the values of the initial parameters. for feedforward neural networks, it is important to initialize all weights to small random values. the biases may be initialized to zero or to small positive values. the iterative gradient - based opti - mization algorithms used to train feedforward networks and almost all other deep models will be described in detail in chapter


random values. the biases may be initialized to zero or to small positive values. the iterative gradient - based opti - mization algorithms used to train feedforward networks and almost all other deep models will be described in detail in chapter, with parameter initialization in 8 particular discussed in section. for the moment, it [UNK] to understand that 8. 4 the training algorithm is almost always based on using the gradient to descend the cost function in one way or another. the speciﬁc algorithms are improvements and reﬁnements on the ideas of gradient descent, introduced in section, and, 4. 3 177


chapter 6. deep feedforward networks more speciﬁcally, are most often improvements of the stochastic gradient descent algorithm, introduced in section. 5. 9 we can of course, train models such as linear regression and support vector machines with gradient descent too, and in fact this is common when the training set is extremely large. from this point of view, training a neural network is not much [UNK] from training any other model. computing the gradient is slightly more complicated for a neural network, but can still be done [UNK] and exactly. section will describe how to obtain the gradient using the back - propagation 6. 5 algorithm and modern generalizations of the back - propagation algorithm. as with other machine learning models, to apply gradient - based learning we must choose a cost function, and we must choose how to represent the output of the model. we now revisit these design considerations with special emphasis on the neural networks scenario. 6. 2. 1 cost functions an important aspect of the design of a deep neural network is the choice of the cost function. fortunately, the cost functions for neural networks are more or less the same as those for other parametric models, such as linear models. in most cases, our parametric model deﬁnes a distribution p (


is the choice of the cost function. fortunately, the cost functions for neural networks are more or less the same as those for other parametric models, such as linear models. in most cases, our parametric model deﬁnes a distribution p ( y x | ; θ ) and we simply use the principle of maximum likelihood. this means we use the cross - entropy between the training data and the model ’ s predictions as the cost function. sometimes, we take a simpler approach, where rather than predicting a complete probability distribution over y, we merely predict some statistic of y conditioned on. specialized loss functions allow us to train a predictor of these estimates. x the total cost function used to train a neural network will often combine one of the primary cost functions described here with a regularization term. we have already seen some simple examples of regularization applied to linear models in section. the weight decay approach used for linear models is also directly 5. 2. 2 applicable to deep neural networks and is among the most popular regularization strategies. more advanced regularization strategies for neural networks will be described in chapter. 7 6. 2. 1. 1 learning conditional distributions with maximum likelihood most modern neural networks are trained using maximum likelihood. this means that the cost function is simply the


##ization strategies. more advanced regularization strategies for neural networks will be described in chapter. 7 6. 2. 1. 1 learning conditional distributions with maximum likelihood most modern neural networks are trained using maximum likelihood. this means that the cost function is simply the negative log - likelihood, equivalently described 178


chapter 6. deep feedforward networks as the cross - entropy between the training data and the model distribution. this cost function is given by j ( ) = θ −ex y, [UNK] log pmodel ( ) y x |. ( 6. 12 ) the speciﬁc form of the cost function changes from model to model, depending on the speciﬁc form of log pmodel. the expansion of the above equation typically yields some terms that do not depend on the model parameters and may be dis - carded. for example, as we saw in section, if 5. 5. 1 pmodel ( y x | ) = n ( y ; f ( x ; θ ), i ), then we recover the mean squared error cost, j θ ( ) = 1 2ex y, [UNK] | | − | | y f ( ; ) x θ 2 + const, ( 6. 13 ) up to a scaling factor of 1 2 and a term that does not depend on. the discarded θ constant is based on the variance of the gaussian distribution, which in this case we chose not to parametrize. previously, we saw that the equivalence between maximum likelihood estimation with an output distribution and minimization of mean


. the discarded θ constant is based on the variance of the gaussian distribution, which in this case we chose not to parametrize. previously, we saw that the equivalence between maximum likelihood estimation with an output distribution and minimization of mean squared error holds for a linear model, but in fact, the equivalence holds regardless of the used to predict the mean of the gaussian. f ( ; ) x θ an advantage of this approach of deriving the cost function from maximum likelihood is that it removes the burden of designing cost functions for each model. specifying a model p ( y x | ) automatically determines a cost function log p ( y x | ). one recurring theme throughout neural network design is that the gradient of the cost function must be large and predictable enough to serve as a good guide for the learning algorithm. functions that saturate ( become very ﬂat ) undermine this objective because they make the gradient become very small. in many cases this happens because the activation functions used to produce the output of the hidden units or the output units saturate. the negative log - likelihood helps to avoid this problem for many models. many output units involve an exp function that can saturate when its argument is very negative. the log function in the


the hidden units or the output units saturate. the negative log - likelihood helps to avoid this problem for many models. many output units involve an exp function that can saturate when its argument is very negative. the log function in the negative log - likelihood cost function undoes the exp of some output units. we will discuss the interaction between the cost function and the choice of output unit in section. 6. 2. 2 one unusual property of the cross - entropy cost used to perform maximum likelihood estimation is that it usually does not have a minimum value when applied to the models commonly used in practice. for discrete output variables, most models are parametrized in such a way that they cannot represent a probability of zero or one, but can come arbitrarily close to doing so. logistic regression is an example of such a model. for real - valued output variables, if the model 179


chapter 6. deep feedforward networks can control the density of the output distribution ( for example, by learning the variance parameter of a gaussian output distribution ) then it becomes possible to assign extremely high density to the correct training set outputs, resulting in cross - entropy approaching negative inﬁnity. regularization techniques described in chapter provide several [UNK] ways of modifying the learning problem so 7 that the model cannot reap unlimited reward in this way. 6. 2. 1. 2 learning conditional statistics instead of learning a full probability distribution p ( y x | ; θ ) we often want to learn just one conditional statistic of given. y x for example, we may have a predictor f ( x ; θ ) that we wish to predict the mean of. y if we use a [UNK] powerful neural network, we can think of the neural network as being able to represent any function f from a wide class of functions, with this class being limited only by features such as continuity and boundedness rather than by having a speciﬁc parametric form. from this point of view, we can view the cost function as being a functional rather than just a function. a functional is a mapping from functions to real numbers. we can thus think of learning as choosing a function rather


##ﬁc parametric form. from this point of view, we can view the cost function as being a functional rather than just a function. a functional is a mapping from functions to real numbers. we can thus think of learning as choosing a function rather than merely choosing a set of parameters. we can design our cost functional to have its minimum occur at some speciﬁc function we desire. for example, we can design the cost functional to have its minimum lie on the function that maps x to the expected value of y given x. solving an optimization problem with respect to a function requires a mathematical tool called calculus of variations, described in section. it is not necessary 19. 4. 2 to understand calculus of variations to understand the content of this chapter. at the moment, it is only necessary to understand that calculus of variations may be used to derive the following two results. our ﬁrst result derived using calculus of variations is that solving the optimiza - tion problem f ∗ = arg min f ex y, [UNK] | | − | | y f ( ) x 2 ( 6. 14 ) yields f∗ ( ) = x [UNK] ( ) y x | [ ] y, ( 6. 15 ) so long as this function lies within the class we opt


[UNK] | | − | | y f ( ) x 2 ( 6. 14 ) yields f∗ ( ) = x [UNK] ( ) y x | [ ] y, ( 6. 15 ) so long as this function lies within the class we optimize over. in other words, if we could train on inﬁnitely many samples from the true data generating distribution, minimizing the mean squared error cost function gives a function that predicts the mean of for each value of. y x 180


chapter 6. deep feedforward networks [UNK] cost functions give [UNK] statistics. a second result derived using calculus of variations is that f ∗ = arg min f ex y, [UNK] | | − | | y f ( ) x 1 ( 6. 16 ) yields a function that predicts the median value of y for each x, so long as such a function may be described by the family of functions we optimize over. this cost function is commonly called. mean absolute error unfortunately, mean squared error and mean absolute error often lead to poor results when used with gradient - based optimization. some output units that saturate produce very small gradients when combined with these cost functions. this is one reason that the cross - entropy cost function is more popular than mean squared error or mean absolute error, even when it is not necessary to estimate an entire distribution. p ( ) y x | 6. 2. 2 output units the choice of cost function is tightly coupled with the choice of output unit. most of the time, we simply use the cross - entropy between the data distribution and the model distribution. the choice of how to represent the output then determines the form of the cross - entropy function. any kind of neural network unit that may be used as an output can also


we simply use the cross - entropy between the data distribution and the model distribution. the choice of how to represent the output then determines the form of the cross - entropy function. any kind of neural network unit that may be used as an output can also be used as a hidden unit. here, we focus on the use of these units as outputs of the model, but in principle they can be used internally as well. we revisit these units with additional detail about their use as hidden units in section. 6. 3 throughout this section, we suppose that the feedforward network provides a set of hidden features deﬁned by h = f ( x ; θ ). the role of the output layer is then to provide some additional transformation from the features to complete the task that the network must perform. 6. 2. 2. 1 linear units for gaussian output distributions one simple kind of output unit is an output unit based on an [UNK] transformation with no nonlinearity. these are often just called linear units. given features h, a layer of linear output units produces a vector [UNK] = wh + b. linear output layers are often used to produce the mean of a conditional gaussian distribution : p ( ) = ( ; y x | n y [UNK] i


features h, a layer of linear output units produces a vector [UNK] = wh + b. linear output layers are often used to produce the mean of a conditional gaussian distribution : p ( ) = ( ; y x | n y [UNK] i, ). ( 6. 17 ) 181


chapter 6. deep feedforward networks maximizing the log - likelihood is then equivalent to minimizing the mean squared error. the maximum likelihood framework makes it straightforward to learn the covariance of the gaussian too, or to make the covariance of the gaussian be a function of the input. however, the covariance must be constrained to be a positive deﬁnite matrix for all inputs. it is [UNK] to satisfy such constraints with a linear output layer, so typically other output units are used to parametrize the covariance. approaches to modeling the covariance are described shortly, in section. 6. 2. 2. 4 because linear units do not saturate, they pose little [UNK] for gradient - based optimization algorithms and may be used with a wide variety of optimization algorithms. 6. 2. 2. 2 sigmoid units for bernoulli output distributions many tasks require predicting the value of a binary variable y. classiﬁcation problems with two classes can be cast in this form. the maximum - likelihood approach is to deﬁne a bernoulli distribution over y conditioned on. x a bernoulli distribution is deﬁned by just a single number. the neural net needs to predict only p


be cast in this form. the maximum - likelihood approach is to deﬁne a bernoulli distribution over y conditioned on. x a bernoulli distribution is deﬁned by just a single number. the neural net needs to predict only p ( y = 1 | x ). for this number to be a valid probability, it must lie in the interval [ 0, 1 ]. satisfying this constraint requires some careful design [UNK]. suppose we were to use a linear unit, and threshold its value to obtain a valid probability : p y ( = 1 ) = max | x 0 min, 1, wh + b. ( 6. 18 ) this would indeed deﬁne a valid conditional distribution, but we would not be able to train it very [UNK] with gradient descent. any time that wh + b strayed outside the unit interval, the gradient of the output of the model with respect to its parameters would be 0. a gradient of 0 is typically problematic because the learning algorithm no longer has a guide for how to improve the corresponding parameters. instead, it is better to use a [UNK] approach that ensures there is always a strong gradient whenever the model has the wrong answer. this approach is based on using sigmoid output units combined with maximum likelihood. a


how to improve the corresponding parameters. instead, it is better to use a [UNK] approach that ensures there is always a strong gradient whenever the model has the wrong answer. this approach is based on using sigmoid output units combined with maximum likelihood. a sigmoid output unit is deﬁned by [UNK] σ = wh + b ( 6. 19 ) 182


chapter 6. deep feedforward networks where is the logistic sigmoid function described in section. σ 3. 10 we can think of the sigmoid output unit as having two components. first, it uses a linear layer to compute z = wh + b. next, it uses the sigmoid activation function to convert into a probability. z we omit the dependence on x for the moment to discuss how to deﬁne a probability distribution over y using the value z. the sigmoid can be motivated by constructing an unnormalized probability distribution [UNK] ( y ), which does not sum to 1. we can then divide by an appropriate constant to obtain a valid probability distribution. if we begin with the assumption that the unnormalized log probabilities are linear in y and z, we can exponentiate to obtain the unnormalized probabilities. we then normalize to see that this yields a bernoulli distribution controlled by a sigmoidal transformation of : z log [UNK] y yz ( ) = ( 6. 20 ) [UNK] y yz ( ) = exp ( ) ( 6. 21 ) p y ( ) = exp ( ) yz 1 y = 0 exp ( yz


z log [UNK] y yz ( ) = ( 6. 20 ) [UNK] y yz ( ) = exp ( ) ( 6. 21 ) p y ( ) = exp ( ) yz 1 y = 0 exp ( yz ) ( 6. 22 ) p y σ y z. ( ) = ( ( 2 −1 ) ) ( 6. 23 ) probability distributions based on exponentiation and normalization are common throughout the statistical modeling literature. the z variable deﬁning such a distribution over binary variables is called a. logit this approach to predicting the probabilities in log - space is natural to use with maximum likelihood learning. because the cost function used with maximum likelihood is −log p ( y | x ), the log in the cost function undoes the exp of the sigmoid. without this [UNK], the saturation of the sigmoid could prevent gradient - based learning from making good progress. the loss function for maximum likelihood learning of a bernoulli parametrized by a sigmoid is j p y ( ) = log θ − ( | x ) ( 6. 24 ) = log ( ( 2 1 ) ) − σ y − z ( 6. 25 ) = ( ( 1 2 ) )


##rized by a sigmoid is j p y ( ) = log θ − ( | x ) ( 6. 24 ) = log ( ( 2 1 ) ) − σ y − z ( 6. 25 ) = ( ( 1 2 ) ) ζ −y z. ( 6. 26 ) this derivation makes use of some properties from section. by rewriting 3. 10 the loss in terms of the softplus function, we can see that it saturates only when ( 1 −2y ) z is very negative. saturation thus occurs only when the model already has the right answer — when y = 1 and z is very positive, or y = 0 and z is very negative. when z has the wrong sign, the argument to the softplus function, 183


chapter 6. deep feedforward networks ( 1 −2y ) z, may be simpliﬁed to | | z. as | | z becomes large while z has the wrong sign, the softplus function asymptotes toward simply returning its argument | | z. the derivative with respect to z asymptotes to sign ( z ), so, in the limit of extremely incorrect z, the softplus function does not shrink the gradient at all. this property is very useful because it means that gradient - based learning can act to quickly correct a mistaken. z when we use other loss functions, such as mean squared error, the loss can saturate anytime σ ( z ) saturates. the sigmoid activation function saturates to 0 when z becomes very negative and saturates to when 1 z becomes very positive. the gradient can shrink too small to be useful for learning whenever this happens, whether the model has the correct answer or the incorrect answer. for this reason, maximum likelihood is almost always the preferred approach to training sigmoid output units. analytically, the logarithm of the sigmoid is always deﬁned and ﬁnite, because the sigmoid returns values restricted


this reason, maximum likelihood is almost always the preferred approach to training sigmoid output units. analytically, the logarithm of the sigmoid is always deﬁned and ﬁnite, because the sigmoid returns values restricted to the open interval ( 0, 1 ), rather than using the entire closed interval of valid probabilities [ 0, 1 ]. in software implementations, to avoid numerical problems, it is best to write the negative log - likelihood as a function of z, rather than as a function of [UNK] = σ ( z ). if the sigmoid function underﬂows to zero, then taking the logarithm of [UNK] yields negative inﬁnity. 6. 2. 2. 3 softmax units for multinoulli output distributions any time we wish to represent a probability distribution over a discrete variable with n possible values, we may use the softmax function. this can be seen as a generalization of the sigmoid function which was used to represent a probability distribution over a binary variable. softmax functions are most often used as the output of a classiﬁer, to represent the probability distribution over n [UNK] classes. more rarely, softmax functions can be used inside the model itself, if we


probability distribution over a binary variable. softmax functions are most often used as the output of a classiﬁer, to represent the probability distribution over n [UNK] classes. more rarely, softmax functions can be used inside the model itself, if we wish the model to choose between one of n [UNK] options for some internal variable. in the case of binary variables, we wished to produce a single number [UNK] p y. = ( = 1 ) | x ( 6. 27 ) because this number needed to lie between and, and because we wanted the 0 1 logarithm of the number to be well - behaved for gradient - based optimization of the log - likelihood, we chose to instead predict a number z = log [UNK] ( y = 1 | x ). exponentiating and normalizing gave us a bernoulli distribution controlled by the sigmoid function. 184


chapter 6. deep feedforward networks to generalize to the case of a discrete variable with n values, we now need to produce a vector [UNK], with [UNK] = p ( y = i | x ). we require not only that each element of [UNK] be between and, but also that the entire vector sums to so that 0 1 1 it represents a valid probability distribution. the same approach that worked for the bernoulli distribution generalizes to the multinoulli distribution. first, a linear layer predicts unnormalized log probabilities : z w = h b +, ( 6. 28 ) where zi = log [UNK] ( y = i | x ). the softmax function can then exponentiate and normalize to obtain the desired z [UNK]. formally, the softmax function is given by softmax ( ) z i = exp ( zi ) j exp ( zj ). ( 6. 29 ) as with the logistic sigmoid, the use of the exp function works very well when training the softmax to output a target value y using maximum log - likelihood. in this case, we wish to maximize log p ( y = i ; z ) = log softmax ( z ) i. de


exp function works very well when training the softmax to output a target value y using maximum log - likelihood. in this case, we wish to maximize log p ( y = i ; z ) = log softmax ( z ) i. deﬁning the softmax in terms of exp is natural because the log in the log - likelihood can undo the of the softmax : exp log softmax ( ) z i = zi −log j exp ( zj ). ( 6. 30 ) the ﬁrst term of equation shows that the input 6. 30 zi always has a direct contribution to the cost function. because this term cannot saturate, we know that learning can proceed, even if the contribution of zi to the second term of equation becomes very small. when maximizing the log - likelihood, the ﬁrst 6. 30 term encourages zi to be pushed up, while the second term encourages all ofz to be pushed down. to gain some intuition for the second term, log j exp ( zj ), observe that this term can be roughly approximated by maxj zj. this approximation is based on the idea that exp ( zk ) is insigniﬁcant for any zk


, log j exp ( zj ), observe that this term can be roughly approximated by maxj zj. this approximation is based on the idea that exp ( zk ) is insigniﬁcant for any zk that is noticeably less than maxj zj. the intuition we can gain from this approximation is that the negative log - likelihood cost function always strongly penalizes the most active incorrect prediction. if the correct answer already has the largest input to the softmax, then the −zi term and the log j exp ( zj ) ≈maxj zj = zi terms will roughly cancel. this example will then contribute little to the overall training cost, which will be dominated by other examples that are not yet correctly classiﬁed. so far we have discussed only a single example. overall, unregularized maximum likelihood will drive the model to learn parameters that drive the softmax to predict 185


chapter 6. deep feedforward networks the fraction of counts of each outcome observed in the training set : softmax ( ( ; ) ) z x θ i ≈ m j = 1 1y ( ) j = i, x ( ) j = x m j = 1 1x ( ) j = x. ( 6. 31 ) because maximum likelihood is a consistent estimator, this is guaranteed to happen so long as the model family is capable of representing the training distribution. in practice, limited model capacity and imperfect optimization will mean that the model is only able to approximate these fractions. many objective functions other than the log - likelihood do not work as well with the softmax function. speciﬁcally, objective functions that do not use a log to undo the exp of the softmax fail to learn when the argument to the exp becomes very negative, causing the gradient to vanish. in particular, squared error is a poor loss function for softmax units, and can fail to train the model to change its output, even when the model makes highly conﬁdent incorrect predictions (, bridle 1990 ). to understand why these other loss functions can fail, we need to examine the softmax function itself. like the sigmoid, the


change its output, even when the model makes highly conﬁdent incorrect predictions (, bridle 1990 ). to understand why these other loss functions can fail, we need to examine the softmax function itself. like the sigmoid, the softmax activation can saturate. the sigmoid function has a single output that saturates when its input is extremely negative or extremely positive. in the case of the softmax, there are multiple output values. these output values can saturate when the [UNK] between input values become extreme. when the softmax saturates, many cost functions based on the softmax also saturate, unless they are able to invert the saturating activating function. to see that the softmax function responds to the [UNK] between its inputs, observe that the softmax output is invariant to adding the same scalar to all of its inputs : softmax ( ) = softmax ( + ) z z c. ( 6. 32 ) using this property, we can derive a numerically stable variant of the softmax : softmax ( ) = softmax ( max z z − i zi ). ( 6. 33 ) the reformulated version allows us to evaluate softmax with only small numerical errors even


can derive a numerically stable variant of the softmax : softmax ( ) = softmax ( max z z − i zi ). ( 6. 33 ) the reformulated version allows us to evaluate softmax with only small numerical errors even when z contains extremely large or extremely negative numbers. ex - amining the numerically stable variant, we see that the softmax function is driven by the amount that its arguments deviate from maxizi. an output softmax ( z ) i saturates to when the corresponding input is maximal 1 ( zi = maxizi ) and zi is much greater than all of the other inputs. the output softmax ( z ) i can also saturate to when 0 zi is not maximal and the maximum is much greater. this is a generalization of the way that sigmoid units saturate, and 186


chapter 6. deep feedforward networks can cause similar [UNK] for learning if the loss function is not designed to compensate for it. the argument z to the softmax function can be produced in two [UNK] ways. the most common is simply to have an earlier layer of the neural network output every element of z, as described above using the linear layer z = w h + b. while straightforward, this approach actually overparametrizes the distribution. the constraint that the n outputs must sum to means that only 1 n −1 parameters are necessary ; the probability of the n - th value may be obtained by subtracting the ﬁrst n−1 1 probabilities from. we can thus impose a requirement that one element of z be ﬁxed. for example, we can require that zn = 0. indeed, this is exactly what the sigmoid unit does. deﬁning p ( y = 1 | x ) = σ ( z ) is equivalent to deﬁning p ( y = 1 | x ) = softmax ( z ) 1 with a two - dimensional z and z1 = 0. both the n −1 argument and the n argument approaches to the softmax can describe the same set of probability distributions, but have [UNK] learning dynamics. in


= softmax ( z ) 1 with a two - dimensional z and z1 = 0. both the n −1 argument and the n argument approaches to the softmax can describe the same set of probability distributions, but have [UNK] learning dynamics. in practice, there is rarely much [UNK] between using the overparametrized version or the restricted version, and it is simpler to implement the overparametrized version. from a neuroscientiﬁc point of view, it is interesting to think of the softmax as a way to create a form of competition between the units that participate in it : the softmax outputs always sum to 1 so an increase in the value of one unit necessarily corresponds to a decrease in the value of others. this is analogous to the lateral inhibition that is believed to exist between nearby neurons in the cortex. at the extreme ( when the [UNK] between the maximal ai and the others is large in magnitude ) it becomes a form of winner - take - all ( one of the outputs is nearly 1 and the others are nearly 0 ). the name “ softmax ” can be somewhat confusing. the function is more closely related to the arg max function than the max function. the term “ soft ” derives from the fact that the softmax function


and the others are nearly 0 ). the name “ softmax ” can be somewhat confusing. the function is more closely related to the arg max function than the max function. the term “ soft ” derives from the fact that the softmax function is continuous and [UNK]. the arg max function, with its result represented as a one - hot vector, is not continuous or [UNK]. the softmax function thus provides a “ softened ” version of the arg max. the corresponding soft version of the maximum function is softmax ( z ) z. it would perhaps be better to call the softmax function “ softargmax, ” but the current name is an entrenched convention. 6. 2. 2. 4 other output types the linear, sigmoid, and softmax output units described above are the most common. neural networks can generalize to almost any kind of output layer that we wish. the principle of maximum likelihood provides a guide for how to design 187


chapter 6. deep feedforward networks a good cost function for nearly any kind of output layer. in general, if we deﬁne a conditional distribution p ( y x | ; θ ), the principle of maximum likelihood suggests we use as our cost function. − | log ( p y x θ ; ) in general, we can think of the neural network as representing a function f ( x ; θ ). the outputs of this function are not direct predictions of the value y. instead, f ( x ; θ ) = ω provides the parameters for a distribution over y. our loss function can then be interpreted as. −log ( ; ( ) ) p y ω x for example, we may wish to learn the variance of a conditional gaussian for y, given x. in the simple case, where the variance σ 2 is a constant, there is a closed form expression because the maximum likelihood estimator of variance is simply the empirical mean of the squared [UNK] between observationsy and their expected value. a computationally more expensive approach that does not require writing special - case code is to simply include the variance as one of the properties of the distribution p ( y | x ) that is controlled by ω = f ( x ; θ ). the negative log


computationally more expensive approach that does not require writing special - case code is to simply include the variance as one of the properties of the distribution p ( y | x ) that is controlled by ω = f ( x ; θ ). the negative log - likelihood −log p ( y ; ω ( x ) ) will then provide a cost function with the appropriate terms necessary to make our optimization procedure incrementally learn the variance. in the simple case where the standard deviation does not depend on the input, we can make a new parameter in the network that is copied directly into ω. this new parameter might be σ itself or could be a parameter v representing σ2 or it could be a parameter β representing 1 σ 2, depending on how we choose to parametrize the distribution. we may wish our model to predict a [UNK] amount of variance in y for [UNK] values of x. this is called a heteroscedastic model. in the heteroscedastic case, we simply make the speciﬁcation of the variance be one of the values output by f ( x ; θ ). a typical way to do this is to formulate the gaussian distribution using precision, rather than variance, as described in equation. 3. 22 in the multivariate


variance be one of the values output by f ( x ; θ ). a typical way to do this is to formulate the gaussian distribution using precision, rather than variance, as described in equation. 3. 22 in the multivariate case it is most common to use a diagonal precision matrix diag ( 6. 34 ) ( ) β. this formulation works well with gradient descent because the formula for the log - likelihood of the gaussian distribution parametrized by β involves only mul - tiplication by βi and addition of log βi. the gradient of multiplication, addition, and logarithm operations is well - behaved. by comparison, if we parametrized the output in terms of variance, we would need to use division. the division function becomes arbitrarily steep near zero. while large gradients can help learning, arbitrarily large gradients usually result in instability. if we parametrized the output in terms of standard deviation, the log - likelihood would still involve division, and would also involve squaring. the gradient through the squaring operation can vanish near zero, making it [UNK] to learn parameters that are squared. 188


chapter 6. deep feedforward networks regardless of whether we use standard deviation, variance, or precision, we must ensure that the covariance matrix of the gaussian is positive deﬁnite. because the eigenvalues of the precision matrix are the reciprocals of the eigenvalues of the covariance matrix, this is equivalent to ensuring that the precision matrix is positive deﬁnite. if we use a diagonal matrix, or a scalar times the diagonal matrix, then the only condition we need to enforce on the output of the model is positivity. if we suppose that a is the raw activation of the model used to determine the diagonal precision, we can use the softplus function to obtain a positive precision vector : β = ζ ( a ). this same strategy applies equally if using variance or standard deviation rather than precision or if using a scalar times identity rather than diagonal matrix. it is rare to learn a covariance or precision matrix with richer structure than diagonal. if the covariance is full and conditional, then a parametrization must be chosen that guarantees positive - deﬁniteness of the predicted covariance matrix. this can be achieved by writing σ ( ) = ( ) x b x


if the covariance is full and conditional, then a parametrization must be chosen that guarantees positive - deﬁniteness of the predicted covariance matrix. this can be achieved by writing σ ( ) = ( ) x b x b ( ) x, whereb is an unconstrained square matrix. one practical issue if the matrix is full rank is that computing the likelihood is expensive, with a d d × matrix requiring o ( d3 ) computation for the determinant and inverse of σ ( x ) ( or equivalently, and more commonly done, its eigendecomposition or that of ). b x ( ) we often want to perform multimodal regression, that is, to predict real values that come from a conditional distribution p ( y x | ) that can have several [UNK] peaks in y space for the same value of x. in this case, a gaussian mixture is a natural representation for the output (, ;, ). jacobs et al. 1991 bishop 1994 neural networks with gaussian mixtures as their output are often called mixture density networks. a gaussian mixture output with n components is deﬁned by the conditional probability distribution p ( ) = y x | n i = 1 p i


1994 neural networks with gaussian mixtures as their output are often called mixture density networks. a gaussian mixture output with n components is deﬁned by the conditional probability distribution p ( ) = y x | n i = 1 p i ( = c | n x ) ( ; y µ ( ) i ( ) x, σ ( ) i ( ) ) x. ( 6. 35 ) the neural network must have three outputs : a vector deﬁning p ( c = i | x ), a matrix providing µ ( ) i ( x ) for all i, and a tensor providing σ ( ) i ( x ) for all i. these outputs must satisfy [UNK] constraints : 1. mixture components p ( c = i | x ) : these form a multinoulli distribution over the n [UNK] components associated with latent variable1 c, and can 1we consider c to be latent because we do not observe it in the data : given input x and target y, it is not possible to know with certainty which gaussian component was responsible for y, but we can imagine that y was generated by picking one of them, and make that unobserved choice a random variable. 189


chapter 6. deep feedforward networks typically be obtained by a softmax over an n - dimensional vector, to guarantee that these outputs are positive and sum to 1. 2. means µ ( ) i ( x ) : these indicate the center or mean associated with the i - th gaussian component, and are unconstrained ( typically with no nonlinearity at all for these output units ). if y is a d - vector, then the network must output an n d × matrix containing all n of these d - dimensional vectors. learning these means with maximum likelihood is slightly more complicated than learning the means of a distribution with only one output mode. we only want to update the mean for the component that actually produced the observation. in practice, we do not know which component produced each observation. the expression for the negative log - likelihood naturally weights each example ’ s contribution to the loss for each component by the probability that the component produced the example. 3. covariances σ ( ) i ( x ) : these specify the covariance matrix for each component i. as when learning a single gaussian component, we typically use a diagonal matrix to avoid needing to compute determinants. as with learning the means of the mixture, maximum likelihood is


: these specify the covariance matrix for each component i. as when learning a single gaussian component, we typically use a diagonal matrix to avoid needing to compute determinants. as with learning the means of the mixture, maximum likelihood is complicated by needing to assign partial responsibility for each point to each mixture component. gradient descent will automatically follow the correct process if given the correct speciﬁcation of the negative log - likelihood under the mixture model. it has been reported that gradient - based optimization of conditional gaussian mixtures ( on the output of neural networks ) can be unreliable, in part because one gets divisions ( by the variance ) which can be numerically unstable ( when some variance gets to be small for a particular example, yielding very large gradients ). one solution is to clip gradients ( see section ) while another is to scale 10. 11. 1 the gradients heuristically (, ). murray and larochelle 2014 gaussian mixture outputs are particularly [UNK] in generative models of speech ( schuster 1999, ) or movements of physical objects ( graves 2013, ). the mixture density strategy gives a way for the network to represent multiple output modes and to control the variance of its output, which is crucial for obtaining a


##tive models of speech ( schuster 1999, ) or movements of physical objects ( graves 2013, ). the mixture density strategy gives a way for the network to represent multiple output modes and to control the variance of its output, which is crucial for obtaining a high degree of quality in these real - valued domains. an example of a mixture density network is shown in ﬁgure. 6. 4 in general, we may wish to continue to model larger vectors y containing more variables, and to impose richer and richer structures on these output variables. for example, we may wish for our neural network to output a sequence of characters that forms a sentence. in these cases, we may continue to use the principle of maximum likelihood applied to our model p ( y ; ω ( x ) ), but the model we use 190


chapter 6. deep feedforward networks x y figure 6. 4 : samples drawn from a neural network with a mixture density output layer. the input x is sampled from a uniform distribution and the output y is sampled from pmodel ( y x | ). the neural network is able to learn nonlinear mappings from the input to the parameters of the output distribution. these parameters include the probabilities governing which of three mixture components will generate the output as well as the parameters for each mixture component. each mixture component is gaussian with predicted mean and variance. all of these aspects of the output distribution are able to vary with respect to the input, and to do so in nonlinear ways. x to describe y becomes complex enough to be beyond the scope of this chapter. chapter describes how to use recurrent neural networks to deﬁne such models 10 over sequences, and part describes advanced techniques for modeling arbitrary iii probability distributions. 6. 3 hidden units so far we have focused our discussion on design choices for neural networks that are common to most parametric machine learning models trained with gradient - based optimization. now we turn to an issue that is unique to feedforward neural networks : how to choose the type of hidden unit to use in the hidden layers of the model.


are common to most parametric machine learning models trained with gradient - based optimization. now we turn to an issue that is unique to feedforward neural networks : how to choose the type of hidden unit to use in the hidden layers of the model. the design of hidden units is an extremely active area of research and does not yet have many deﬁnitive guiding theoretical principles. rectiﬁed linear units are an excellent default choice of hidden unit. many other types of hidden units are available. it can be [UNK] to determine when to use which kind ( though rectiﬁed linear units are usually an acceptable choice ). we 191


chapter 6. deep feedforward networks describe here some of the basic intuitions motivating each type of hidden units. these intuitions can help decide when to try out each of these units. it is usually impossible to predict in advance which will work best. the design process consists of trial and error, intuiting that a kind of hidden unit may work well, and then training a network with that kind of hidden unit and evaluating its performance on a validation set. some of the hidden units included in this list are not actually [UNK] at all input points. for example, the rectiﬁed linear function g ( z ) = max { 0, z } is not [UNK] at z = 0. this may seem like it invalidates g for use with a gradient - based learning algorithm. in practice, gradient descent still performs well enough for these models to be used for machine learning tasks. this is in part because neural network training algorithms do not usually arrive at a local minimum of the cost function, but instead merely reduce its value signiﬁcantly, as shown in ﬁgure. these ideas will be described further in chapter. because we do not 4. 3 8 expect training to actually reach a point where the gradient is 0, it is acceptable for the


its value signiﬁcantly, as shown in ﬁgure. these ideas will be described further in chapter. because we do not 4. 3 8 expect training to actually reach a point where the gradient is 0, it is acceptable for the minima of the cost function to correspond to points with undeﬁned gradient. hidden units that are not [UNK] are usually non - [UNK] at only a small number of points. in general, a function g ( z ) has a left derivative deﬁned by the slope of the function immediately to the left of z and a right derivative deﬁned by the slope of the function immediately to the right of z. a function is [UNK] at z only if both the left derivative and the right derivative are deﬁned and equal to each other. the functions used in the context of neural networks usually have deﬁned left derivatives and deﬁned right derivatives. in the case of g ( z ) = max { 0, z }, the left derivative at z = 0 0 is and the right derivative is. software implementations of neural network training usually return one of 1 the one - sided derivatives rather than reporting that the derivative is undeﬁned or raising an error. this may be heuristically justiﬁ


0 is and the right derivative is. software implementations of neural network training usually return one of 1 the one - sided derivatives rather than reporting that the derivative is undeﬁned or raising an error. this may be heuristically justiﬁed by observing that gradient - based optimization on a digital computer is subject to numerical error anyway. when a function is asked to evaluate g ( 0 ), it is very unlikely that the underlying value truly was. instead, it was likely to be some small value 0 that was rounded to. in some contexts, more theoretically pleasing justiﬁcations are available, but 0 these usually do not apply to neural network training. the important point is that in practice one can safely disregard the non - [UNK] of the hidden unit activation functions described below. unless indicated otherwise, most hidden units can be described as accepting a vector of inputs x, computing an [UNK] transformation z = w x + b, and then applying an element - wise nonlinear function g ( z ). most hidden units are distinguished from each other only by the choice of the form of the activation function. g ( ) z 192


chapter 6. deep feedforward networks 6. 3. 1 rectiﬁed linear units and their generalizations rectiﬁed linear units use the activation function. g z, z ( ) = max 0 { } rectiﬁed linear units are easy to optimize because they are so similar to linear units. the only [UNK] between a linear unit and a rectiﬁed linear unit is that a rectiﬁed linear unit outputs zero across half its domain. this makes the derivatives through a rectiﬁed linear unit remain large whenever the unit is active. the gradients are not only large but also consistent. the second derivative of the rectifying operation is almost everywhere, and the derivative of the rectifying 0 operation is everywhere that the unit is active. this means that the gradient 1 direction is far more useful for learning than it would be with activation functions that introduce second - order [UNK]. rectiﬁed linear units are typically used on top of an [UNK] transformation : h w = ( g x b + ). ( 6. 36 ) when initializing the parameters of the [UNK] transformation, it can be a good practice to set all elements of b to a small, positive value, such as 0. 1. this makes it very


g x b + ). ( 6. 36 ) when initializing the parameters of the [UNK] transformation, it can be a good practice to set all elements of b to a small, positive value, such as 0. 1. this makes it very likely that the rectiﬁed linear units will be initially active for most inputs in the training set and allow the derivatives to pass through. several generalizations of rectiﬁed linear units exist. most of these general - izations perform comparably to rectiﬁed linear units and occasionally perform better. one drawback to rectiﬁed linear units is that they cannot learn via gradient - based methods on examples for which their activation is zero. a variety of generalizations of rectiﬁed linear units guarantee that they receive gradient every - where. three generalizations of rectiﬁed linear units are based on using a non - zero slope αi when zi < 0 : hi = g ( z α, ) i = max ( 0, zi ) + αi min ( 0, zi ). absolute value rectiﬁcation ﬁxes αi = −1 to obtain g ( z ) = | | z. it is used for object recognition from images (, ), where


##i ) + αi min ( 0, zi ). absolute value rectiﬁcation ﬁxes αi = −1 to obtain g ( z ) = | | z. it is used for object recognition from images (, ), where it makes sense to seek features that are jarrett et al. 2009 invariant under a polarity reversal of the input illumination. other generalizations of rectiﬁed linear units are more broadly applicable. a leaky relu (, maas et al. 2013 ) ﬁxes αi to a small value like 0. 01 while a parametric relu or prelu treats αi as a learnable parameter (, ). he et al. 2015 maxout units (, ) generalize rectiﬁed linear units goodfellow et al. 2013a further. instead of applying an element - wise function g ( z ), maxout units divide z into groups of k values. each maxout unit then outputs the maximum element of 193


chapter 6. deep feedforward networks one of these groups : g ( ) z i = max j∈g ( ) i zj ( 6. 37 ) where g ( ) i is the set of indices into the inputs for group i, { ( i −1 ) k + 1,..., ik }. this provides a way of learning a piecewise linear function that responds to multiple directions in the input space. x a maxout unit can learn a piecewise linear, convex function with up to k pieces. maxout units can thus be seen as learning the activation function itself rather than just the relationship between units. with large enough k, a maxout unit can learn to approximate any convex function with arbitrary ﬁdelity. in particular, a maxout layer with two pieces can learn to implement the same function of the input x as a traditional layer using the rectiﬁed linear activation function, absolute value rectiﬁcation function, or the leaky or parametric relu, or can learn to implement a totally [UNK] function altogether. the maxout layer will of course be parametrized [UNK] from any of these other layer types, so the learning dynamics will be [UNK] even in the cases where maxout learns to implement the same function of