there is no constraint that the factor must sum or integrate to 1 like a probability distribution. the probability of a conﬁguration of random variables is proportional to the product of all of these factors — assignments that result in larger factor values are more likely. of course, there is no guarantee that this product will sum to 1. we therefore divide by a normalizing constant z, deﬁned to be the sum or integral over all states of the product of the φ functions, in order to obtain a normalized probability distribution : p ( ) = x 1 z i φ ( ) i c ( ) i. ( 3. 55 ) see ﬁgure for an example of an undirected graph and the factorization of 3. 8 probability distributions it represents. keep in mind that these graphical representations of factorizations are a language for describing probability distributions. they are not mutually exclusive families of probability distributions. being directed or undirected is not a property of a probability distribution ; it is a property of a particular description of a 78


chapter 3. probability and information theory a c b e d figure 3. 8 : an undirected graphical model over random variablesa, b, c, d and e. this graph corresponds to probability distributions that can be factored as p,,,, ( a b c d e ) = 1 zφ ( 1 ) ( ) a b c,, φ ( 2 ) ( ) b d, φ ( 3 ) ( ) c e,. ( 3. 56 ) this graph allows us to quickly see some properties of the distribution. for example, a and c interact directly, but a and e interact only indirectly via c. probability distribution, but any probability distribution may be described in both ways. throughout parts and of this book, we will use structured probabilistic i ii models merely as a language to describe which direct probabilistic relationships [UNK] machine learning algorithms choose to represent. no further understanding of structured probabilistic models is needed until the discussion of research topics, in part, where we will explore structured probabilistic models in much greater iii detail. this chapter has reviewed the basic concepts of probability theory that are most relevant to deep learning. one more set of fundamental mathematical tools remains : numerical methods. 79

/home/ricoiban/GEMMA/mnlp_chatsplaining/RAG/Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville (z-lib.org).pdf


chapter 4 numerical computation machine learning algorithms usually require a high amount of numerical compu - tation. this typically refers to algorithms that solve mathematical problems by methods that update estimates of the solution via an iterative process, rather than analytically deriving a formula providing a symbolic expression for the correct so - lution. common operations include optimization ( ﬁnding the value of an argument that minimizes or maximizes a function ) and solving systems of linear equations. even just evaluating a mathematical function on a digital computer can be [UNK] when the function involves real numbers, which cannot be represented precisely using a ﬁnite amount of memory. 4. 1 overﬂow and underﬂow the fundamental [UNK] in performing continuous math on a digital computer is that we need to represent inﬁnitely many real numbers with a ﬁnite number of bit patterns. this means that for almost all real numbers, we incur some approximation error when we represent the number in the computer. in many cases, this is just rounding error. rounding error is problematic, especially when it compounds across many operations, and can cause algorithms that work in theory to fail in practice if they are not designed to minimize the accumulation of rounding error. one form of rounding error that is particularly devastating is underﬂow.


is problematic, especially when it compounds across many operations, and can cause algorithms that work in theory to fail in practice if they are not designed to minimize the accumulation of rounding error. one form of rounding error that is particularly devastating is underﬂow. underﬂow occurs when numbers near zero are rounded to zero. many functions behave qualitatively [UNK] when their argument is zero rather than a small positive number. for example, we usually want to avoid division by zero ( some 80


chapter 4. numerical computation software environments will raise exceptions when this occurs, others will return a result with a placeholder not - a - number value ) or taking the logarithm of zero ( this is usually treated as −∞, which then becomes not - a - number if it is used for many further arithmetic operations ). another highly damaging form of numerical error is overﬂow. overﬂow occurs when numbers with large magnitude are approximated as ∞or −∞. further arithmetic will usually change these inﬁnite values into not - a - number values. one example of a function that must be stabilized against underﬂow and overﬂow is the softmax function. the softmax function is often used to predict the probabilities associated with a multinoulli distribution. the softmax function is deﬁned to be softmax ( ) x i = exp ( xi ) n j = 1 exp ( xj ). ( 4. 1 ) consider what happens when all of the xi are equal to some constant c. analytically, we can see that all of the outputs should be equal to 1 n. numerically, this may not occur when c has large magnitude. if c is very negative, then exp ( c


xi are equal to some constant c. analytically, we can see that all of the outputs should be equal to 1 n. numerically, this may not occur when c has large magnitude. if c is very negative, then exp ( c ) will underﬂow. this means the denominator of the softmax will become 0, so the ﬁnal result is undeﬁned. when c is very large and positive, exp ( c ) will overﬂow, again resulting in the expression as a whole being undeﬁned. both of these [UNK] can be resolved by instead evaluating softmax ( z ) where z = x −maxi xi. simple algebra shows that the value of the softmax function is not changed analytically by adding or subtracting a scalar from the input vector. subtracting maxi xi results in the largest argument to exp being 0, which rules out the possibility of overﬂow. likewise, at least one term in the denominator has a value of 1, which rules out the possibility of underﬂow in the denominator leading to a division by zero. there is still one small problem. underﬂow in the numerator can still cause the expression as a whole to evaluate


value of 1, which rules out the possibility of underﬂow in the denominator leading to a division by zero. there is still one small problem. underﬂow in the numerator can still cause the expression as a whole to evaluate to zero. this means that if we implement log softmax ( x ) by ﬁrst running the softmax subroutine then passing the result to the log function, we could erroneously obtain −∞. instead, we must implement a separate function that calculates log softmax in a numerically stable way. the log softmax function can be stabilized using the same trick as we used to stabilize the function. softmax for the most part, we do not explicitly detail all of the numerical considerations involved in implementing the various algorithms described in this book. developers of low - level libraries should keep numerical issues in mind when implementing deep learning algorithms. most readers of this book can simply rely on low - level libraries that provide stable implementations. in some cases, it is possible to implement a new algorithm and have the new implementation automatically 81


chapter 4. numerical computation stabilized. theano (, ;, ) is an example bergstra et al. 2010 bastien et al. 2012 of a software package that automatically detects and stabilizes many common numerically unstable expressions that arise in the context of deep learning. 4. 2 poor conditioning conditioning refers to how rapidly a function changes with respect to small changes in its inputs. functions that change rapidly when their inputs are perturbed slightly can be problematic for scientiﬁc computation because rounding errors in the inputs can result in large changes in the output. consider the function f ( x ) = a−1x. when a ∈rn n × has an eigenvalue decomposition, its condition number is max i, j λi λj. ( 4. 2 ) this is the ratio of the magnitude of the largest and smallest eigenvalue. when this number is large, matrix inversion is particularly sensitive to error in the input. this sensitivity is an intrinsic property of the matrix itself, not the result of rounding error during matrix inversion. poorly conditioned matrices amplify pre - existing errors when we multiply by the true matrix inverse. in practice, the error will be compounded further by numerical errors in the inversion process itself. 4. 3


result of rounding error during matrix inversion. poorly conditioned matrices amplify pre - existing errors when we multiply by the true matrix inverse. in practice, the error will be compounded further by numerical errors in the inversion process itself. 4. 3 gradient - based optimization most deep learning algorithms involve optimization of some sort. optimization refers to the task of either minimizing or maximizing some function f ( x ) by altering x. we usually phrase most optimization problems in terms of minimizing f ( x ). maximization may be accomplished via a minimization algorithm by minimizing −f ( ) x. the function we want to minimize or maximize is called the objective func - tion or criterion. when we are minimizing it, we may also call it the cost function, loss function, or error function. in this book, we use these terms interchangeably, though some machine learning publications assign special meaning to some of these terms. we often denote the value that minimizes or maximizes a function with a superscript. for example, we might say ∗ x∗ = arg min ( ) f x. 82


chapter 4. numerical computation − − − − 2 0. 1 5. 1 0. 0 5 0 0 0 5 1 0 1 5 2 0...... x −2 0. −1 5. −1 0. −0 5. 0 0. 0 5. 1 0. 1 5. 2 0. global minimum at = 0. x since f ( ) = 0, gradient x descent halts here. for 0, we have x < f ( ) 0, x < so we can decrease by f moving rightward. for 0, we have x > f ( ) 0, x > so we can decrease by f moving leftward. f x ( ) = 1 2 x2 f ( ) = x x figure 4. 1 : an illustration of how the gradient descent algorithm uses the derivatives of a function can be used to follow the function downhill to a minimum. we assume the reader is already familiar with calculus, but provide a brief review of how calculus concepts relate to optimization here. suppose we have a function y = f ( x ), where both x and y are real numbers. the derivative of this function is denoted as f ( x ) or as dy dx. the derivative f ( x ) gives the slope


. suppose we have a function y = f ( x ), where both x and y are real numbers. the derivative of this function is denoted as f ( x ) or as dy dx. the derivative f ( x ) gives the slope of f ( x ) at the point x. in other words, it speciﬁes how to scale a small change in the input in order to obtain the corresponding change in the output : f x f x f ( + ) ≈ ( ) + ( ) x. the derivative is therefore useful for minimizing a function because it tells us how to change x in order to make a small improvement in y. for example, we know that f ( x − sign ( f ( x ) ) ) is less than f ( x ) for small enough. we can thus reduce f ( x ) by moving x in small steps with opposite sign of the derivative. this technique is called gradient descent ( cauchy 1847, ). see ﬁgure for an 4. 1 example of this technique. when f ( x ) = 0, the derivative provides no information about which direction to move. points where f ( x ) = 0 are known as critical points or stationary points. a local minimum is a point where f


1 example of this technique. when f ( x ) = 0, the derivative provides no information about which direction to move. points where f ( x ) = 0 are known as critical points or stationary points. a local minimum is a point where f ( x ) is lower than at all neighboring points, so it is no longer possible to decrease f ( x ) by making inﬁnitesimal steps. a local maximum is a point where f ( x ) is higher than at all neighboring points, 83


chapter 4. numerical computation minimum maximum saddle point figure 4. 2 : examples of each of the three types of critical points in 1 - d. a critical point is a point with zero slope. such a point can either be a local minimum, which is lower than the neighboring points, a local maximum, which is higher than the neighboring points, or a saddle point, which has neighbors that are both higher and lower than the point itself. so it is not possible to increase f ( x ) by making inﬁnitesimal steps. some critical points are neither maxima nor minima. these are known as saddle points. see ﬁgure for examples of each type of critical point. 4. 2 a point that obtains the absolute lowest value of f ( x ) is a global minimum. it is possible for there to be only one global minimum or multiple global minima of the function. it is also possible for there to be local minima that are not globally optimal. in the context of deep learning, we optimize functions that may have many local minima that are not optimal, and many saddle points surrounded by very ﬂat regions. all of this makes optimization very [UNK], especially when the input to the function is multidimensional. we


we optimize functions that may have many local minima that are not optimal, and many saddle points surrounded by very ﬂat regions. all of this makes optimization very [UNK], especially when the input to the function is multidimensional. we therefore usually settle for ﬁnding a value of f that is very low, but not necessarily minimal in any formal sense. see ﬁgure for an example. 4. 3 we often minimize functions that have multiple inputs : f : rn →r. for the concept of “ minimization ” to make sense, there must still be only one ( scalar ) output. for functions with multiple inputs, we must make use of the concept of partial derivatives. the partial derivative ∂ ∂xi f ( x ) measures how f changes as only the variable xi increases at point x. the gradient generalizes the notion of derivative to the case where the derivative is with respect to a vector : the gradient of f is the vector containing all of the partial derivatives, denoted ∇xf ( x ). element i of the gradient is the partial derivative of f with respect to xi. in multiple dimensions, 84


chapter 4. numerical computation x f x ( ) ideally, we would like to arrive at the global minimum, but this might not be possible. this local minimum performs nearly as well as the global one, so it is an acceptable halting point. this local minimum performs poorly and should be avoided. figure 4. 3 : optimization algorithms may fail to ﬁnd a global minimum when there are multiple local minima or plateaus present. in the context of deep learning, we generally accept such solutions even though they are not truly minimal, so long as they correspond to signiﬁcantly low values of the cost function. critical points are points where every element of the gradient is equal to zero. the directional derivative in direction ( a unit vector ) is the slope of the u function f in direction u. in other words, the directional derivative is the derivative of the function f ( x + αu ) with respect to α, evaluated at α = 0. using the chain rule, we can see that ∂ ∂αf α ( + x u ) evaluates to u∇xf α ( ) x when = 0. to minimize f, we would like to ﬁnd the direction in which f decreases the fastest. we can do this using the directional derivative : min


α ( + x u ) evaluates to u∇xf α ( ) x when = 0. to minimize f, we would like to ﬁnd the direction in which f decreases the fastest. we can do this using the directional derivative : min u u, u = 1 u∇xf ( ) x ( 4. 3 ) = min u u, u = 1 | | | | u 2 | | ∇xf ( ) x | | 2 cos θ ( 4. 4 ) where θ is the angle between u and the gradient. substituting in | | | | u 2 = 1 and ignoring factors that do not depend on u, this simpliﬁes to minu cos θ. this is minimized when u points in the opposite direction as the gradient. in other words, the gradient points directly uphill, and the negative gradient points directly downhill. we can decrease f by moving in the direction of the negative gradient. this is known as the or. method of steepest descent gradient descent steepest descent proposes a new point x = x −∇ xf ( ) x ( 4. 5 ) 85


chapter 4. numerical computation where is the learning rate, a positive scalar determining the size of the step. we can choose in several [UNK] ways. a popular approach is to set to a small constant. sometimes, we can solve for the step size that makes the directional derivative vanish. another approach is to evaluate f ( x −∇xf ( ) ) x for several values of and choose the one that results in the smallest objective function value. this last strategy is called a line search. steepest descent converges when every element of the gradient is zero ( or, in practice, very close to zero ). in some cases, we may be able to avoid running this iterative algorithm, and just jump directly to the critical point by solving the equation ∇xf ( ) = 0 x for. x although gradient descent is limited to optimization in continuous spaces, the general concept of repeatedly making a small move ( that is approximately the best small move ) towards better conﬁgurations can be generalized to discrete spaces. ascending an objective function of discrete parameters is called hill climbing (, ). russel and norvig 2003 4. 3. 1 beyond the gradient : jacobian and hessian matrices sometimes we need to ﬁnd all of the partial derivatives of a function


an objective function of discrete parameters is called hill climbing (, ). russel and norvig 2003 4. 3. 1 beyond the gradient : jacobian and hessian matrices sometimes we need to ﬁnd all of the partial derivatives of a function whose input and output are both vectors. the matrix containing all such partial derivatives is known as a jacobian matrix. speciﬁcally, if we have a function f : rm →rn, then the jacobian matrix j ∈rn m × of is deﬁned such that f ji, j = ∂ ∂xjf ( ) x i. we are also sometimes interested in a derivative of a derivative. this is known as a second derivative. for example, for a function f : rn →r, the derivative with respect to xi of the derivative of f with respect to xj is denoted as ∂2 ∂xi∂xj f. in a single dimension, we can denote d2 dx2 f by f ( x ). the second derivative tells us how the ﬁrst derivative will change as we vary the input. this is important because it tells us whether a gradient step will cause as much of an improvement as we would expect based on the gradient alone. we can think of the second derivative as


us how the ﬁrst derivative will change as we vary the input. this is important because it tells us whether a gradient step will cause as much of an improvement as we would expect based on the gradient alone. we can think of the second derivative as measuring curvature. suppose we have a quadratic function ( many functions that arise in practice are not quadratic but can be approximated well as quadratic, at least locally ). if such a function has a second derivative of zero, then there is no curvature. it is a perfectly ﬂat line, and its value can be predicted using only the gradient. if the gradient is, then we can make a step of size 1 along the negative gradient, and the cost function will decrease by. if the second derivative is negative, the function curves downward, so the cost function will actually decrease by more than. finally, if the second derivative is positive, the function curves upward, so the cost function can decrease by less than. see 86


chapter 4. numerical computation x f x ( ) negative curvature x f x ( ) no curvature x f x ( ) positive curvature figure 4. 4 : the second derivative determines the curvature of a function. here we show quadratic functions with various curvature. the dashed line indicates the value of the cost function we would expect based on the gradient information alone as we make a gradient step downhill. in the case of negative curvature, the cost function actually decreases faster than the gradient predicts. in the case of no curvature, the gradient predicts the decrease correctly. in the case of positive curvature, the function decreases slower than expected and eventually begins to increase, so steps that are too large can actually increase the function inadvertently. ﬁgure to see how [UNK] forms of curvature [UNK] the relationship between 4. 4 the value of the cost function predicted by the gradient and the true value. when our function has multiple input dimensions, there are many second derivatives. these derivatives can be collected together into a matrix called the hessian matrix. the hessian matrix is deﬁned such that h x ( ) ( f ) h x ( ) ( f ) i, j = ∂2 ∂xi∂xj f. ( ) x ( 4. 6 ) equivalently, the hessian


hessian matrix is deﬁned such that h x ( ) ( f ) h x ( ) ( f ) i, j = ∂2 ∂xi∂xj f. ( ) x ( 4. 6 ) equivalently, the hessian is the jacobian of the gradient. anywhere that the second partial derivatives are continuous, the [UNK] operators are commutative, i. e. their order can be swapped : ∂2 ∂xi∂xj f ( ) = x ∂2 ∂x j∂xi f. ( ) x ( 4. 7 ) this implies that hi, j = h j, i, so the hessian matrix is symmetric at such points. most of the functions we encounter in the context of deep learning have a symmetric hessian almost everywhere. because the hessian matrix is real and symmetric, we can decompose it into a set of real eigenvalues and an orthogonal basis of 87


chapter 4. numerical computation eigenvectors. the second derivative in a speciﬁc direction represented by a unit vector d is given by dhd. when d is an eigenvector of h, the second derivative in that direction is given by the corresponding eigenvalue. for other directions of d, the directional second derivative is a weighted average of all of the eigenvalues, with weights between 0 and 1, and eigenvectors that have smaller angle with d receiving more weight. the maximum eigenvalue determines the maximum second derivative and the minimum eigenvalue determines the minimum second derivative. the ( directional ) second derivative tells us how well we can expect a gradient descent step to perform. we can make a second - order taylor series approximation to the function around the current point f ( ) x x ( 0 ) : f f ( ) x ≈ ( x ( 0 ) ) + ( x x − ( 0 ) ) g + 1 2 ( x x − ( 0 ) ) h x x ( − ( 0 ) ). ( 4. 8 ) where g is the gradient and h is the hessian at x ( 0 ). if we use a learning rate of, then the new point x will be given by


) h x x ( − ( 0 ) ). ( 4. 8 ) where g is the gradient and h is the hessian at x ( 0 ). if we use a learning rate of, then the new point x will be given by x ( 0 ) −g. substituting this into our approximation, we obtain f ( x ( 0 ) − ≈ g ) f ( x ( 0 ) ) −g g + 1 22ghg. ( 4. 9 ) there are three terms here : the original value of the function, the expected improvement due to the slope of the function, and the correction we must apply to account for the curvature of the function. when this last term is too large, the gradient descent step can actually move uphill. when ghg is zero or negative, the taylor series approximation predicts that increasing forever will decrease f forever. in practice, the taylor series is unlikely to remain accurate for large, so one must resort to more heuristic choices of in this case. when ghg is positive, solving for the optimal step size that decreases the taylor series approximation of the function the most yields ∗ = gg ghg. ( 4. 10 ) in the worst case, when g aligns with the eigenvector


ghg is positive, solving for the optimal step size that decreases the taylor series approximation of the function the most yields ∗ = gg ghg. ( 4. 10 ) in the worst case, when g aligns with the eigenvector of h corresponding to the maximal eigenvalue λmax, then this optimal step size is given by 1 λ max. to the extent that the function we minimize can be approximated well by a quadratic function, the eigenvalues of the hessian thus determine the scale of the learning rate. the second derivative can be used to determine whether a critical point is a local maximum, a local minimum, or saddle point. recall that on a critical point, f ( x ) = 0. when the second derivative f ( x ) > 0, the ﬁrst derivative f ( x ) increases as we move to the right and decreases as we move to the left. this means 88


chapter 4. numerical computation f ( x − ) < 0 and f ( x + ) > 0 for small enough. in other words, as we move right, the slope begins to point uphill to the right, and as we move left, the slope begins to point uphill to the left. thus, when f ( x ) = 0 and f ( x ) > 0, we can conclude that x is a local minimum. similarly, when f ( x ) = 0 and f ( x ) < 0, we can conclude that x is a local maximum. this is known as the second derivative test. unfortunately, when f ( x ) = 0, the test is inconclusive. in this case x may be a saddle point, or a part of a ﬂat region. in multiple dimensions, we need to examine all of the second derivatives of the function. using the eigendecomposition of the hessian matrix, we can generalize the second derivative test to multiple dimensions. at a critical point, where ∇xf ( x ) = 0, we can examine the eigenvalues of the hessian to determine whether the critical point is a local maximum, local minimum, or saddle point. when the hessian is positive deﬁni


, where ∇xf ( x ) = 0, we can examine the eigenvalues of the hessian to determine whether the critical point is a local maximum, local minimum, or saddle point. when the hessian is positive deﬁnite ( all its eigenvalues are positive ), the point is a local minimum. this can be seen by observing that the directional second derivative in any direction must be positive, and making reference to the univariate second derivative test. likewise, when the hessian is negative deﬁnite ( all its eigenvalues are negative ), the point is a local maximum. in multiple dimensions, it is actually possible to ﬁnd positive evidence of saddle points in some cases. when at least one eigenvalue is positive and at least one eigenvalue is negative, we know that x is a local maximum on one cross section of f but a local minimum on another cross section. see ﬁgure for an example. finally, the multidimensional second 4. 5 derivative test can be inconclusive, just like the univariate version. the test is inconclusive whenever all of the non - zero eigenvalues have the same sign, but at least one eigen


##ensional second 4. 5 derivative test can be inconclusive, just like the univariate version. the test is inconclusive whenever all of the non - zero eigenvalues have the same sign, but at least one eigenvalue is zero. this is because the univariate second derivative test is inconclusive in the cross section corresponding to the zero eigenvalue. in multiple dimensions, there is a [UNK] second derivative for each direction at a single point. the condition number of the hessian at this point measures how much the second derivatives [UNK] from each other. when the hessian has a poor condition number, gradient descent performs poorly. this is because in one direction, the derivative increases rapidly, while in another direction, it increases slowly. gradient descent is unaware of this change in the derivative so it does not know that it needs to explore preferentially in the direction where the derivative remains negative for longer. it also makes it [UNK] to choose a good step size. the step size must be small enough to avoid overshooting the minimum and going uphill in directions with strong positive curvature. this usually means that the step size is too small to make signiﬁcant progress in other directions with less curvature. see ﬁgur


must be small enough to avoid overshooting the minimum and going uphill in directions with strong positive curvature. this usually means that the step size is too small to make signiﬁcant progress in other directions with less curvature. see ﬁgure for an example. 4. 6 this issue can be resolved by using information from the hessian matrix to guide 89


chapter 4. numerical computation figure 4. 5 : a saddle point containing both positive and negative curvature. the function in this example is f ( x ) = x2 1 −x2 2. along the axis corresponding to x1, the function curves upward. this axis is an eigenvector of the hessian and has a positive eigenvalue. along the axis corresponding to x2, the function curves downward. this direction is an eigenvector of the hessian with negative eigenvalue. the name “ saddle point ” derives from the saddle - like shape of this function. this is the quintessential example of a function with a saddle point. in more than one dimension, it is not necessary to have an eigenvalue of 0 in order to get a saddle point : it is only necessary to have both positive and negative eigenvalues. we can think of a saddle point with both signs of eigenvalues as being a local maximum within one cross section and a local minimum within another cross section. 90


chapter 4. numerical computation − − − 30 20 10 0 10 20 x1 −30 −20 −10 0 10 20 x2 figure 4. 6 : gradient descent fails to exploit the curvature information contained in the hessian matrix. here we use gradient descent to minimize a quadratic functionf ( x ) whose hessian matrix has condition number 5. this means that the direction of most curvature has ﬁve times more curvature than the direction of least curvature. in this case, the most curvature is in the direction [ 1, 1 ] and the least curvature is in the direction [ 1, −1 ]. the red lines indicate the path followed by gradient descent. this very elongated quadratic function resembles a long canyon. gradient descent wastes time repeatedly descending canyon walls, because they are the steepest feature. because the step size is somewhat too large, it has a tendency to overshoot the bottom of the function and thus needs to descend the opposite canyon wall on the next iteration. the large positive eigenvalue of the hessian corresponding to the eigenvector pointed in this direction indicates that this directional derivative is rapidly increasing, so an optimization algorithm based on the hessian could predict that the steepest direction is not actually a promising search direction in this context.


the hessian corresponding to the eigenvector pointed in this direction indicates that this directional derivative is rapidly increasing, so an optimization algorithm based on the hessian could predict that the steepest direction is not actually a promising search direction in this context. 91


chapter 4. numerical computation the search. the simplest method for doing so is known as newton ’ s method. newton ’ s method is based on using a second - order taylor series expansion to approximate near some point f ( ) x x ( 0 ) : f f ( ) x ≈ ( x ( 0 ) ) + ( x x − ( 0 ) ) ∇xf ( x ( 0 ) ) + 1 2 ( x x − ( 0 ) ) h x ( ) ( f ( 0 ) ) ( x x − ( 0 ) ). ( 4. 11 ) if we then solve for the critical point of this function, we obtain : x∗ = x ( 0 ) −h x ( ) ( f ( 0 ) ) −1∇xf ( x ( 0 ) ). ( 4. 12 ) when f is a positive deﬁnite quadratic function, newton ’ s method consists of applying equation once to jump to the minimum of the function directly. 4. 12 when f is not truly quadratic but can be locally approximated as a positive deﬁnite quadratic, newton ’ s method consists of applying equation multiple 4. 12 times. iteratively updating the approximation and jumping to the minimum of the approximation can reach the


quadratic but can be locally approximated as a positive deﬁnite quadratic, newton ’ s method consists of applying equation multiple 4. 12 times. iteratively updating the approximation and jumping to the minimum of the approximation can reach the critical point much faster than gradient descent would. this is a useful property near a local minimum, but it can be a harmful property near a saddle point. as discussed in section, newton ’ s method is 8. 2. 3 only appropriate when the nearby critical point is a minimum ( all the eigenvalues of the hessian are positive ), whereas gradient descent is not attracted to saddle points unless the gradient points toward them. optimization algorithms that use only the gradient, such as gradient descent, are called ﬁrst - order optimization algorithms. optimization algorithms that also use the hessian matrix, such as newton ’ s method, are called second - order optimization algorithms ( nocedal and wright 2006, ). the optimization algorithms employed in most contexts in this book are applicable to a wide variety of functions, but come with almost no guarantees. deep learning algorithms tend to lack guarantees because the family of functions used in deep learning is quite complicated. in many other ﬁelds, the dominant approach to optimization is to design optimization algorithms


a wide variety of functions, but come with almost no guarantees. deep learning algorithms tend to lack guarantees because the family of functions used in deep learning is quite complicated. in many other ﬁelds, the dominant approach to optimization is to design optimization algorithms for a limited family of functions. in the context of deep learning, we sometimes gain some guarantees by restrict - ing ourselves to functions that are either lipschitz continuous or have lipschitz continuous derivatives. a lipschitz continuous function is a function f whose rate of change is bounded by a lipschitz constant l : [UNK] [UNK] | − | ≤l | | − | | x, y, f ( ) x f ( ) y x y 2. ( 4. 13 ) this property is useful because it allows us to quantify our assumption that a small change in the input made by an algorithm such as gradient descent will have 92


chapter 4. numerical computation a small change in the output. lipschitz continuity is also a fairly weak constraint, and many optimization problems in deep learning can be made lipschitz continuous with relatively minor modiﬁcations. perhaps the most successful ﬁeld of specialized optimization is convex op - timization. convex optimization algorithms are able to provide many more guarantees by making stronger restrictions. convex optimization algorithms are applicable only to convex functions — functions for which the hessian is positive semideﬁnite everywhere. such functions are well - behaved because they lack saddle points and all of their local minima are necessarily global minima. however, most problems in deep learning are [UNK] to express in terms of convex optimization. convex optimization is used only as a subroutine of some deep learning algorithms. ideas from the analysis of convex optimization algorithms can be useful for proving the convergence of deep learning algorithms. however, in general, the importance of convex optimization is greatly diminished in the context of deep learning. for more information about convex optimization, see boyd and vandenberghe 2004 ( ) or rockafellar 1997 ( ). 4. 4 constrained optimization sometimes we wish not only to maximize or minimize a function f ( x ) over all possible values of x. instead we may wish


see boyd and vandenberghe 2004 ( ) or rockafellar 1997 ( ). 4. 4 constrained optimization sometimes we wish not only to maximize or minimize a function f ( x ) over all possible values of x. instead we may wish to ﬁnd the maximal or minimal value of f ( x ) for values of x in some set s. this is known as constrained optimization. points x that lie within the set s are called feasible points in constrained optimization terminology. we often wish to ﬁnd a solution that is small in some sense. a common approach in such situations is to impose a norm constraint, such as. | | | | ≤ x 1 one simple approach to constrained optimization is simply to modify gradient descent taking the constraint into account. if we use a small constant step size, we can make gradient descent steps, then project the result back into s. if we use a line search, we can search only over step sizes that yield new x points that are feasible, or we can project each point on the line back into the constraint region. when possible, this method can be made more [UNK] by projecting the gradient into the tangent space of the feasible region before taking the step or beginning the line search (, ). rosen 1960 a more sophisticated approach is


the line back into the constraint region. when possible, this method can be made more [UNK] by projecting the gradient into the tangent space of the feasible region before taking the step or beginning the line search (, ). rosen 1960 a more sophisticated approach is to design a [UNK], unconstrained opti - mization problem whose solution can be converted into a solution to the original, constrained optimization problem. for example, if we want to minimize f ( x ) for 93


chapter 4. numerical computation x ∈r2 with x constrained to have exactly unit l2 norm, we can instead minimize g ( θ ) = f ( [ cos sin θ, θ ] ) with respect to θ, then return [ cos sin θ, θ ] as the solution to the original problem. this approach requires creativity ; the transformation between optimization problems must be designed speciﬁcally for each case we encounter. the karush – kuhn – tucker ( kkt ) approach1 provides a very general so - lution to constrained optimization. with the kkt approach, we introduce a new function called the generalized lagrangian or generalized lagrange function. to deﬁne the lagrangian, we ﬁrst need to describe s in terms of equations and inequalities. we want a description of s in terms of m functions g ( ) i and n functions h ( ) j so that s = { | [UNK] x i, g ( ) i ( x ) = 0 and [UNK], h ( ) j ( x ) ≤0 }. the equations involving g ( ) i are called the equality constraints and the inequalities involving h ( ) j are called. inequality constraints we introduce new variables λi and


0 and [UNK], h ( ) j ( x ) ≤0 }. the equations involving g ( ) i are called the equality constraints and the inequalities involving h ( ) j are called. inequality constraints we introduce new variables λi andα j for each constraint, these are called the kkt multipliers. the generalized lagrangian is then deﬁned as l,, f ( x λ α ) = ( ) + x i λig ( ) i ( ) + x j αjh ( ) j ( ) x. ( 4. 14 ) we can now solve a constrained minimization problem using unconstrained optimization of the generalized lagrangian. observe that, so long as at least one feasible point exists and is not permitted to have value, then f ( ) x ∞ min x max λ max α α, ≥0 l,,. ( x λ α ) ( 4. 15 ) has the same optimal objective function value and set of optimal points as x min x∈s f. ( ) x ( 4. 16 ) this follows because any time the constraints are satisﬁed, max λ max α α, ≥0l,, f, ( x λ α ) = ( ) x (


min x∈s f. ( ) x ( 4. 16 ) this follows because any time the constraints are satisﬁed, max λ max α α, ≥0l,, f, ( x λ α ) = ( ) x ( 4. 17 ) while any time a constraint is violated, max λ max α α, ≥0 l,,. ( x λ α ) = ∞ ( 4. 18 ) 1the kkt approach generalizes the method of lagrange multipliers which allows equality constraints but not inequality constraints. 94


chapter 4. numerical computation these properties guarantee that no infeasible point can be optimal, and that the optimum within the feasible points is unchanged. to perform constrained maximization, we can construct the generalized la - grange function of, which leads to this optimization problem : −f ( ) x min x max λ max α α, ≥0−f ( ) + x i λig ( ) i ( ) + x j αjh ( ) j ( ) x. ( 4. 19 ) we may also convert this to a problem with maximization in the outer loop : max x min λ min α α, ≥0f ( ) + x i λig ( ) i ( ) x − j αjh ( ) j ( ) x. ( 4. 20 ) the sign of the term for the equality constraints does not matter ; we may deﬁne it with addition or subtraction as we wish, because the optimization is free to choose any sign for each λi. the inequality constraints are particularly interesting. we say that a constraint h ( ) i ( x ) is active if h ( ) i ( x∗ ) = 0. if a constraint is not active, then the solution to the problem found using that constraint would remain at least


particularly interesting. we say that a constraint h ( ) i ( x ) is active if h ( ) i ( x∗ ) = 0. if a constraint is not active, then the solution to the problem found using that constraint would remain at least a local solution if that constraint were removed. it is possible that an inactive constraint excludes other solutions. for example, a convex problem with an entire region of globally optimal points ( a wide, ﬂat, region of equal cost ) could have a subset of this region eliminated by constraints, or a non - convex problem could have better local stationary points excluded by a constraint that is inactive at convergence. however, the point found at convergence remains a stationary point whether or not the inactive constraints are included. because an inactive h ( ) i has negative value, then the solution to minx maxλ maxα α, ≥0 l ( x λ α,, ) will have αi = 0. we can thus observe that at the solution, α h ( x ) = 0. in other words, for all i, we know that at least one of the constraints αi ≥0 and h ( ) i ( x ) ≤0 must be active at the solution. to gain some intuition for this idea, we can say


. in other words, for all i, we know that at least one of the constraints αi ≥0 and h ( ) i ( x ) ≤0 must be active at the solution. to gain some intuition for this idea, we can say that either the solution is on the boundary imposed by the inequality and we must use its kkt multiplier to inﬂuence the solution to x, or the inequality has no inﬂuence on the solution and we represent this by zeroing out its kkt multiplier. a simple set of properties describe the optimal points of constrained opti - mization problems. these properties are called the karush - kuhn - tucker ( kkt ) conditions (, ; karush 1939 kuhn and tucker 1951, ). they are necessary conditions, but not always [UNK] conditions, for a point to be optimal. the conditions are : • the gradient of the generalized lagrangian is zero. • all constraints on both and the kkt multipliers are satisﬁed. x 95


chapter 4. numerical computation • the inequality constraints exhibit “ complementary slackness ” : α h ( x ) = 0. for more information about the kkt approach, see nocedal and wright 2006 ( ). 4. 5 example : linear least squares suppose we want to ﬁnd the value of that minimizes x f ( ) = x 1 2 | | − | | ax b 2 2. ( 4. 21 ) there are specialized linear algebra algorithms that can solve this problem [UNK]. however, we can also explore how to solve it using gradient - based optimization as a simple example of how these techniques work. first, we need to obtain the gradient : ∇x f ( ) = x a ( ) = ax b − aax a − b. ( 4. 22 ) we can then follow this gradient downhill, taking small steps. see algorithm 4. 1 for details. algorithm 4. 1 an algorithm to minimize f ( x ) = 1 2 | | − | | ax b 2 2 with respect to x using gradient descent, starting from an arbitrary value of. x set the step size ( ) and tolerance ( ) to small, positive numbers. δ while | | aax a − b | | 2 > δ do x x ← − aax a


using gradient descent, starting from an arbitrary value of. x set the step size ( ) and tolerance ( ) to small, positive numbers. δ while | | aax a − b | | 2 > δ do x x ← − aax a − b end while one can also solve this problem using newton ’ s method. in this case, because the true function is quadratic, the quadratic approximation employed by newton ’ s method is exact, and the algorithm converges to the global minimum in a single step. now suppose we wish to minimize the same function, but subject to the constraint xx ≤1. to do so, we introduce the lagrangian l, λ f λ ( x ) = ( ) + x xx −1. ( 4. 23 ) we can now solve the problem min x max λ, λ≥0l, λ. ( x ) ( 4. 24 ) 96


chapter 4. numerical computation the smallest - norm solution to the unconstrained least squares problem may be found using the moore - penrose pseudoinverse : x = a + b. if this point is feasible, then it is the solution to the constrained problem. otherwise, we must ﬁnd a solution where the constraint is active. by [UNK] the lagrangian with respect to, we obtain the equation x aax a − b x + 2λ = 0. ( 4. 25 ) this tells us that the solution will take the form x a = ( a i + 2λ ) −1ab. ( 4. 26 ) the magnitude of λ must be chosen such that the result obeys the constraint. we can ﬁnd this value by performing gradient ascent on. to do so, observe λ ∂ ∂λl, λ ( x ) = xx −1. ( 4. 27 ) when the norm of x exceeds 1, this derivative is positive, so to follow the derivative uphill and increase the lagrangian with respect to λ, we increase λ. because the [UNK] on the xx penalty has increased, solving the linear equation for x will now yield a solution with smaller norm. the process of solving the linear equation and adjusting λ continues until


lagrangian with respect to λ, we increase λ. because the [UNK] on the xx penalty has increased, solving the linear equation for x will now yield a solution with smaller norm. the process of solving the linear equation and adjusting λ continues until x has the correct norm and the derivative on λ is 0. this concludes the mathematical preliminaries that we use to develop machine learning algorithms. we are now ready to build and analyze some full - ﬂedged learning systems. 97


chapter 5 machine learning basics deep learning is a speciﬁc kind of machine learning. in order to understand deep learning well, one must have a solid understanding of the basic principles of machine learning. this chapter provides a brief course in the most important general principles that will be applied throughout the rest of the book. novice readers or those who want a wider perspective are encouraged to consider machine learning textbooks with a more comprehensive coverage of the fundamentals, such as murphy ( ) or ( ). if you are already familiar with machine learning basics, 2012 bishop 2006 feel free to skip ahead to section. that section covers some perspectives 5. 11 on traditional machine learning techniques that have strongly inﬂuenced the development of deep learning algorithms. we begin with a deﬁnition of what a learning algorithm is, and present an example : the linear regression algorithm. we then proceed to describe how the challenge of ﬁtting the training data [UNK] from the challenge of ﬁnding patterns that generalize to new data. most machine learning algorithms have settings called hyperparameters that must be determined external to the learning algorithm itself ; we discuss how to set these using additional data. machine learning is essentially a form of applied statistics with increased emphasis on the use of computers to statistically estimate complicated functions and


called hyperparameters that must be determined external to the learning algorithm itself ; we discuss how to set these using additional data. machine learning is essentially a form of applied statistics with increased emphasis on the use of computers to statistically estimate complicated functions and a decreased emphasis on proving conﬁdence intervals around these functions ; we therefore present the two central approaches to statistics : frequentist estimators and bayesian inference. most machine learning algorithms can be divided into the categories of supervised learning and unsupervised learning ; we describe these categories and give some examples of simple learning algorithms from each category. most deep learning algorithms are based on an optimization algorithm called stochastic gradient descent. we describe how to combine various algorithm components such as 98


chapter 5. machine learning basics an optimization algorithm, a cost function, a model, and a dataset to build a machine learning algorithm. finally, in section, we describe some of the 5. 11 factors that have limited the ability of traditional machine learning to generalize. these challenges have motivated the development of deep learning algorithms that overcome these obstacles. 5. 1 learning algorithms a machine learning algorithm is an algorithm that is able to learn from data. but what do we mean by learning? mitchell 1997 ( ) provides the deﬁnition “ a computer program is said to learn from experience e with respect to some class of tasks t and performance measure p, if its performance at tasks in t, as measured by p, improves with experience e. ” one can imagine a very wide variety of experiences e, tasks t, and performance measures p, and we do not make any attempt in this book to provide a formal deﬁnition of what may be used for each of these entities. instead, the following sections provide intuitive descriptions and examples of the [UNK] kinds of tasks, performance measures and experiences that can be used to construct machine learning algorithms. 5. 1. 1 the task, t machine learning allows us to tackle tasks that are too [UNK] to solve with ﬁxed programs


descriptions and examples of the [UNK] kinds of tasks, performance measures and experiences that can be used to construct machine learning algorithms. 5. 1. 1 the task, t machine learning allows us to tackle tasks that are too [UNK] to solve with ﬁxed programs written and designed by human beings. from a scientiﬁc and philosophical point of view, machine learning is interesting because developing our understanding of machine learning entails developing our understanding of the principles that underlie intelligence. in this relatively formal deﬁnition of the word “ task, ” the process of learning itself is not the task. learning is our means of attaining the ability to perform the task. for example, if we want a robot to be able to walk, then walking is the task. we could program the robot to learn to walk, or we could attempt to directly write a program that speciﬁes how to walk manually. machine learning tasks are usually described in terms of how the machine learning system should process an example. an example is a collection of features that have been quantitatively measured from some object or event that we want the machine learning system to process. we typically represent an example as a vector x ∈rn where each entry xi of the vector is another feature. for example, the features of


features that have been quantitatively measured from some object or event that we want the machine learning system to process. we typically represent an example as a vector x ∈rn where each entry xi of the vector is another feature. for example, the features of an image are usually the values of the pixels in the image. 99


chapter 5. machine learning basics many kinds of tasks can be solved with machine learning. some of the most common machine learning tasks include the following : • classiﬁcation : in this type of task, the computer program is asked to specify which of k categories some input belongs to. to solve this task, the learning algorithm is usually asked to produce a function f : rn → { 1,..., k }. when y = f ( x ), the model assigns an input described by vector x to a category identiﬁed by numeric code y. there are other variants of the classiﬁcation task, for example, where f outputs a probability distribution over classes. an example of a classiﬁcation task is object recognition, where the input is an image ( usually described as a set of pixel brightness values ), and the output is a numeric code identifying the object in the image. for example, the willow garage pr2 robot is able to act as a waiter that can recognize [UNK] kinds of drinks and deliver them to people on command ( good - fellow 2010 et al., ). modern object recognition is best accomplished with deep learning (, ;, ). object krizhevsky et al. 2012 [UNK] and sze


[UNK] kinds of drinks and deliver them to people on command ( good - fellow 2010 et al., ). modern object recognition is best accomplished with deep learning (, ;, ). object krizhevsky et al. 2012 [UNK] and szegedy 2015 recognition is the same basic technology that allows computers to recognize faces ( taigman 2014 et al., ), which can be used to automatically tag people in photo collections and allow computers to interact more naturally with their users. • classiﬁcation with missing inputs : classiﬁcation becomes more chal - lenging if the computer program is not guaranteed that every measurement in its input vector will always be provided. in order to solve the classiﬁcation task, the learning algorithm only has to deﬁne a function mapping single from a vector input to a categorical output. when some of the inputs may be missing, rather than providing a single classiﬁcation function, the learning algorithm must learn a of functions. each function corresponds to classi - set fying x with a [UNK] subset of its inputs missing. this kind of situation arises frequently in medical diagnosis, because many kinds of medical tests are expensive or invasive. one way to [UNK] deﬁne such a large set of functions is to learn


set fying x with a [UNK] subset of its inputs missing. this kind of situation arises frequently in medical diagnosis, because many kinds of medical tests are expensive or invasive. one way to [UNK] deﬁne such a large set of functions is to learn a probability distribution over all of the relevant variables, then solve the classiﬁcation task by marginalizing out the missing variables. with n input variables, we can now obtain all 2n [UNK] classiﬁ - cation functions needed for each possible set of missing inputs, but we only need to learn a single function describing the joint probability distribution. see goodfellow 2013b et al. ( ) for an example of a deep probabilistic model applied to such a task in this way. many of the other tasks described in this section can also be generalized to work with missing inputs ; classiﬁcation with missing inputs is just one example of what machine learning can do. 100


chapter 5. machine learning basics • regression : in this type of task, the computer program is asked to predict a numerical value given some input. to solve this task, the learning algorithm is asked to output a function f : rn →r. this type of task is similar to classiﬁcation, except that the format of output is [UNK]. an example of a regression task is the prediction of the expected claim amount that an insured person will make ( used to set insurance premiums ), or the prediction of future prices of securities. these kinds of predictions are also used for algorithmic trading. • transcription : in this type of task, the machine learning system is asked to observe a relatively unstructured representation of some kind of data and transcribe it into discrete, textual form. for example, in optical character recognition, the computer program is shown a photograph containing an image of text and is asked to return this text in the form of a sequence of characters ( e. g., in ascii or unicode format ). google street view uses deep learning to process address numbers in this way (, goodfellow et al. 2014d ). another example is speech recognition, where the computer program is provided an audio waveform and emits a sequence


unicode format ). google street view uses deep learning to process address numbers in this way (, goodfellow et al. 2014d ). another example is speech recognition, where the computer program is provided an audio waveform and emits a sequence of characters or word id codes describing the words that were spoken in the audio recording. deep learning is a crucial component of modern speech recognition systems used at major companies including microsoft, ibm and google (, hinton et al. 2012b ). • machine translation : in a machine translation task, the input already consists of a sequence of symbols in some language, and the computer program must convert this into a sequence of symbols in another language. this is commonly applied to natural languages, such as translating from english to french. deep learning has recently begun to have an important impact on this kind of task ( sutskever 2014 bahdanau 2015 et al., ; et al., ). • structured output : structured output tasks involve any task where the output is a vector ( or other data structure containing multiple values ) with important relationships between the [UNK] elements. this is a broad category, and subsumes the transcription and translation tasks described above, but also many other tasks. one example is parsing — mapping a natural


( or other data structure containing multiple values ) with important relationships between the [UNK] elements. this is a broad category, and subsumes the transcription and translation tasks described above, but also many other tasks. one example is parsing — mapping a natural language sentence into a tree that describes its grammatical structure and tagging nodes of the trees as being verbs, nouns, or adverbs, and so on. see ( ) for an example of deep learning applied to a parsing collobert 2011 task. another example is pixel - wise segmentation of images, where the computer program assigns every pixel in an image to a speciﬁc category. for 101


chapter 5. machine learning basics example, deep learning can be used to annotate the locations of roads in aerial photographs ( mnih and hinton 2010, ). the output need not have its form mirror the structure of the input as closely as in these annotation - style tasks. for example, in image captioning, the computer program observes an image and outputs a natural language sentence describing the image ( kiros et al. et al.,, ; 2014a b mao, ; 2015 vinyals 2015b donahue 2014 et al., ; et al., ; karpathy and li 2015 fang 2015 xu 2015, ; et al., ; et al., ). these tasks are called structured output tasks because the program must output several values that are all tightly inter - related. for example, the words produced by an image captioning program must form a valid sentence. • anomaly detection : in this type of task, the computer program sifts through a set of events or objects, and ﬂags some of them as being unusual or atypical. an example of an anomaly detection task is credit card fraud detection. by modeling your purchasing habits, a credit card company can detect misuse of your cards. if a thief


objects, and ﬂags some of them as being unusual or atypical. an example of an anomaly detection task is credit card fraud detection. by modeling your purchasing habits, a credit card company can detect misuse of your cards. if a thief steals your credit card or credit card information, the thief ’ s purchases will often come from a [UNK] probability distribution over purchase types than your own. the credit card company can prevent fraud by placing a hold on an account as soon as that card has been used for an uncharacteristic purchase. see ( ) for a chandola et al. 2009 survey of anomaly detection methods. • synthesis and sampling : in this type of task, the machine learning al - gorithm is asked to generate new examples that are similar to those in the training data. synthesis and sampling via machine learning can be useful for media applications where it can be expensive or boring for an artist to generate large volumes of content by hand. for example, video games can automatically generate textures for large objects or landscapes, rather than requiring an artist to manually label each pixel (, ). in some luo et al. 2013 cases, we want the sampling or synthesis procedure to generate some speciﬁc kind of output given the input. for example


or landscapes, rather than requiring an artist to manually label each pixel (, ). in some luo et al. 2013 cases, we want the sampling or synthesis procedure to generate some speciﬁc kind of output given the input. for example, in a speech synthesis task, we provide a written sentence and ask the program to emit an audio waveform containing a spoken version of that sentence. this is a kind of structured output task, but with the added qualiﬁcation that there is no single correct output for each input, and we explicitly desire a large amount of variation in the output, in order for the output to seem more natural and realistic. • imputation of missing values : in this type of task, the machine learning algorithm is given a new example x ∈rn, but with some entries xi of x missing. the algorithm must provide a prediction of the values of the missing entries. 102


chapter 5. machine learning basics • denoising : in this type of task, the machine learning algorithm is given in input a corrupted example [UNK] ∈rn obtained by an unknown corruption process from a clean example x ∈rn. the learner must predict the clean example x from its corrupted version [UNK], or more generally predict the conditional probability distribution p ( x | [UNK] ). • density estimation or probability mass function estimation : in the density estimation problem, the machine learning algorithm is asked to learn a function pmodel : rn →r, where pmodel ( x ) can be interpreted as a probability density function ( if x is continuous ) or a probability mass function ( if x is discrete ) on the space that the examples were drawn from. to do such a task well ( we will specify exactly what that means when we discuss performance measures p ), the algorithm needs to learn the structure of the data it has seen. it must know where examples cluster tightly and where they are unlikely to occur. most of the tasks described above require the learning algorithm to at least implicitly capture the structure of the probability distribution. density estimation allows us to explicitly capture that distribution. in principle, we can then perform computations on that distribution in order to solve the other tasks as well.


above require the learning algorithm to at least implicitly capture the structure of the probability distribution. density estimation allows us to explicitly capture that distribution. in principle, we can then perform computations on that distribution in order to solve the other tasks as well. for example, if we have performed density estimation to obtain a probability distribution p ( x ), we can use that distribution to solve the missing value imputation task. if a value xi is missing and all of the other values, denoted x−i, are given, then we know the distribution over it is given by p ( xi | x−i ). in practice, density estimation does not always allow us to solve all of these related tasks, because in many cases the required operations on p ( x ) are computationally intractable. of course, many other tasks and types of tasks are possible. the types of tasks we list here are intended only to provide examples of what machine learning can do, not to deﬁne a rigid taxonomy of tasks. 5. 1. 2 the performance measure, p in order to evaluate the abilities of a machine learning algorithm, we must design a quantitative measure of its performance. usually this performance measure p is speciﬁc to the task being carried out by the system. t


. 2 the performance measure, p in order to evaluate the abilities of a machine learning algorithm, we must design a quantitative measure of its performance. usually this performance measure p is speciﬁc to the task being carried out by the system. t for tasks such as classiﬁcation, classiﬁcation with missing inputs, and tran - scription, we often measure the accuracy of the model. accuracy is just the proportion of examples for which the model produces the correct output. we can 103


chapter 5. machine learning basics also obtain equivalent information by measuring the error rate, the proportion of examples for which the model produces an incorrect output. we often refer to the error rate as the expected 0 - 1 loss. the 0 - 1 loss on a particular example is 0 if it is correctly classiﬁed and 1 if it is not. for tasks such as density estimation, it does not make sense to measure accuracy, error rate, or any other kind of 0 - 1 loss. instead, we must use a [UNK] performance metric that gives the model a continuous - valued score for each example. the most common approach is to report the average log - probability the model assigns to some examples. usually we are interested in how well the machine learning algorithm performs on data that it has not seen before, since this determines how well it will work when deployed in the real world. we therefore evaluate these performance measures using a test set of data that is separate from the data used for training the machine learning system. the choice of performance measure may seem straightforward and objective, but it is often [UNK] to choose a performance measure that corresponds well to the desired behavior of the system. in some cases, this is because it is [UNK] to decide what should be measured. for example, when performing


measure may seem straightforward and objective, but it is often [UNK] to choose a performance measure that corresponds well to the desired behavior of the system. in some cases, this is because it is [UNK] to decide what should be measured. for example, when performing a transcription task, should we measure the accuracy of the system at transcribing entire sequences, or should we use a more ﬁne - grained performance measure that gives partial credit for getting some elements of the sequence correct? when performing a regression task, should we penalize the system more if it frequently makes medium - sized mistakes or if it rarely makes very large mistakes? these kinds of design choices depend on the application. in other cases, we know what quantity we would ideally like to measure, but measuring it is impractical. for example, this arises frequently in the context of density estimation. many of the best probabilistic models represent probability distributions only implicitly. computing the actual probability value assigned to a speciﬁc point in space in many such models is intractable. in these cases, one must design an alternative criterion that still corresponds to the design objectives, or design a good approximation to the desired criterion. 5. 1. 3 the experience, e machine learning algorithms can be broadly categorized as


models is intractable. in these cases, one must design an alternative criterion that still corresponds to the design objectives, or design a good approximation to the desired criterion. 5. 1. 3 the experience, e machine learning algorithms can be broadly categorized as unsupervised or supervised by what kind of experience they are allowed to have during the learning process. most of the learning algorithms in this book can be understood as being allowed to experience an entire dataset. a dataset is a collection of many examples, as 104


chapter 5. machine learning basics deﬁned in section. sometimes we will also call examples. 5. 1. 1 data points one of the oldest datasets studied by statisticians and machine learning re - searchers is the iris dataset (, ). it is a collection of measurements of fisher 1936 [UNK] parts of 150 iris plants. each individual plant corresponds to one example. the features within each example are the measurements of each of the parts of the plant : the sepal length, sepal width, petal length and petal width. the dataset also records which species each plant belonged to. three [UNK] species are represented in the dataset. unsupervised learning algorithms experience a dataset containing many features, then learn useful properties of the structure of this dataset. in the context of deep learning, we usually want to learn the entire probability distribution that generated a dataset, whether explicitly as in density estimation or implicitly for tasks like synthesis or denoising. some other unsupervised learning algorithms perform other roles, like clustering, which consists of dividing the dataset into clusters of similar examples. supervised learning algorithms experience a dataset containing features, but each example is also associated with a label or target. for example,


##vised learning algorithms perform other roles, like clustering, which consists of dividing the dataset into clusters of similar examples. supervised learning algorithms experience a dataset containing features, but each example is also associated with a label or target. for example, the iris dataset is annotated with the species of each iris plant. a supervised learning algorithm can study the iris dataset and learn to classify iris plants into three [UNK] species based on their measurements. roughly speaking, unsupervised learning involves observing several examples of a random vector x, and attempting to implicitly or explicitly learn the proba - bility distribution p ( x ), or some interesting properties of that distribution, while supervised learning involves observing several examples of a random vector x and an associated value or vector y, and learning to predict y from x, usually by estimating p ( y x | ). the term supervised learning originates from the view of the target y being provided by an instructor or teacher who shows the machine learning system what to do. in unsupervised learning, there is no instructor or teacher, and the algorithm must learn to make sense of the data without this guide. unsupervised learning and supervised learning are not formally deﬁned terms. the lines between


in unsupervised learning, there is no instructor or teacher, and the algorithm must learn to make sense of the data without this guide. unsupervised learning and supervised learning are not formally deﬁned terms. the lines between them are often blurred. many machine learning technologies can be used to perform both tasks. for example, the chain rule of probability states that for a vector x ∈rn, the joint distribution can be decomposed as p ( ) = x n i = 1 p ( xi | x1,..., xi−1 ). ( 5. 1 ) this decomposition means that we can solve the ostensibly unsupervised problem of modeling p ( x ) by splitting it into n supervised learning problems. alternatively, we 105


chapter 5. machine learning basics can solve the supervised learning problem of learning p ( y | x ) by using traditional unsupervised learning technologies to learn the joint distribution p ( x, y ) and inferring p y ( | x ) = p, y ( x ) yp, y ( x ). ( 5. 2 ) though unsupervised learning and supervised learning are not completely formal or distinct concepts, they do help to roughly categorize some of the things we do with machine learning algorithms. traditionally, people refer to regression, classiﬁcation and structured output problems as supervised learning. density estimation in support of other tasks is usually considered unsupervised learning. other variants of the learning paradigm are possible. for example, in semi - supervised learning, some examples include a supervision target but others do not. in multi - instance learning, an entire collection of examples is labeled as containing or not containing an example of a class, but the individual members of the collection are not labeled. for a recent example of multi - instance learning with deep models, see kotzias 2015 et al. ( ). some machine learning algorithms do not just experience a ﬁxed dataset. for example, reinforcement learning algorithms interact with an environment


labeled. for a recent example of multi - instance learning with deep models, see kotzias 2015 et al. ( ). some machine learning algorithms do not just experience a ﬁxed dataset. for example, reinforcement learning algorithms interact with an environment, so there is a feedback loop between the learning system and its experiences. such algorithms are beyond the scope of this book. please see ( ) sutton and barto 1998 or bertsekas and tsitsiklis 1996 ( ) for information about reinforcement learning, and ( ) for the deep learning approach to reinforcement learning. mnih et al. 2013 most machine learning algorithms simply experience a dataset. a dataset can be described in many ways. in all cases, a dataset is a collection of examples, which are in turn collections of features. one common way of describing a dataset is with a. a design design matrix matrix is a matrix containing a [UNK] example in each row. each column of the matrix corresponds to a [UNK] feature. for instance, the iris dataset contains 150 examples with four features for each example. this means we can represent the dataset with a design matrix x ∈r150 4 ×, where xi, 1 is the sepal length of plant i, xi, 2 is the sep


##set contains 150 examples with four features for each example. this means we can represent the dataset with a design matrix x ∈r150 4 ×, where xi, 1 is the sepal length of plant i, xi, 2 is the sepal width of plant i, etc. we will describe most of the learning algorithms in this book in terms of how they operate on design matrix datasets. of course, to describe a dataset as a design matrix, it must be possible to describe each example as a vector, and each of these vectors must be the same size. this is not always possible. for example, if you have a collection of photographs with [UNK] widths and heights, then [UNK] photographs will contain [UNK] numbers of pixels, so not all of the photographs may be described with the same length of vector. section and chapter describe how to handle [UNK] 9. 7 10 106


chapter 5. machine learning basics types of such heterogeneous data. in cases like these, rather than describing the dataset as a matrix with m rows, we will describe it as a set containing m elements : { x ( 1 ), x ( 2 ),..., x ( ) m }. this notation does not imply that any two example vectors x ( ) i and x ( ) j have the same size. in the case of supervised learning, the example contains a label or target as well as a collection of features. for example, if we want to use a learning algorithm to perform object recognition from photographs, we need to specify which object appears in each of the photos. we might do this with a numeric code, with 0 signifying a person, 1 signifying a car, 2 signifying a cat, etc. often when working with a dataset containing a design matrix of feature observations x, we also provide a vector of labels, with y yi providing the label for example. i of course, sometimes the label may be more than just a single number. for example, if we want to train a speech recognition system to transcribe entire sentences, then the label for each example sentence is a sequence of words. just as


i of course, sometimes the label may be more than just a single number. for example, if we want to train a speech recognition system to transcribe entire sentences, then the label for each example sentence is a sequence of words. just as there is no formal deﬁnition of supervised and unsupervised learning, there is no rigid taxonomy of datasets or experiences. the structures described here cover most cases, but it is always possible to design new ones for new applications. 5. 1. 4 example : linear regression our deﬁnition of a machine learning algorithm as an algorithm that is capable of improving a computer program ’ s performance at some task via experience is somewhat abstract. to make this more concrete, we present an example of a simple machine learning algorithm : linear regression. we will return to this example repeatedly as we introduce more machine learning concepts that help to understand its behavior. as the name implies, linear regression solves a regression problem. in other words, the goal is to build a system that can take a vector x ∈rn as input and predict the value of a scalar y ∈r as its output. in the case of linear regression, the output is a linear function of the input. let [UNK] be the value that


a system that can take a vector x ∈rn as input and predict the value of a scalar y ∈r as its output. in the case of linear regression, the output is a linear function of the input. let [UNK] be the value that our model predicts should take on. we deﬁne the output to be y [UNK] = wx ( 5. 3 ) where w ∈rn is a vector of. parameters parameters are values that control the behavior of the system. in this case, wi is the [UNK] that we multiply by feature xi before summing up the contributions from all the features. we can think of w as a set of weights that determine how each feature [UNK] the prediction. if a feature xi receives a positive weight wi, 107


chapter 5. machine learning basics then increasing the value of that feature increases the value of our prediction [UNK]. if a feature receives a negative weight, then increasing the value of that feature decreases the value of our prediction. if a feature ’ s weight is large in magnitude, then it has a large [UNK] on the prediction. if a feature ’ s weight is zero, it has no [UNK] on the prediction. we thus have a deﬁnition of our task t : to predict y from x by outputting [UNK] = wx. next we need a deﬁnition of our performance measure,. p suppose that we have a design matrix of m example inputs that we will not use for training, only for evaluating how well the model performs. we also have a vector of regression targets providing the correct value of y for each of these examples. because this dataset will only be used for evaluation, we call it the test set. we refer to the design matrix of inputs as x ( ) test and the vector of regression targets as y ( ) test. one way of measuring the performance of the model is to compute the mean squared error of the model on the test set. if [UNK] ( ) test gives the predictions of the model on the test set, then the mean


targets as y ( ) test. one way of measuring the performance of the model is to compute the mean squared error of the model on the test set. if [UNK] ( ) test gives the predictions of the model on the test set, then the mean squared error is given by msetest = 1 m i ( [UNK] ( ) test −y ( ) test ) 2 i. ( 5. 4 ) intuitively, one can see that this error measure decreases to 0 when [UNK] ( ) test = y ( ) test. we can also see that msetest = 1 m | | [UNK] ( ) test −y ( ) test | | 2 2, ( 5. 5 ) so the error increases whenever the euclidean distance between the predictions and the targets increases. to make a machine learning algorithm, we need to design an algorithm that will improve the weights w in a way that reduces msetest when the algorithm is allowed to gain experience by observing a training set ( x ( ) train, y ( ) train ). one intuitive way of doing this ( which we will justify later, in section ) is just to 5. 5. 1 minimize the mean squared error on the training set, msetrain. to minimize msetrain, we can simply solve for where its


intuitive way of doing this ( which we will justify later, in section ) is just to 5. 5. 1 minimize the mean squared error on the training set, msetrain. to minimize msetrain, we can simply solve for where its gradient is : 0 ∇wmsetrain = 0 ( 5. 6 ) ⇒∇w 1 m | | [UNK] ( ) train −y ( ) train | | 2 2 = 0 ( 5. 7 ) ⇒1 m∇w | | x ( ) train w y − ( ) train | | 2 2 = 0 ( 5. 8 ) 108


chapter 5. machine learning basics − − 1 0. 0 5 0 0 0 5 1 0.... x1 −3 −2 −1 0 1 2 3 y linear regression example 0 5 1 0 1 5... w1 0 20. 0 25. 0 30. 0 35. 0 40. 0 45. 0 50. 0 55. mse ( train ) optimization of w figure 5. 1 : a linear regression problem, with a training set consisting of ten data points, each containing one feature. because there is only one feature, the weight vector w contains only a single parameter to learn, w1. ( left ) observe that linear regression learns to set w1 such that the line y = w1x comes as close as possible to passing through all the training points. the plotted point indicates the value of ( right ) w1 found by the normal equations, which we can see minimizes the mean squared error on the training set. ⇒∇w x ( ) train w y − ( ) train x ( ) train w y − ( ) train = 0 ( 5. 9 ) ⇒∇w wx ( ) train x ( ) train w w −2 x ( ) train y ( ) train + y ( ) train y


( ) train x ( ) train w y − ( ) train = 0 ( 5. 9 ) ⇒∇w wx ( ) train x ( ) train w w −2 x ( ) train y ( ) train + y ( ) train y ( ) train = 0 ( 5. 10 ) ⇒2x ( ) train x ( ) train w x −2 ( ) train y ( ) train = 0 ( 5. 11 ) ⇒w = x ( ) train x ( ) train −1 x ( ) train y ( ) train ( 5. 12 ) the system of equations whose solution is given by equation is known as 5. 12 the normal equations. evaluating equation constitutes a simple learning 5. 12 algorithm. for an example of the linear regression learning algorithm in action, see ﬁgure. 5. 1 it is worth noting that the term linear regression is often used to refer to a slightly more sophisticated model with one additional parameter — an intercept term. in this model b [UNK] = wx + b ( 5. 13 ) so the mapping from parameters to predictions is still a linear function but the mapping from features to predictions is now an [UNK] function. this extension to [UNK] functions means that the plot of the model ’ s predictions still looks like a line, but


. 13 ) so the mapping from parameters to predictions is still a linear function but the mapping from features to predictions is now an [UNK] function. this extension to [UNK] functions means that the plot of the model ’ s predictions still looks like a line, but it need not pass through the origin. instead of adding the bias parameter 109


chapter 5. machine learning basics b, one can continue to use the model with only weights but augment x with an extra entry that is always set to. the weight corresponding to the extra entry 1 1 plays the role of the bias parameter. we will frequently use the term “ linear ” when referring to [UNK] functions throughout this book. the intercept term b is often called the bias parameter of the [UNK] transfor - mation. this terminology derives from the point of view that the output of the transformation is biased toward being b in the absence of any input. this term is [UNK] from the idea of a statistical bias, in which a statistical estimation algorithm ’ s expected estimate of a quantity is not equal to the true quantity. linear regression is of course an extremely simple and limited learning algorithm, but it provides an example of how a learning algorithm can work. in the subsequent sections we will describe some of the basic principles underlying learning algorithm design and demonstrate how these principles can be used to build more complicated learning algorithms. 5. 2 capacity, overﬁtting and underﬁtting the central challenge in machine learning is that we must perform well on new, previously unseen inputs — not just those on which our model was trained. the ability to perform well on previously unobserved inputs is called


, overﬁtting and underﬁtting the central challenge in machine learning is that we must perform well on new, previously unseen inputs — not just those on which our model was trained. the ability to perform well on previously unobserved inputs is called generalization. typically, when training a machine learning model, we have access to a training set, we can compute some error measure on the training set called the training error, and we reduce this training error. so far, what we have described is simply an optimization problem. what separates machine learning from optimization is that we want the generalization error, also called the test error, to be low as well. the generalization error is deﬁned as the expected value of the error on a new input. here the expectation is taken across [UNK] possible inputs, drawn from the distribution of inputs we expect the system to encounter in practice. we typically estimate the generalization error of a machine learning model by measuring its performance on a test set of examples that were collected separately from the training set. in our linear regression example, we trained the model by minimizing the training error, 1 m ( ) train | | x ( ) train w y − ( ) train | | 2 2, ( 5. 14 ) but we actually


set. in our linear regression example, we trained the model by minimizing the training error, 1 m ( ) train | | x ( ) train w y − ( ) train | | 2 2, ( 5. 14 ) but we actually care about the test error, 1 m ( ) test | | x ( ) test w y − ( ) test | | 2 2. how can we [UNK] performance on the test set when we get to observe only the 110


chapter 5. machine learning basics training set? the ﬁeld of statistical learning theory provides some answers. if the training and the test set are collected arbitrarily, there is indeed little we can do. if we are allowed to make some assumptions about how the training and test set are collected, then we can make some progress. the train and test data are generated by a probability distribution over datasets called the data generating process. we typically make a set of assumptions known collectively as the i. i. d. assumptions. these assumptions are that the examples in each dataset are independent from each other, and that the train set and test set are identically distributed, drawn from the same probability distribution as each other. this assumption allows us to describe the data gen - erating process with a probability distribution over a single example. the same distribution is then used to generate every train example and every test example. we call that shared underlying distribution the data generating distribution, denoted pdata. this probabilistic framework and the i. i. d. assumptions allow us to mathematically study the relationship between training error and test error. one immediate connection we can observe between the training and test error is that the expected training error of a randomly selected model is equal to the


the i. i. d. assumptions allow us to mathematically study the relationship between training error and test error. one immediate connection we can observe between the training and test error is that the expected training error of a randomly selected model is equal to the expected test error of that model. suppose we have a probability distribution p ( x, y ) and we sample from it repeatedly to generate the train set and the test set. for some ﬁxed value w, the expected training set error is exactly the same as the expected test set error, because both expectations are formed using the same dataset sampling process. the only [UNK] between the two conditions is the name we assign to the dataset we sample. of course, when we use a machine learning algorithm, we do not ﬁx the parameters ahead of time, then sample both datasets. we sample the training set, then use it to choose the parameters to reduce training set error, then sample the test set. under this process, the expected test error is greater than or equal to the expected value of training error. the factors determining how well a machine learning algorithm will perform are its ability to : 1. make the training error small. 2. make the gap between training and test error small. these two factors correspond to


equal to the expected value of training error. the factors determining how well a machine learning algorithm will perform are its ability to : 1. make the training error small. 2. make the gap between training and test error small. these two factors correspond to the two central challenges in machine learning : underﬁtting and overﬁtting. underﬁtting occurs when the model is not able to obtain a [UNK] low error value on the training set. overﬁtting occurs when the gap between the training error and test error is too large. we can control whether a model is more likely to overﬁt or underﬁt by altering its capacity. informally, a model ’ s capacity is its ability to ﬁt a wide variety of 111


chapter 5. machine learning basics functions. models with low capacity may struggle to ﬁt the training set. models with high capacity can overﬁt by memorizing properties of the training set that do not serve them well on the test set. one way to control the capacity of a learning algorithm is by choosing its hypothesis space, the set of functions that the learning algorithm is allowed to select as being the solution. for example, the linear regression algorithm has the set of all linear functions of its input as its hypothesis space. we can generalize linear regression to include polynomials, rather than just linear functions, in its hypothesis space. doing so increases the model ’ s capacity. a polynomial of degree one gives us the linear regression model with which we are already familiar, with prediction [UNK] b wx. = + ( 5. 15 ) by introducing x2 as another feature provided to the linear regression model, we can learn a model that is quadratic as a function of : x [UNK] b w = + 1x w + 2x2. ( 5. 16 ) though this model implements a quadratic function of its, the output is input still a linear function of the parameters, so we can still use the normal equations to train the model in closed form. we can continue


##x2. ( 5. 16 ) though this model implements a quadratic function of its, the output is input still a linear function of the parameters, so we can still use the normal equations to train the model in closed form. we can continue to add more powers of x as additional features, for example to obtain a polynomial of degree 9 : [UNK] b = + 9 i = 1 wixi. ( 5. 17 ) machine learning algorithms will generally perform best when their capacity is appropriate for the true complexity of the task they need to perform and the amount of training data they are provided with. models with [UNK] capacity are unable to solve complex tasks. models with high capacity can solve complex tasks, but when their capacity is higher than needed to solve the present task they may overﬁt. figure shows this principle in action. we compare a linear, quadratic 5. 2 and degree - 9 predictor attempting to ﬁt a problem where the true underlying function is quadratic. the linear function is unable to capture the curvature in the true underlying problem, so it underﬁts. the degree - 9 predictor is capable of representing the correct function, but it is also capable of representing inﬁnitely many other functions that pass exactly through the training points, because


in the true underlying problem, so it underﬁts. the degree - 9 predictor is capable of representing the correct function, but it is also capable of representing inﬁnitely many other functions that pass exactly through the training points, because we 112


chapter 5. machine learning basics have more parameters than training examples. we have little chance of choosing a solution that generalizes well when so many wildly [UNK] solutions exist. in this example, the quadratic model is perfectly matched to the true structure of the task so it generalizes well to new data. figure 5. 2 : we ﬁt three models to this example training set. the training data was generated synthetically, by randomly sampling x values and choosing y deterministically by evaluating a quadratic function. ( left ) a linear function ﬁt to the data [UNK] from underﬁtting — it cannot capture the curvature that is present in the data. a ( center ) quadratic function ﬁt to the data generalizes well to unseen points. it does not [UNK] from a signiﬁcant amount of overﬁtting or underﬁtting. a polynomial of degree 9 ﬁt to ( right ) the data [UNK] from overﬁtting. here we used the moore - penrose pseudoinverse to solve the underdetermined normal equations. the solution passes through all of the training points exactly, but we have not been lucky enough for it to extract the correct structure. it now has a deep valley in between two training points that does not appear in the true


##determined normal equations. the solution passes through all of the training points exactly, but we have not been lucky enough for it to extract the correct structure. it now has a deep valley in between two training points that does not appear in the true underlying function. it also increases sharply on the left side of the data, while the true function decreases in this area. so far we have described only one way of changing a model ’ s capacity : by changing the number of input features it has, and simultaneously adding new parameters associated with those features. there are in fact many ways of changing a model ’ s capacity. capacity is not determined only by the choice of model. the model speciﬁes which family of functions the learning algorithm can choose from when varying the parameters in order to reduce a training objective. this is called the representational capacity of the model. in many cases, ﬁnding the best function within this family is a very [UNK] optimization problem. in practice, the learning algorithm does not actually ﬁnd the best function, but merely one that signiﬁcantly reduces the training error. these additional limitations, such as 113


chapter 5. machine learning basics the imperfection of the optimization algorithm, mean that the learning algorithm ’ s [UNK] capacity may be less than the representational capacity of the model family. our modern ideas about improving the generalization of machine learning models are reﬁnements of thought dating back to philosophers at least as early as ptolemy. many early scholars invoke a principle of parsimony that is now most widely known as occam ’ s razor ( c. 1287 - 1347 ). this principle states that among competing hypotheses that explain known observations equally well, one should choose the “ simplest ” one. this idea was formalized and made more precise in the 20th century by the founders of statistical learning theory ( vapnik and chervonenkis 1971 vapnik 1982 blumer 1989 vapnik 1995, ;, ; et al., ;, ). statistical learning theory provides various means of quantifying model capacity. among these, the most well - known is the vapnik - chervonenkis dimension, or vc dimension. the vc dimension measures the capacity of a binary classiﬁer. the vc dimension is deﬁned as being the largest possible value of m for which there exists a training set of m [UNK]


- chervonenkis dimension, or vc dimension. the vc dimension measures the capacity of a binary classiﬁer. the vc dimension is deﬁned as being the largest possible value of m for which there exists a training set of m [UNK] x points that the classiﬁer can label arbitrarily. quantifying the capacity of the model allows statistical learning theory to make quantitative predictions. the most important results in statistical learning theory show that the discrepancy between training error and generalization error is bounded from above by a quantity that grows as the model capacity grows but shrinks as the number of training examples increases ( vapnik and chervonenkis, 1971 vapnik 1982 blumer 1989 vapnik 1995 ;, ; et al., ;, ). these bounds provide intellectual justiﬁcation that machine learning algorithms can work, but they are rarely used in practice when working with deep learning algorithms. this is in part because the bounds are often quite loose and in part because it can be quite [UNK] to determine the capacity of deep learning algorithms. the problem of determining the capacity of a deep learning model is especially [UNK] because the [UNK] capacity is limited by the capabilities of the optimization algorithm, and we have little theoretical understanding of the very


can be quite [UNK] to determine the capacity of deep learning algorithms. the problem of determining the capacity of a deep learning model is especially [UNK] because the [UNK] capacity is limited by the capabilities of the optimization algorithm, and we have little theoretical understanding of the very general non - convex optimization problems involved in deep learning. we must remember that while simpler functions are more likely to generalize ( to have a small gap between training and test error ) we must still choose a [UNK] complex hypothesis to achieve low training error. typically, training error decreases until it asymptotes to the minimum possible error value as model capacity increases ( assuming the error measure has a minimum value ). typically, generalization error has a u - shaped curve as a function of model capacity. this is illustrated in ﬁgure. 5. 3 to reach the most extreme case of arbitrarily high capacity, we introduce 114


chapter 5. machine learning basics 0 optimal capacity capacity error underﬁtting zone overﬁtting zone generalization gap training error generalization error figure 5. 3 : typical relationship between capacity and error. training and test error behave [UNK]. at the left end of the graph, training error and generalization error are both high. this is the underﬁtting regime. as we increase capacity, training error decreases, but the gap between training and generalization error increases. eventually, the size of this gap outweighs the decrease in training error, and we enter theoverﬁtting regime, where capacity is too large, above the optimal capacity. the concept of non - parametric models. so far, we have seen only parametric models, such as linear regression. parametric models learn a function described by a parameter vector whose size is ﬁnite and ﬁxed before any data is observed. non - parametric models have no such limitation. sometimes, non - parametric models are just theoretical abstractions ( such as an algorithm that searches over all possible probability distributions ) that cannot be implemented in practice. however, we can also design practical non - parametric models by making their complexity a function of the training set size. one example of such an algorithm is nearest neighbor regression.


that searches over all possible probability distributions ) that cannot be implemented in practice. however, we can also design practical non - parametric models by making their complexity a function of the training set size. one example of such an algorithm is nearest neighbor regression. unlike linear regression, which has a ﬁxed - length vector of weights, the nearest neighbor regression model simply stores the x and y from the training set. when asked to classify a test point x, the model looks up the nearest entry in the training set and returns the associated regression target. in other words, [UNK] = yi where i = arg min | | xi, : − | | x 2 2. the algorithm can also be generalized to distance metrics other than the l2 norm, such as learned distance metrics (, ). if the algorithm is goldberger et al. 2005 allowed to break ties by averaging the yi values for all xi, : that are tied for nearest, then this algorithm is able to achieve the minimum possible training error ( which might be greater than zero, if two identical inputs are associated with [UNK] outputs ) on any regression dataset. finally, we can also create a non - parametric learning algorithm by wrapping a 115


chapter 5. machine learning basics parametric learning algorithm inside another algorithm that increases the number of parameters as needed. for example, we could imagine an outer loop of learning that changes the degree of the polynomial learned by linear regression on top of a polynomial expansion of the input. the ideal model is an oracle that simply knows the true probability distribution that generates the data. even such a model will still incur some error on many problems, because there may still be some noise in the distribution. in the case of supervised learning, the mapping from x to y may be inherently stochastic, or y may be a deterministic function that involves other variables besides those included in x. the error incurred by an oracle making predictions from the true distribution is called the p, y ( x ) bayes error. training and generalization error vary as the size of the training set varies. expected generalization error can never increase as the number of training examples increases. for non - parametric models, more data yields better generalization until the best possible error is achieved. any ﬁxed parametric model with less than optimal capacity will asymptote to an error value that exceeds the bayes error. see ﬁgure for an illustration. note that it is possible for the model to