best possible error is achieved. any ﬁxed parametric model with less than optimal capacity will asymptote to an error value that exceeds the bayes error. see ﬁgure for an illustration. note that it is possible for the model to have optimal 5. 4 capacity and yet still have a large gap between training and generalization error. in this situation, we may be able to reduce this gap by gathering more training examples. 5. 2. 1 the no free lunch theorem learning theory claims that a machine learning algorithm can generalize well from a ﬁnite training set of examples. this seems to contradict some basic principles of logic. inductive reasoning, or inferring general rules from a limited set of examples, is not logically valid. to logically infer a rule describing every member of a set, one must have information about every member of that set. in part, machine learning avoids this problem by [UNK] only probabilistic rules, rather than the entirely certain rules used in purely logical reasoning. machine learning promises to ﬁnd rules that are probably most correct about members of the set they concern. unfortunately, even this does not resolve the entire problem. the no free lunch theorem for machine learning ( wolpert 1996, ) states


logical reasoning. machine learning promises to ﬁnd rules that are probably most correct about members of the set they concern. unfortunately, even this does not resolve the entire problem. the no free lunch theorem for machine learning ( wolpert 1996, ) states that, averaged over all possible data generating distributions, every classiﬁcation algorithm has the same error rate when classifying previously unobserved points. in other words, in some sense, no machine learning algorithm is universally any better than any other. the most sophisticated algorithm we can conceive of has the same average 116

/home/ricoiban/GEMMA/mnlp_chatsplaining/RAG/Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville (z-lib.org).pdf


chapter 5. machine learning basics figure 5. 4 : the [UNK] of the training dataset size on the train and test error, as well as on the optimal model capacity. we constructed a synthetic regression problem based on adding a moderate amount of noise to a degree - 5 polynomial, generated a single test set, and then generated several [UNK] sizes of training set. for each size, we generated 40 [UNK] training sets in order to plot error bars showing 95 percent conﬁdence intervals. ( top ) the mse on the training and test set for two [UNK] models : a quadratic model, and a model with degree chosen to minimize the test error. both are ﬁt in closed form. for the quadratic model, the training error increases as the size of the training set increases. this is because larger datasets are harder to ﬁt. simultaneously, the test error decreases, because fewer incorrect hypotheses are consistent with the training data. the quadratic model does not have enough capacity to solve the task, so its test error asymptotes to a high value. the test error at optimal capacity asymptotes to the bayes error. the training error can fall below the bayes error, due to the ability of the training algorithm


so its test error asymptotes to a high value. the test error at optimal capacity asymptotes to the bayes error. the training error can fall below the bayes error, due to the ability of the training algorithm to memorize speciﬁc instances of the training set. as the training size increases to inﬁnity, the training error of any ﬁxed - capacity model ( here, the quadratic model ) must rise to at least the bayes error. as the training set size increases, the optimal capacity ( bottom ) ( shown here as the degree of the optimal polynomial regressor ) increases. the optimal capacity plateaus after reaching [UNK] complexity to solve the task. 117


chapter 5. machine learning basics performance ( over all possible tasks ) as merely predicting that every point belongs to the same class. fortunately, these results hold only when we average over possible data all generating distributions. if we make assumptions about the kinds of probability distributions we encounter in real - world applications, then we can design learning algorithms that perform well on these distributions. this means that the goal of machine learning research is not to seek a universal learning algorithm or the absolute best learning algorithm. instead, our goal is to understand what kinds of distributions are relevant to the “ real world ” that an ai agent experiences, and what kinds of machine learning algorithms perform well on data drawn from the kinds of data generating distributions we care about. 5. 2. 2 regularization the no free lunch theorem implies that we must design our machine learning algorithms to perform well on a speciﬁc task. we do so by building a set of preferences into the learning algorithm. when these preferences are aligned with the learning problems we ask the algorithm to solve, it performs better. so far, the only method of modifying a learning algorithm that we have discussed concretely is to increase or decrease the model ’ s representational capacity by adding or removing functions from the hypothesis space of solutions the learning algorithm is able to


it performs better. so far, the only method of modifying a learning algorithm that we have discussed concretely is to increase or decrease the model ’ s representational capacity by adding or removing functions from the hypothesis space of solutions the learning algorithm is able to choose. we gave the speciﬁc example of increasing or decreasing the degree of a polynomial for a regression problem. the view we have described so far is oversimpliﬁed. the behavior of our algorithm is strongly [UNK] not just by how large we make the set of functions allowed in its hypothesis space, but by the speciﬁc identity of those functions. the learning algorithm we have studied so far, linear regression, has a hypothesis space consisting of the set of linear functions of its input. these linear functions can be very useful for problems where the relationship between inputs and outputs truly is close to linear. they are less useful for problems that behave in a very nonlinear fashion. for example, linear regression would not perform very well if we tried to use it to predict sin ( x ) from x. we can thus control the performance of our algorithms by choosing what kind of functions we allow them to draw solutions from, as well as by controlling the amount of these functions. we can also give a learning algorithm a


predict sin ( x ) from x. we can thus control the performance of our algorithms by choosing what kind of functions we allow them to draw solutions from, as well as by controlling the amount of these functions. we can also give a learning algorithm a preference for one solution in its hypothesis space to another. this means that both functions are eligible, but one is preferred. the unpreferred solution will be chosen only if it ﬁts the training 118


chapter 5. machine learning basics data signiﬁcantly better than the preferred solution. for example, we can modify the training criterion for linear regression to include weight decay. to perform linear regression with weight decay, we minimize a sum comprising both the mean squared error on the training and a criterion j ( w ) that expresses a preference for the weights to have smaller squaredl2 norm. speciﬁcally, j ( ) = w msetrain + λww, ( 5. 18 ) where λ is a value chosen ahead of time that controls the strength of our preference for smaller weights. when λ = 0, we impose no preference, and larger λ forces the weights to become smaller. minimizing j ( w ) results in a choice of weights that make a [UNK] ﬁtting the training data and being small. this gives us solutions that have a smaller slope, or put weight on fewer of the features. as an example of how we can control a model ’ s tendency to overﬁt or underﬁt via weight decay, we can train a high - degree polynomial regression model with [UNK] values of. see ﬁgure for the results. λ 5. 5 figure 5. 5 : we ﬁt a high - degree polynomial regression model to our


##ﬁt via weight decay, we can train a high - degree polynomial regression model with [UNK] values of. see ﬁgure for the results. λ 5. 5 figure 5. 5 : we ﬁt a high - degree polynomial regression model to our example training set from ﬁgure. the true function is quadratic, but here we use only models with degree 9. 5. 2 we vary the amount of weight decay to prevent these high - degree models from overﬁtting. ( left ) with very large λ, we can force the model to learn a function with no slope at all. this underﬁts because it can only represent a constant function. with a ( center ) medium value of, the learning algorithm recovers a curve with the right general shape. λ even though the model is capable of representing functions with much more complicated shape, weight decay has encouraged it to use a simpler function described by smaller [UNK]. with weight decay approaching zero ( i. e., using the moore - penrose ( right ) pseudoinverse to solve the underdetermined problem with minimal regularization ), the degree - 9 polynomial overﬁts signiﬁcantly, as we saw in ﬁgure. 5. 2 119


chapter 5. machine learning basics more generally, we can regularize a model that learns a function f ( x ; θ ) by adding a penalty called a regularizer to the cost function. in the case of weight decay, the regularizer is ω ( w ) = w w. in chapter, we will see that many other 7 regularizers are possible. expressing preferences for one function over another is a more general way of controlling a model ’ s capacity than including or excluding members from the hypothesis space. we can think of excluding a function from a hypothesis space as expressing an inﬁnitely strong preference against that function. in our weight decay example, we expressed our preference for linear functions deﬁned with smaller weights explicitly, via an extra term in the criterion we minimize. there are many other ways of expressing preferences for [UNK] solutions, both implicitly and explicitly. together, these [UNK] approaches are known as regularization. regularization is any modiﬁcation we make to a learning algorithm that is intended to reduce its generalization error but not its training error. regularization is one of the central concerns of the ﬁeld of machine learning, rivaled in its importance only by optimization. the no free lunch theorem has made it clear that there is no best machine


its generalization error but not its training error. regularization is one of the central concerns of the ﬁeld of machine learning, rivaled in its importance only by optimization. the no free lunch theorem has made it clear that there is no best machine learning algorithm, and, in particular, no best form of regularization. instead we must choose a form of regularization that is well - suited to the particular task we want to solve. the philosophy of deep learning in general and this book in particular is that a very wide range of tasks ( such as all of the intellectual tasks that people can do ) may all be solved [UNK] using very general - purpose forms of regularization. 5. 3 hyperparameters and validation sets most machine learning algorithms have several settings that we can use to control the behavior of the learning algorithm. these settings are called hyperparame - ters. the values of hyperparameters are not adapted by the learning algorithm itself ( though we can design a nested learning procedure where one learning algorithm learns the best hyperparameters for another learning algorithm ). in the polynomial regression example we saw in ﬁgure, there is a single 5. 2 hyperparameter : the degree of the polynomial, which acts as a capacity hyper - parameter


learns the best hyperparameters for another learning algorithm ). in the polynomial regression example we saw in ﬁgure, there is a single 5. 2 hyperparameter : the degree of the polynomial, which acts as a capacity hyper - parameter. the λ value used to control the strength of weight decay is another example of a hyperparameter. sometimes a setting is chosen to be a hyperparameter that the learning al - gorithm does not learn because it is [UNK] to optimize. more frequently, the 120


chapter 5. machine learning basics setting must be a hyperparameter because it is not appropriate to learn that hyperparameter on the training set. this applies to all hyperparameters that control model capacity. if learned on the training set, such hyperparameters would always choose the maximum possible model capacity, resulting in overﬁtting ( refer to ﬁgure ). for example, we can always ﬁt the training set better with a higher 5. 3 degree polynomial and a weight decay setting of λ = 0 than we could with a lower degree polynomial and a positive weight decay setting. to solve this problem, we need a validation set of examples that the training algorithm does not observe. earlier we discussed how a held - out test set, composed of examples coming from the same distribution as the training set, can be used to estimate the generalization error of a learner, after the learning process has completed. it is important that the test examples are not used in any way to make choices about the model, including its hyperparameters. for this reason, no example from the test set can be used in the validation set. therefore, we always construct the validation set from the training data. speciﬁcally, we split the training data into two


its hyperparameters. for this reason, no example from the test set can be used in the validation set. therefore, we always construct the validation set from the training data. speciﬁcally, we split the training data into two disjoint subsets. one of these subsets is used to learn the parameters. the other subset is our validation set, used to estimate the generalization error during or after training, allowing for the hyperparameters to be updated accordingly. the subset of data used to learn the parameters is still typically called the training set, even though this may be confused with the larger pool of data used for the entire training process. the subset of data used to guide the selection of hyperparameters is called the validation set. typically, one uses about 80 % of the training data for training and 20 % for validation. since the validation set is used to “ train ” the hyperparameters, the validation set error will underestimate the generalization error, though typically by a smaller amount than the training error. after all hyperparameter optimization is complete, the generalization error may be estimated using the test set. in practice, when the same test set has been used repeatedly to evaluate performance of [UNK] algorithms over


typically by a smaller amount than the training error. after all hyperparameter optimization is complete, the generalization error may be estimated using the test set. in practice, when the same test set has been used repeatedly to evaluate performance of [UNK] algorithms over many years, and especially if we consider all the attempts from the scientiﬁc community at beating the reported state - of - the - art performance on that test set, we end up having optimistic evaluations with the test set as well. benchmarks can thus become stale and then do not reﬂect the true ﬁeld performance of a trained system. thankfully, the community tends to move on to new ( and usually more ambitious and larger ) benchmark datasets. 121


chapter 5. machine learning basics 5. 3. 1 cross - validation dividing the dataset into a ﬁxed training set and a ﬁxed test set can be problematic if it results in the test set being small. a small test set implies statistical uncertainty around the estimated average test error, making it [UNK] to claim that algorithm a works better than algorithm on the given task. b when the dataset has hundreds of thousands of examples or more, this is not a serious issue. when the dataset is too small, are alternative procedures enable one to use all of the examples in the estimation of the mean test error, at the price of increased computational cost. these procedures are based on the idea of repeating the training and testing computation on [UNK] randomly chosen subsets or splits of the original dataset. the most common of these is the k - fold cross - validation procedure, shown in algorithm, in which a partition of the dataset is formed by 5. 1 splitting it into k non - overlapping subsets. the test error may then be estimated by taking the average test error across k trials. on trial i, the i - th subset of the data is used as the test set and the rest of the data is used as the training set. one problem is that there exist


then be estimated by taking the average test error across k trials. on trial i, the i - th subset of the data is used as the test set and the rest of the data is used as the training set. one problem is that there exist no unbiased estimators of the variance of such average error estimators ( bengio and grandvalet 2004, ), but approximations are typically used. 5. 4 estimators, bias and variance the ﬁeld of statistics gives us many tools that can be used to achieve the machine learning goal of solving a task not only on the training set but also to generalize. foundational concepts such as parameter estimation, bias and variance are useful to formally characterize notions of generalization, underﬁtting and overﬁtting. 5. 4. 1 point estimation point estimation is the attempt to provide the single “ best ” prediction of some quantity of interest. in general the quantity of interest can be a single parameter or a vector of parameters in some parametric model, such as the weights in our linear regression example in section, but it can also be a whole function. 5. 1. 4 in order to distinguish estimates of parameters from their true value, our convention will be to denote a point estimate of


model, such as the weights in our linear regression example in section, but it can also be a whole function. 5. 1. 4 in order to distinguish estimates of parameters from their true value, our convention will be to denote a point estimate of a parameter by θ [UNK]. let { x ( 1 ),..., x ( ) m } be a set of m independent and identically distributed 122


chapter 5. machine learning basics algorithm 5. 1 the k - fold cross - validation algorithm. it can be used to estimate generalization error of a learning algorithm a when the given dataset d is too small for a simple train / test or train / valid split to yield accurate estimation of generalization error, because the mean of a loss l on a small test set may have too high variance. the dataset d contains as elements the abstract examples z ( ) i ( for the i - th example ), which could stand for an ( input, target ) pair z ( ) i = ( x ( ) i, y ( ) i ) in the case of supervised learning, or for just an input z ( ) i = x ( ) i in the case of unsupervised learning. the algorithm returns the vector of errors e for each example in d, whose mean is the estimated generalization error. the errors on individual examples can be used to compute a conﬁdence interval around the mean ( equation ). while these conﬁdence intervals are not well - justiﬁed after the 5. 47 use of cross - validation, it is still common practice to use them to declare that algorithm a is better than algorithm b only if the con


). while these conﬁdence intervals are not well - justiﬁed after the 5. 47 use of cross - validation, it is still common practice to use them to declare that algorithm a is better than algorithm b only if the conﬁdence interval of the error of algorithm a lies below and does not intersect the conﬁdence interval of algorithm b. deﬁne kfoldxv ( ) : d, a, l, k require : d, the given dataset, with elements z ( ) i require : a, the learning algorithm, seen as a function that takes a dataset as input and outputs a learned function require : l, the loss function, seen as a function from a learned function f and an example z ( ) i ∈ ∈ d to a scalar r require : k, the number of folds split into mutually exclusive subsets d k di, whose union is. d for do i k from to 1 fi = ( a d d \ i ) for z ( ) j in di do ej = ( l fi, z ( ) j ) end for end for return e 123


chapter 5. machine learning basics ( i. i. d. ) data points. a or is any function of the data : point estimator statistic [UNK] = ( g x ( 1 ),..., x ( ) m ). ( 5. 19 ) the deﬁnition does not require that g return a value that is close to the true θ or even that the range of g is the same as the set of allowable values of θ. this deﬁnition of a point estimator is very general and allows the designer of an estimator great ﬂexibility. while almost any function thus qualiﬁes as an estimator, a good estimator is a function whose output is close to the true underlying θ that generated the training data. for now, we take the frequentist perspective on statistics. that is, we assume that the true parameter value θ is ﬁxed but unknown, while the point estimate [UNK] is a function of the data. since the data is drawn from a random process, any function of the data is random. therefore [UNK] is a random variable. point estimation can also refer to the estimation of the relationship between input and target variables. we refer to these types of point estimates as function est


is drawn from a random process, any function of the data is random. therefore [UNK] is a random variable. point estimation can also refer to the estimation of the relationship between input and target variables. we refer to these types of point estimates as function estimators. function estimation as we mentioned above, sometimes we are interested in performing function estimation ( or function approximation ). here we are trying to predict a variable y given an input vector x. we assume that there is a function f ( x ) that describes the approximate relationship between y and x. for example, we may assume that y = f ( x ) +, where stands for the part of y that is not predictable from x. in function estimation, we are interested in approximating f with a model or estimate [UNK]. function estimation is really just the same as estimating a parameter θ ; the function estimator [UNK] is simply a point estimator in function space. the linear regression example ( discussed above in section ) and 5. 1. 4 the polynomial regression example ( discussed in section ) are both examples of 5. 2 scenarios that may be interpreted either as estimating a parameter w or estimating a function [UNK] y mapping from to x. we now review the most commonly studied properties of point estima


regression example ( discussed in section ) are both examples of 5. 2 scenarios that may be interpreted either as estimating a parameter w or estimating a function [UNK] y mapping from to x. we now review the most commonly studied properties of point estimators and discuss what they tell us about these estimators. 5. 4. 2 bias the bias of an estimator is deﬁned as : bias ( [UNK] ) = ( e [UNK] ) −θ ( 5. 20 ) 124


chapter 5. machine learning basics where the expectation is over the data ( seen as samples from a random variable ) and θ is the true underlying value of θ used to deﬁne the data generating distri - bution. an estimator [UNK] is said to be unbiased if bias ( [UNK] ) = 0, which implies that e ( [UNK] ) = θ. an estimator [UNK] is said to be asymptotically unbiased if limm→∞bias ( [UNK] ) = 0, which implies that limm→∞e ( [UNK] m ) = θ. example : bernoulli distribution consider a set of samples { x ( 1 ),..., x ( ) m } that are independently and identically distributed according to a bernoulli distri - bution with mean : θ p x ( ( ) i ; ) = θ θx ( ) i ( 1 ) −θ ( 1−x ( ) i ). ( 5. 21 ) a common estimator for the θ parameter of this distribution is the mean of the training samples : [UNK] = 1 m m i = 1 x ( ) i. ( 5. 22 ) to determine whether this estimator is biased, we can substitute equation


estimator for the θ parameter of this distribution is the mean of the training samples : [UNK] = 1 m m i = 1 x ( ) i. ( 5. 22 ) to determine whether this estimator is biased, we can substitute equation 5. 22 into equation : 5. 20 bias ( [UNK] ) = [ e [UNK] ] −θ ( 5. 23 ) = e 1 m m i = 1 x ( ) i −θ ( 5. 24 ) = 1 m m i = 1 e x ( ) i −θ ( 5. 25 ) = 1 m m i = 1 1 x ( ) i = 0 x ( ) i θx ( ) i ( 1 ) −θ ( 1−x ( ) i ) −θ ( 5. 26 ) = 1 m m i = 1 ( ) θ −θ ( 5. 27 ) = = 0 θ θ − ( 5. 28 ) since bias ( [UNK] ) = 0, we say that our estimator [UNK] is unbiased. example : gaussian distribution estimator of the mean now, consider a set of samples { x ( 1 ),..., x ( ) m } that are independently and identically distributed according to a gaussian distribution p ( x


: gaussian distribution estimator of the mean now, consider a set of samples { x ( 1 ),..., x ( ) m } that are independently and identically distributed according to a gaussian distribution p ( x ( ) i ) = n ( x ( ) i ; µ, σ2 ), where i ∈ { 1,..., m }. 125


chapter 5. machine learning basics recall that the gaussian probability density function is given by p x ( ( ) i ; µ, σ 2 ) = 1 √ 2πσ2 exp −1 2 ( x ( ) i −µ ) 2 σ2. ( 5. 29 ) a common estimator of the gaussian mean parameter is known as the sample mean : [UNK] = 1 m m i = 1 x ( ) i ( 5. 30 ) to determine the bias of the sample mean, we are again interested in calculating its expectation : bias ( [UNK] ) = [ [UNK] e µm ] −µ ( 5. 31 ) = e 1 m m i = 1 x ( ) i −µ ( 5. 32 ) = 1 m m i = 1 e x ( ) i −µ ( 5. 33 ) = 1 m m i = 1 µ −µ ( 5. 34 ) = = 0 µ µ − ( 5. 35 ) thus we ﬁnd that the sample mean is an unbiased estimator of gaussian mean parameter. example : estimators of the variance of a gaussian distribution as an example, we compare two [UNK] estimators of the variance parameter σ2 of a gaussian distribution.


##biased estimator of gaussian mean parameter. example : estimators of the variance of a gaussian distribution as an example, we compare two [UNK] estimators of the variance parameter σ2 of a gaussian distribution. we are interested in knowing if either estimator is biased. the ﬁrst estimator of σ2 we consider is known as the sample variance : [UNK] m = 1 m m i = 1 x ( ) i [UNK] m 2, ( 5. 36 ) where [UNK] is the sample mean, deﬁned above. more formally, we are interested in computing bias ( [UNK] 2 m ) = [ [UNK] e σ 2 m ] −σ2 ( 5. 37 ) 126


chapter 5. machine learning basics we begin by evaluating the term e [ [UNK] 2 m ] : e [ [UNK] m ] = e 1 m m i = 1 x ( ) i [UNK] 2 ( 5. 38 ) = m −1 m σ2 ( 5. 39 ) returning to equation, we conclude that the bias of 5. 37 [UNK] 2 m is −σ2 / m. therefore, the sample variance is a biased estimator. the unbiased sample variance estimator [UNK] m = 1 m −1 m i = 1 x ( ) i [UNK] 2 ( 5. 40 ) provides an alternative approach. as the name suggests this estimator is unbiased. that is, we ﬁnd that e [ [UNK] m ] = σ2 : e [ [UNK] m ] = e 1 m −1 m i = 1 x ( ) i [UNK] 2 ( 5. 41 ) = m m −1e [ [UNK] 2 m ] ( 5. 42 ) = m m −1 m −1 m σ 2 ( 5. 43 ) = σ2. ( 5. 44 ) we have two estimators : one is biased and the other is not. while unbiased estimators are clearly desirable, they are not always the “ best


σ 2 ( 5. 43 ) = σ2. ( 5. 44 ) we have two estimators : one is biased and the other is not. while unbiased estimators are clearly desirable, they are not always the “ best ” estimators. as we will see we often use biased estimators that possess other important properties. 5. 4. 3 variance and standard error another property of the estimator that we might want to consider is how much we expect it to vary as a function of the data sample. just as we computed the expectation of the estimator to determine its bias, we can compute its variance. the variance of an estimator is simply the variance var ( [UNK] ) ( 5. 45 ) where the random variable is the training set. alternately, the square root of the variance is called the, denoted standard error se ( [UNK] ). 127


chapter 5. machine learning basics the variance or the standard error of an estimator provides a measure of how we would expect the estimate we compute from data to vary as we independently resample the dataset from the underlying data generating process. just as we might like an estimator to exhibit low bias we would also like it to have relatively low variance. when we compute any statistic using a ﬁnite number of samples, our estimate of the true underlying parameter is uncertain, in the sense that we could have obtained other samples from the same distribution and their statistics would have been [UNK]. the expected degree of variation in any estimator is a source of error that we want to quantify. the standard error of the mean is given by se ( [UNK] ) = var 1 m m i = 1 x ( ) i = σ √m, ( 5. 46 ) where σ2 is the true variance of the samples xi. the standard error is often estimated by using an estimate of σ. unfortunately, neither the square root of the sample variance nor the square root of the unbiased estimator of the variance provide an unbiased estimate of the standard deviation. both approaches tend to underestimate the true standard deviation, but are still used in practice


square root of the sample variance nor the square root of the unbiased estimator of the variance provide an unbiased estimate of the standard deviation. both approaches tend to underestimate the true standard deviation, but are still used in practice. the square root of the unbiased estimator of the variance is less of an underestimate. for large, the approximation is quite reasonable. m the standard error of the mean is very useful in machine learning experiments. we often estimate the generalization error by computing the sample mean of the error on the test set. the number of examples in the test set determines the accuracy of this estimate. taking advantage of the central limit theorem, which tells us that the mean will be approximately distributed with a normal distribution, we can use the standard error to compute the probability that the true expectation falls in any chosen interval. for example, the 95 % conﬁdence interval centered on the mean [UNK] is ( [UNK] −1 96se ( [UNK]. µm ) [UNK], µm + 1 96se ( [UNK]. µm ) ), ( 5. 47 ) under the normal distribution with mean [UNK] and variance se ( [UNK] ) 2. in machine learning experiments, it is common to say that algorithma is better than


µm + 1 96se ( [UNK]. µm ) ), ( 5. 47 ) under the normal distribution with mean [UNK] and variance se ( [UNK] ) 2. in machine learning experiments, it is common to say that algorithma is better than algorithm b if the upper bound of the 95 % conﬁdence interval for the error of algorithm a is less than the lower bound of the 95 % conﬁdence interval for the error of algorithm b. 128


chapter 5. machine learning basics example : bernoulli distribution we once again consider a set of samples { x ( 1 ),..., x ( ) m } drawn independently and identically from a bernoulli distribution ( recall p ( x ( ) i ; θ ) = θ x ( ) i ( 1 −θ ) ( 1−x ( ) i ) ). this time we are interested in computing the variance of the estimator [UNK] = 1 m m i = 1 x ( ) i. var [UNK] = var 1 m m i = 1 x ( ) i ( 5. 48 ) = 1 m2 m i = 1 var x ( ) i ( 5. 49 ) = 1 m2 m i = 1 θ θ ( 1 − ) ( 5. 50 ) = 1 m2 mθ θ ( 1 − ) ( 5. 51 ) = 1 mθ θ ( 1 − ) ( 5. 52 ) the variance of the estimator decreases as a function of m, the number of examples in the dataset. this is a common property of popular estimators that we will return to when we discuss consistency ( see section ). 5. 4. 5 5. 4. 4 trading [UNK] and variance to minimize mean squared error bias and


in the dataset. this is a common property of popular estimators that we will return to when we discuss consistency ( see section ). 5. 4. 5 5. 4. 4 trading [UNK] and variance to minimize mean squared error bias and variance measure two [UNK] sources of error in an estimator. bias measures the expected deviation from the true value of the function or parameter. variance on the other hand, provides a measure of the deviation from the expected estimator value that any particular sampling of the data is likely to cause. what happens when we are given a choice between two estimators, one with more bias and one with more variance? how do we choose between them? for example, imagine that we are interested in approximating the function shown in ﬁgure and we are only [UNK] the choice between a model with large bias and 5. 2 one that [UNK] from large variance. how do we choose between them? the most common way to negotiate this trade - [UNK] to use cross - validation. empirically, cross - validation is highly successful on many real - world tasks. alter - natively, we can also compare the mean squared error ( mse ) of the estimates : mse = [ ( e [UNK] −θ ) 2 ] ( 5


##ly, cross - validation is highly successful on many real - world tasks. alter - natively, we can also compare the mean squared error ( mse ) of the estimates : mse = [ ( e [UNK] −θ ) 2 ] ( 5. 53 ) = bias ( [UNK] ) 2 + var ( [UNK] ) ( 5. 54 ) 129


chapter 5. machine learning basics the mse measures the overall expected deviation — in a squared error sense — between the estimator and the true value of the parameter θ. as is clear from equation, evaluating the mse incorporates both the bias and the variance. 5. 54 desirable estimators are those with small mse and these are estimators that manage to keep both their bias and variance somewhat in check. capacity bias generalization error variance optimal capacity overﬁtting zone underﬁtting zone figure 5. 6 : as capacity increases ( x - axis ), bias ( dotted ) tends to decrease and variance ( dashed ) tends to increase, yielding another u - shaped curve for generalization error ( bold curve ). if we vary capacity along one axis, there is an optimal capacity, with underﬁtting when the capacity is below this optimum and overﬁtting when it is above. this relationship is similar to the relationship between capacity, underﬁtting, and overﬁtting, discussed in section and ﬁgure. 5. 2 5. 3 the relationship between bias and variance is tightly linked to the machine learning concepts of capacity, underﬁtting and overﬁtting. in the case where gen - eralization error is measured by the mse ( where


##e. 5. 2 5. 3 the relationship between bias and variance is tightly linked to the machine learning concepts of capacity, underﬁtting and overﬁtting. in the case where gen - eralization error is measured by the mse ( where bias and variance are meaningful components of generalization error ), increasing capacity tends to increase variance and decrease bias. this is illustrated in ﬁgure, where we see again the u - shaped 5. 6 curve of generalization error as a function of capacity. 5. 4. 5 consistency so far we have discussed the properties of various estimators for a training set of ﬁxed size. usually, we are also concerned with the behavior of an estimator as the amount of training data grows. in particular, we usually wish that, as the number of data points m in our dataset increases, our point estimates converge to the true 130


chapter 5. machine learning basics value of the corresponding parameters. more formally, we would like that [UNK] = θ. ( 5. 55 ) the symbol plim indicates convergence in probability, meaning that for any > 0, p ( | [UNK] − | θ > ) →0 as m →∞. the condition described by equation is 5. 55 known as consistency. it is sometimes referred to as weak consistency, with strong consistency referring to the almost sure convergence of [UNK] to θ. almost sure convergence of a sequence of random variables x ( 1 ), x ( 2 ),... to a value x occurs when p ( limm→∞x ( ) m = ) = 1 x. consistency ensures that the bias induced by the estimator diminishes as the number of data examples grows. however, the reverse is not true — asymptotic unbiasedness does not imply consistency. for example, consider estimating the mean parameter µ of a normal distribution n ( x ; µ, σ2 ), with a dataset consisting of m samples : { x ( 1 ),..., x ( ) m }. we could use the ﬁrst sample x ( 1 ) of the dataset as an unbiased estimator


), with a dataset consisting of m samples : { x ( 1 ),..., x ( ) m }. we could use the ﬁrst sample x ( 1 ) of the dataset as an unbiased estimator : [UNK] = x ( 1 ). in that case, e ( [UNK] m ) = θ so the estimator is unbiased no matter how many data points are seen. this, of course, implies that the estimate is asymptotically unbiased. however, this is not a consistent estimator as it is the case that not [UNK] → →∞ θ m as. 5. 5 maximum likelihood estimation previously, we have seen some deﬁnitions of common estimators and analyzed their properties. but where did these estimators come from? rather than guessing that some function might make a good estimator and then analyzing its bias and variance, we would like to have some principle from which we can derive speciﬁc functions that are good estimators for [UNK] models. the most common such principle is the maximum likelihood principle. consider a set of m examples x = { x ( 1 ),..., x ( ) m } drawn independently from the true but unknown


are good estimators for [UNK] models. the most common such principle is the maximum likelihood principle. consider a set of m examples x = { x ( 1 ),..., x ( ) m } drawn independently from the true but unknown data generating distribution pdata ( ) x. let pmodel ( x ; θ ) be a parametric family of probability distributions over the same space indexed by θ. in other words, pmodel ( x ; θ ) maps any conﬁguration x to a real number estimating the true probability pdata ( ) x. the maximum likelihood estimator for is then deﬁned as θ θml = arg max θ pmodel ( ; ) x θ ( 5. 56 ) = arg max θ m i = 1 pmodel ( x ( ) i ; ) θ ( 5. 57 ) 131


chapter 5. machine learning basics this product over many probabilities can be inconvenient for a variety of reasons. for example, it is prone to numerical underﬂow. to obtain a more convenient but equivalent optimization problem, we observe that taking the logarithm of the likelihood does not change its arg max but does conveniently transform a product into a sum : θml = arg max θ m i = 1 log pmodel ( x ( ) i ; ) θ. ( 5. 58 ) because the arg max does not change when we rescale the cost function, we can divide by m to obtain a version of the criterion that is expressed as an expectation with respect to the empirical distribution [UNK] deﬁned by the training data : θml = arg max θ [UNK] log pmodel ( ; ) x θ. ( 5. 59 ) one way to interpret maximum likelihood estimation is to view it as minimizing the dissimilarity between the empirical distribution [UNK] deﬁned by the training set and the model distribution, with the degree of dissimilarity between the two measured by the kl divergence. the kl divergence is given by dkl ( [UNK] ) = [UNK] [ log [UNK]


##ﬁned by the training set and the model distribution, with the degree of dissimilarity between the two measured by the kl divergence. the kl divergence is given by dkl ( [UNK] ) = [UNK] [ log [UNK] ( ) log x − pmodel ( ) ] x. ( 5. 60 ) the term on the left is a function only of the data generating process, not the model. this means when we train the model to minimize the kl divergence, we need only minimize [UNK] [ log pmodel ( ) ] x ( 5. 61 ) which is of course the same as the maximization in equation. 5. 59 minimizing this kl divergence corresponds exactly to minimizing the cross - entropy between the distributions. many authors use the term “ cross - entropy ” to identify speciﬁcally the negative log - likelihood of a bernoulli or softmax distribution, but that is a misnomer. any loss consisting of a negative log - likelihood is a cross - entropy between the empirical distribution deﬁned by the training set and the probability distribution deﬁned by model. for example, mean squared error is the cross - entropy between the empirical distribution and a gaussian model.


- likelihood is a cross - entropy between the empirical distribution deﬁned by the training set and the probability distribution deﬁned by model. for example, mean squared error is the cross - entropy between the empirical distribution and a gaussian model. we can thus see maximum likelihood as an attempt to make the model dis - tribution match the empirical distribution [UNK]. ideally, we would like to match the true data generating distribution pdata, but we have no direct access to this distribution. while the optimal θ is the same regardless of whether we are maximizing the likelihood or minimizing the kl divergence, the values of the objective functions 132


chapter 5. machine learning basics are [UNK]. in software, we often phrase both as minimizing a cost function. maximum likelihood thus becomes minimization of the negative log - likelihood ( nll ), or equivalently, minimization of the cross entropy. the perspective of maximum likelihood as minimum kl divergence becomes helpful in this case because the kl divergence has a known minimum value of zero. the negative log - likelihood can actually become negative when is real - valued. x 5. 5. 1 conditional log - likelihood and mean squared error the maximum likelihood estimator can readily be generalized to the case where our goal is to estimate a conditional probability p ( y x | ; θ ) in order to predict y given x. this is actually the most common situation because it forms the basis for most supervised learning. if x represents all our inputs and y all our observed targets, then the conditional maximum likelihood estimator is θml = arg max θ p. ( ; ) y x | θ ( 5. 62 ) if the examples are assumed to be i. i. d., then this can be decomposed into θml = arg max θ m i = 1 log ( p y ( ) i | x ( ) i


( 5. 62 ) if the examples are assumed to be i. i. d., then this can be decomposed into θml = arg max θ m i = 1 log ( p y ( ) i | x ( ) i ; ) θ. ( 5. 63 ) example : linear regression as maximum likelihood linear regression, introduced earlier in section, may be justiﬁed as a maximum likelihood 5. 1. 4 procedure. previously, we motivated linear regression as an algorithm that learns to take an input x and produce an output value [UNK]. the mapping from x to [UNK] is chosen to minimize mean squared error, a criterion that we introduced more or less arbitrarily. we now revisit linear regression from the point of view of maximum likelihood estimation. instead of producing a single prediction [UNK], we now think of the model as producing a conditional distribution p ( y | x ). we can imagine that with an inﬁnitely large training set, we might see several training examples with the same input value x but [UNK] values of y. the goal of the learning algorithm is now to ﬁt the distribution p ( y | x ) to all of those [UNK] y values that are all compatible with x. to derive the same linear regression


the same input value x but [UNK] values of y. the goal of the learning algorithm is now to ﬁt the distribution p ( y | x ) to all of those [UNK] y values that are all compatible with x. to derive the same linear regression algorithm we obtained before, we deﬁne p ( y | x ) = n ( y ; [UNK] ( x ; w ), σ2 ). the function [UNK] ( x ; w ) gives the prediction of the mean of the gaussian. in this example, we assume that the variance is ﬁxed to some constant σ 2 chosen by the user. we will see that this choice of the functional form of p ( y | x ) causes the maximum likelihood estimation procedure to yield the same learning algorithm as we developed before. since the 133


chapter 5. machine learning basics examples are assumed to be i. i. d., the conditional log - likelihood ( equation ) is 5. 63 given by m i = 1 log ( p y ( ) i | x ( ) i ; ) θ ( 5. 64 ) = log −m σ −m 2 log ( 2 ) π − m i = 1 [UNK] ( ) i −y ( ) i 2 2σ2, ( 5. 65 ) where [UNK] ( ) i is the output of the linear regression on the i - th input x ( ) i and m is the number of the training examples. comparing the log - likelihood with the mean squared error, msetrain = 1 m m i = 1 | | [UNK] ( ) i −y ( ) i | | 2, ( 5. 66 ) we immediately see that maximizing the log - likelihood with respect to w yields the same estimate of the parameters w as does minimizing the mean squared error. the two criteria have [UNK] values but the same location of the optimum. this justiﬁes the use of the mse as a maximum likelihood estimation procedure. as we will see, the maximum likelihood estimator has several desirable properties. 5. 5. 2 properties of maximum likelihood


same location of the optimum. this justiﬁes the use of the mse as a maximum likelihood estimation procedure. as we will see, the maximum likelihood estimator has several desirable properties. 5. 5. 2 properties of maximum likelihood the main appeal of the maximum likelihood estimator is that it can be shown to be the best estimator asymptotically, as the number of examples m →∞, in terms of its rate of convergence as increases. m under appropriate conditions, the maximum likelihood estimator has the property of consistency ( see section above ), meaning that as the number 5. 4. 5 of training examples approaches inﬁnity, the maximum likelihood estimate of a parameter converges to the true value of the parameter. these conditions are : • the true distribution pdata must lie within the model family pmodel ( · ; θ ). otherwise, no estimator can recover pdata. • the true distribution pdata must correspond to exactly one value of θ. other - wise, maximum likelihood can recover the correct pdata, but will not be able to determine which value of was used by the data generating processing. θ there are other inductive principles besides the maximum likelihood estima - tor, many of


other - wise, maximum likelihood can recover the correct pdata, but will not be able to determine which value of was used by the data generating processing. θ there are other inductive principles besides the maximum likelihood estima - tor, many of which share the property of being consistent estimators. however, 134


chapter 5. machine learning basics consistent estimators can [UNK] in their statistic [UNK], meaning that one consistent estimator may obtain lower generalization error for a ﬁxed number of samples m, or equivalently, may require fewer examples to obtain a ﬁxed level of generalization error. statistical [UNK] is typically studied in the parametric case ( like in linear regression ) where our goal is to estimate the value of a parameter ( and assuming it is possible to identify the true parameter ), not the value of a function. a way to measure how close we are to the true parameter is by the expected mean squared error, computing the squared [UNK] between the estimated and true parameter values, where the expectation is over m training samples from the data generating distribution. that parametric mean squared error decreases as m increases, and for m large, the cramer - rao lower bound (, ;, ) shows that no rao 1945 cramer 1946 consistent estimator has a lower mean squared error than the maximum likelihood estimator. for these reasons ( consistency and [UNK] ), maximum likelihood is often considered the preferred estimator to use for machine learning. when the number of examples is small enough to yield overﬁtting behavior, regularization strategies such as weight decay may be used to obtain


reasons ( consistency and [UNK] ), maximum likelihood is often considered the preferred estimator to use for machine learning. when the number of examples is small enough to yield overﬁtting behavior, regularization strategies such as weight decay may be used to obtain a biased version of maximum likelihood that has less variance when training data is limited. 5. 6 bayesian statistics so far we have discussed frequentist statistics and approaches based on estimat - ing a single value of θ, then making all predictions thereafter based on that one estimate. another approach is to consider all possible values of θ when making a prediction. the latter is the domain of bayesian statistics. as discussed in section, the frequentist perspective is that the true 5. 4. 1 parameter value θ is ﬁxed but unknown, while the point estimate [UNK] is a random variable on account of it being a function of the dataset ( which is seen as random ). the bayesian perspective on statistics is quite [UNK]. the bayesian uses probability to reﬂect degrees of certainty of states of knowledge. the dataset is directly observed and so is not random. on the other hand, the true parameter θ is unknown or uncertain and thus is represented as a random variable. before observing the data, we represent our knowledge


of certainty of states of knowledge. the dataset is directly observed and so is not random. on the other hand, the true parameter θ is unknown or uncertain and thus is represented as a random variable. before observing the data, we represent our knowledge of θ using the prior probability distribution, p ( θ ) ( sometimes referred to as simply “ the prior ” ). generally, the machine learning practitioner selects a prior distribution that is quite broad ( i. e. with high entropy ) to reﬂect a high degree of uncertainty in the 135


chapter 5. machine learning basics value of θ before observing any data. for example, one might assume that a priori θ lies in some ﬁnite range or volume, with a uniform distribution. many priors instead reﬂect a preference for “ simpler ” solutions ( such as smaller magnitude [UNK], or a function that is closer to being constant ). now consider that we have a set of data samples { x ( 1 ),..., x ( ) m }. we can recover the [UNK] of data on our belief about θ by combining the data likelihood p x ( ( 1 ),..., x ( ) m | θ ) with the prior via bayes ’ rule : p x ( θ | ( 1 ),..., x ( ) m ) = p x ( ( 1 ),..., x ( ) m | θ θ ) ( p ) p x ( ( 1 ),..., x ( ) m ) ( 5. 67 ) in the scenarios where bayesian estimation is typically used, the prior begins as a relatively uniform or gaussian distribution with high entropy, and the observation of the data usually causes the posterior to lose entropy and concentrate around a few highly likely values of the parameters.


scenarios where bayesian estimation is typically used, the prior begins as a relatively uniform or gaussian distribution with high entropy, and the observation of the data usually causes the posterior to lose entropy and concentrate around a few highly likely values of the parameters. relative to maximum likelihood estimation, bayesian estimation [UNK] two important [UNK]. first, unlike the maximum likelihood approach that makes predictions using a point estimate of θ, the bayesian approach is to make predictions using a full distribution over θ. for example, after observing m examples, the predicted distribution over the next data sample, x ( + 1 ) m, is given by p x ( ( + 1 ) m | x ( 1 ),..., x ( ) m ) = p x ( ( + 1 ) m | | θ θ ) ( p x ( 1 ),..., x ( ) m ) d. θ ( 5. 68 ) here each value of θ with positive probability density contributes to the prediction of the next example, with the contribution weighted by the posterior density itself. after having observed { x ( 1 ),..., x ( ) m }, if we are still quite uncertain about the value of θ, then this uncertainty is incorporated directly into any predictions we might make.


by the posterior density itself. after having observed { x ( 1 ),..., x ( ) m }, if we are still quite uncertain about the value of θ, then this uncertainty is incorporated directly into any predictions we might make. in section, we discussed how the frequentist approach addresses the uncer - 5. 4 tainty in a given point estimate of θ by evaluating its variance. the variance of the estimator is an assessment of how the estimate might change with alternative samplings of the observed data. the bayesian answer to the question of how to deal with the uncertainty in the estimator is to simply integrate over it, which tends to protect well against overﬁtting. this integral is of course just an application of the laws of probability, making the bayesian approach simple to justify, while the frequentist machinery for constructing an estimator is based on the rather ad hoc decision to summarize all knowledge contained in the dataset with a single point estimate. the second important [UNK] between the bayesian approach to estimation and the maximum likelihood approach is due to the contribution of the bayesian 136


chapter 5. machine learning basics prior distribution. the prior has an inﬂuence by shifting probability mass density towards regions of the parameter space that are preferred. in practice, a priori the prior often expresses a preference for models that are simpler or more smooth. critics of the bayesian approach identify the prior as a source of subjective human judgment impacting the predictions. bayesian methods typically generalize much better when limited training data is available, but typically [UNK] from high computational cost when the number of training examples is large. example : bayesian linear regression here we consider the bayesian esti - mation approach to learning the linear regression parameters. in linear regression, we learn a linear mapping from an input vector x ∈rn to predict the value of a scalar. the prediction is parametrized by the vector y ∈r w ∈rn : [UNK] = wx. ( 5. 69 ) given a set of m training samples ( x ( ) train, y ( ) train ), we can express the prediction of over the entire training set as : y [UNK] ( ) train = x ( ) train w. ( 5. 70 ) expressed as a gaussian conditional distribution on y ( ) train, we have p ( y ( ) train | x ( )


over the entire training set as : y [UNK] ( ) train = x ( ) train w. ( 5. 70 ) expressed as a gaussian conditional distribution on y ( ) train, we have p ( y ( ) train | x ( ) train, w y ) = ( n ( ) train ; x ( ) train w i, ) ( 5. 71 ) [UNK] −1 2 ( y ( ) train −x ( ) train w ) ( y ( ) train −x ( ) train w ), ( 5. 72 ) where we follow the standard mse formulation in assuming that the gaussian variance on y is one. in what follows, to reduce the notational burden, we refer to ( x ( ) train, y ( ) train ) ( ) as simply x y,. to determine the posterior distribution over the model parameter vector w, we ﬁrst need to specify a prior distribution. the prior should reﬂect our naive belief about the value of these parameters. while it is sometimes [UNK] or unnatural to express our prior beliefs in terms of the parameters of the model, in practice we typically assume a fairly broad distribution expressing a high degree of uncertainty about θ. for real - valued parameters it is common to use a gaussian as a


or unnatural to express our prior beliefs in terms of the parameters of the model, in practice we typically assume a fairly broad distribution expressing a high degree of uncertainty about θ. for real - valued parameters it is common to use a gaussian as a prior distribution : p ( ) = ( ; w n w µ0, λ0 ) exp [UNK] −1 2 ( w µ − 0 ) λ−1 0 ( w µ − 0 ), ( 5. 73 ) 137


chapter 5. machine learning basics where µ0 and λ0 are the prior distribution mean vector and covariance matrix respectively. 1 with the prior thus speciﬁed, we can now proceed in determining the posterior distribution over the model parameters. p, p, p ( w x | y ) [UNK] ( y x | w ) ( ) w ( 5. 74 ) [UNK] −1 2 ( ) y xw − ( ) y xw − exp −1 2 ( w µ − 0 ) λ−1 0 ( w µ − 0 ) ( 5. 75 ) [UNK] −1 2 −2yxw w + xxw w + λ−1 0 w µ −2 0 λ−1 0 w. ( 5. 76 ) we now deﬁne λm = xx + λ−1 0 −1 and µm = λm xy + λ−1 0 µ0. using these new variables, we ﬁnd that the posterior may be rewritten as a gaussian distribution : p, ( w x | y ) exp [UNK] −1 2 ( w µ − m ) λ−1 m ( w µ − m ) + 1 2µ mλ−1 m µm ( 5. 77 ) [UNK] −1 2 ( w µ − m ) λ−1


y ) exp [UNK] −1 2 ( w µ − m ) λ−1 m ( w µ − m ) + 1 2µ mλ−1 m µm ( 5. 77 ) [UNK] −1 2 ( w µ − m ) λ−1 m ( w µ − m ). ( 5. 78 ) all terms that do not include the parameter vector w have been omitted ; they are implied by the fact that the distribution must be normalized to integrate to. 1 equation shows how to normalize a multivariate gaussian distribution. 3. 23 examining this posterior distribution allows us to gain some intuition for the [UNK] of bayesian inference. in most situations, we set µ0 to 0. if we set λ0 = 1 αi, then µm gives the same estimate of w as does frequentist linear regression with a weight decay penalty of αww. one [UNK] is that the bayesian estimate is undeﬁned if α is set to zero — - we are not allowed to begin the bayesian learning process with an inﬁnitely wide prior on w. the more important [UNK] is that the bayesian estimate provides a covariance matrix, showing how likely all the [UNK] values of are, rather than providing only the estimate w µ


##esian learning process with an inﬁnitely wide prior on w. the more important [UNK] is that the bayesian estimate provides a covariance matrix, showing how likely all the [UNK] values of are, rather than providing only the estimate w µm. 5. 6. 1 maximum ( map ) estimation a posteriori while the most principled approach is to make predictions using the full bayesian posterior distribution over the parameter θ, it is still often desirable to have a 1 unless there is a reason to assume a particular covariance structure, we typically assume a diagonal covariance matrix λ0 = diag ( λ0 ). 138


chapter 5. machine learning basics single point estimate. one common reason for desiring a point estimate is that most operations involving the bayesian posterior for most interesting models are intractable, and a point estimate [UNK] a tractable approximation. rather than simply returning to the maximum likelihood estimate, we can still gain some of the beneﬁt of the bayesian approach by allowing the prior to inﬂuence the choice of the point estimate. one rational way to do this is to choose the maximum a posteriori ( map ) point estimate. the map estimate chooses the point of maximal posterior probability ( or maximal probability density in the more common case of continuous ) : θ θmap = arg max θ p ( ) = arg max θ x | θ log ( ) + log ( ) p x θ | p θ. ( 5. 79 ) we recognize, above on the right hand side, log p ( x θ | ), i. e. the standard log - likelihood term, and, corresponding to the prior distribution. log ( ) p θ as an example, consider a linear regression model with a gaussian prior on the weights w. if this prior is given by n ( w ; 0, 1 λi2 ), then the log - prior


distribution. log ( ) p θ as an example, consider a linear regression model with a gaussian prior on the weights w. if this prior is given by n ( w ; 0, 1 λi2 ), then the log - prior term in equation is proportional to the familiar 5. 79 λww weight decay penalty, plus a term that does not depend on w and does not [UNK] the learning process. map bayesian inference with a gaussian prior on the weights thus corresponds to weight decay. as with full bayesian inference, map bayesian inference has the advantage of leveraging information that is brought by the prior and cannot be found in the training data. this additional information helps to reduce the variance in the map point estimate ( in comparison to the ml estimate ). however, it does so at the price of increased bias. many regularized estimation strategies, such as maximum likelihood learning regularized with weight decay, can be interpreted as making the map approxima - tion to bayesian inference. this view applies when the regularization consists of adding an extra term to the objective function that corresponds to log p ( θ ). not all regularization penalties correspond to map bayesian inference. for example, some regularizer terms may not be the logari


view applies when the regularization consists of adding an extra term to the objective function that corresponds to log p ( θ ). not all regularization penalties correspond to map bayesian inference. for example, some regularizer terms may not be the logarithm of a probability distribution. other regularization terms depend on the data, which of course a prior probability distribution is not allowed to do. map bayesian inference provides a straightforward way to design complicated yet interpretable regularization terms. for example, a more complicated penalty term can be derived by using a mixture of gaussians, rather than a single gaussian distribution, as the prior ( nowlan and hinton 1992, ). 139


chapter 5. machine learning basics 5. 7 supervised learning algorithms recall from section that supervised learning algorithms are, roughly speaking, 5. 1. 3 learning algorithms that learn to associate some input with some output, given a training set of examples of inputs x and outputs y. in many cases the outputs y may be [UNK] to collect automatically and must be provided by a human “ supervisor, ” but the term still applies even when the training set targets were collected automatically. 5. 7. 1 probabilistic supervised learning most supervised learning algorithms in this book are based on estimating a probability distribution p ( y | x ). we can do this simply by using maximum likelihood estimation to ﬁnd the best parameter vector θ for a parametric family of distributions. p y ( | x θ ; ) we have already seen that linear regression corresponds to the family p y y ( | n x θ ; ) = ( ; θx i, ). ( 5. 80 ) we can generalize linear regression to the classiﬁcation scenario by deﬁning a [UNK] family of probability distributions. if we have two classes, class 0 and class 1, then we need only specify the probability of one of these classes. the probability of class 1 determines the probability of class 0, because


scenario by deﬁning a [UNK] family of probability distributions. if we have two classes, class 0 and class 1, then we need only specify the probability of one of these classes. the probability of class 1 determines the probability of class 0, because these two values must add up to 1. the normal distribution over real - valued numbers that we used for linear regression is parametrized in terms of a mean. any value we supply for this mean is valid. a distribution over a binary variable is slightly more complicated, because its mean must always be between 0 and 1. one way to solve this problem is to use the logistic sigmoid function to squash the output of the linear function into the interval ( 0, 1 ) and interpret that value as a probability : p y σ ( = 1 ; ) = | x θ ( θx ). ( 5. 81 ) this approach is known as logistic regression ( a somewhat strange name since we use the model for classiﬁcation rather than regression ). in the case of linear regression, we were able to ﬁnd the optimal weights by solving the normal equations. logistic regression is somewhat more [UNK]. there is no closed - form solution for its optimal weights. instead, we must search for them by maxim


the case of linear regression, we were able to ﬁnd the optimal weights by solving the normal equations. logistic regression is somewhat more [UNK]. there is no closed - form solution for its optimal weights. instead, we must search for them by maximizing the log - likelihood. we can do this by minimizing the negative log - likelihood ( nll ) using gradient descent. 140


chapter 5. machine learning basics this same strategy can be applied to essentially any supervised learning problem, by writing down a parametric family of conditional probability distributions over the right kind of input and output variables. 5. 7. 2 support vector machines one of the most inﬂuential approaches to supervised learning is the support vector machine (, ; boser et al. 1992 cortes and vapnik 1995, ). this model is similar to logistic regression in that it is driven by a linear function wx + b. unlike logistic regression, the support vector machine does not provide probabilities, but only outputs a class identity. the svm predicts that the positive class is present when wx + b is positive. likewise, it predicts that the negative class is present when wx + b is negative. one key innovation associated with support vector machines is the kernel trick. the kernel trick consists of observing that many machine learning algorithms can be written exclusively in terms of dot products between examples. for example, it can be shown that the linear function used by the support vector machine can be re - written as wx + = + b b m i = 1 αixx ( ) i ( 5. 82 ) where x ( ) i is a training example


can be shown that the linear function used by the support vector machine can be re - written as wx + = + b b m i = 1 αixx ( ) i ( 5. 82 ) where x ( ) i is a training example and α is a vector of [UNK]. rewriting the learning algorithm this way allows us to replace x by the output of a given feature function φ ( x ) and the dot product with a function k ( x x, ( ) i ) = φ ( x ) · φ ( x ( ) i ) called a kernel. the · operator represents an inner product analogous to φ ( x ) φ ( x ( ) i ). for some feature spaces, we may not use literally the vector inner product. in some inﬁnite dimensional spaces, we need to use other kinds of inner products, for example, inner products based on integration rather than summation. a complete development of these kinds of inner products is beyond the scope of this book. after replacing dot products with kernel evaluations, we can make predictions using the function f b ( ) = x + i α ik, ( x x ( ) i ). ( 5. 83 ) this function is nonlinear with respect to x, but the relationship between φ ( x


evaluations, we can make predictions using the function f b ( ) = x + i α ik, ( x x ( ) i ). ( 5. 83 ) this function is nonlinear with respect to x, but the relationship between φ ( x ) and f ( x ) is linear. also, the relationship between α and f ( x ) is linear. the kernel - based function is exactly equivalent to preprocessing the data by applying φ ( ) x to all inputs, then learning a linear model in the new transformed space. the kernel trick is powerful for two reasons. first, it allows us to learn models that are nonlinear as a function of x using convex optimization techniques that are 141


chapter 5. machine learning basics guaranteed to converge [UNK]. this is possible because we consider φ ﬁxed and optimize only α, i. e., the optimization algorithm can view the decision function as being linear in a [UNK] space. second, the kernel function k often admits an implementation that is signiﬁcantly more computational [UNK] than naively constructing two vectors and explicitly taking their dot product. φ ( ) x in some cases, φ ( x ) can even be inﬁnite dimensional, which would result in an inﬁnite computational cost for the naive, explicit approach. in many cases, k ( x x, ) is a nonlinear, tractable function of x even when φ ( x ) is intractable. as an example of an inﬁnite - dimensional feature space with a tractable kernel, we construct a feature mapping φ ( x ) over the non - negative integers x. suppose that this mapping returns a vector containing x ones followed by inﬁnitely many zeros. we can write a kernel function k ( x, x ( ) i ) = min ( x, x ( ) i ) that is exactly equivalent to the corresponding inﬁnite - dimensional dot product. the most commonly used kernel is the gauss


we can write a kernel function k ( x, x ( ) i ) = min ( x, x ( ) i ) that is exactly equivalent to the corresponding inﬁnite - dimensional dot product. the most commonly used kernel is the gaussian kernel k,, σ ( u v u v ) = ( n − ; 0 2i ) ( 5. 84 ) where n ( x ; µ, σ ) is the standard normal density. this kernel is also known as the radial basis function ( rbf ) kernel, because its value decreases along lines in v space radiating outward from u. the gaussian kernel corresponds to a dot product in an inﬁnite - dimensional space, but the derivation of this space is less straightforward than in our example of the kernel over the integers. min we can think of the gaussian kernel as performing a kind of template match - ing. a training example x associated with training label y becomes a template for class y. when a test point xis near x according to euclidean distance, the gaussian kernel has a large response, indicating that xis very similar to the x template. the model then puts a large weight on the associated training label y. overall, the prediction will combine many such training labels weighted


to euclidean distance, the gaussian kernel has a large response, indicating that xis very similar to the x template. the model then puts a large weight on the associated training label y. overall, the prediction will combine many such training labels weighted by the similarity of the corresponding training examples. support vector machines are not the only algorithm that can be enhanced using the kernel trick. many other linear models can be enhanced in this way. the category of algorithms that employ the kernel trick is known as kernel machines or kernel methods (, ; williams and rasmussen 1996 scholkopf 1999 et al., ). a major drawback to kernel machines is that the cost of evaluating the decision function is linear in the number of training examples, because the i - th example contributes a term αik ( x x, ( ) i ) to the decision function. support vector machines are able to mitigate this by learning an α vector that contains mostly zeros. classifying a new example then requires evaluating the kernel function only for the training examples that have non - zero α i. these training examples are known 142


chapter 5. machine learning basics as support vectors. kernel machines also [UNK] from a high computational cost of training when the dataset is large. we will revisit this idea in section. kernel machines with 5. 9 generic kernels struggle to generalize well. we will explain why in section. the 5. 11 modern incarnation of deep learning was designed to overcome these limitations of kernel machines. the current deep learning renaissance began when hinton et al. ( ) demonstrated that a neural network could outperform the rbf kernel svm 2006 on the mnist benchmark. 5. 7. 3 other simple supervised learning algorithms we have already brieﬂy encountered another non - probabilistic supervised learning algorithm, nearest neighbor regression. more generally, k - nearest neighbors is a family of techniques that can be used for classiﬁcation or regression. as a non - parametric learning algorithm, k - nearest neighbors is not restricted to a ﬁxed number of parameters. we usually think of the k - nearest neighbors algorithm as not having any parameters, but rather implementing a simple function of the training data. in fact, there is not even really a training stage or learning process. instead, at test time, when we want to produce an output y for a new


algorithm as not having any parameters, but rather implementing a simple function of the training data. in fact, there is not even really a training stage or learning process. instead, at test time, when we want to produce an output y for a new test input x, we ﬁnd the k - nearest neighbors to x in the training data x. we then return the average of the corresponding y values in the training set. this works for essentially any kind of supervised learning where we can deﬁne an average over y values. in the case of classiﬁcation, we can average over one - hot code vectors c with cy = 1 and ci = 0 for all other values of i. we can then interpret the average over these one - hot codes as giving a probability distribution over classes. as a non - parametric learning algorithm, k - nearest neighbor can achieve very high capacity. for example, suppose we have a multiclass classiﬁcation task and measure performance with 0 - 1 loss. in this setting, - nearest neighbor converges to double the bayes error as the 1 number of training examples approaches inﬁnity. the error in excess of the bayes error results from choosing a single neighbor by breaking ties between equally distant neighbors randomly. when there is


- nearest neighbor converges to double the bayes error as the 1 number of training examples approaches inﬁnity. the error in excess of the bayes error results from choosing a single neighbor by breaking ties between equally distant neighbors randomly. when there is inﬁnite training data, all test points x will have inﬁnitely many training set neighbors at distance zero. if we allow the algorithm to use all of these neighbors to vote, rather than randomly choosing one of them, the procedure converges to the bayes error rate. the high capacity of k - nearest neighbors allows it to obtain high accuracy given a large training set. however, it does so at high computational cost, and it may generalize very badly given a small, ﬁnite training set. one weakness of k - nearest neighbors is that it cannot learn that one feature is more discriminative than another. for example, imagine we have a regression task with x ∈r100 drawn from an isotropic gaussian 143


chapter 5. machine learning basics distribution, but only a single variable x1 is relevant to the output. suppose further that this feature simply encodes the output directly, i. e. that y = x1 in all cases. nearest neighbor regression will not be able to detect this simple pattern. the nearest neighbor of most points x will be determined by the large number of features x2 through x100, not by the lone feature x1. thus the output on small training sets will essentially be random. 144


chapter 5. machine learning basics 0 1 01 111 0 1 011 1111 1110 110 10 010 00 1110 1111 110 10 01 00 010 011 11 111 11 figure 5. 7 : diagrams describing how a decision tree works. ( top ) each node of the tree chooses to send the input example to the child node on the left ( 0 ) or or the child node on the right ( 1 ). internal nodes are drawn as circles and leaf nodes as squares. each node is displayed with a binary string identiﬁer corresponding to its position in the tree, obtained by appending a bit to its parent identiﬁer ( 0 = choose left or top, 1 = choose right or bottom ). ( bottom ) the tree divides space into regions. the 2d plane shows how a decision tree might divide r2. the nodes of the tree are plotted in this plane, with each internal node drawn along the dividing line it uses to categorize examples, and leaf nodes drawn in the center of the region of examples they receive. the result is a piecewise - constant function, with one piece per leaf. each leaf requires at least one training example to deﬁne, so it is not possible for the decision tree to learn


the center of the region of examples they receive. the result is a piecewise - constant function, with one piece per leaf. each leaf requires at least one training example to deﬁne, so it is not possible for the decision tree to learn a function that has more local maxima than the number of training examples. 145


chapter 5. machine learning basics another type of learning algorithm that also breaks the input space into regions and has separate parameters for each region is the decision tree (, breiman et al. 1984 ) and its many variants. as shown in ﬁgure, each node of the decision 5. 7 tree is associated with a region in the input space, and internal nodes break that region into one sub - region for each child of the node ( typically using an axis - aligned cut ). space is thus sub - divided into non - overlapping regions, with a one - to - one correspondence between leaf nodes and input regions. each leaf node usually maps every point in its input region to the same output. decision trees are usually trained with specialized algorithms that are beyond the scope of this book. the learning algorithm can be considered non - parametric if it is allowed to learn a tree of arbitrary size, though decision trees are usually regularized with size constraints that turn them into parametric models in practice. decision trees as they are typically used, with axis - aligned splits and constant outputs within each node, struggle to solve some problems that are easy even for logistic regression. for example, if we have a two - class problem and the positive class occurs wherever x2 > x1


used, with axis - aligned splits and constant outputs within each node, struggle to solve some problems that are easy even for logistic regression. for example, if we have a two - class problem and the positive class occurs wherever x2 > x1, the decision boundary is not axis - aligned. the decision tree will thus need to approximate the decision boundary with many nodes, implementing a step function that constantly walks back and forth across the true decision function with axis - aligned steps. as we have seen, nearest neighbor predictors and decision trees have many limitations. nonetheless, they are useful learning algorithms when computational resources are constrained. we can also build intuition for more sophisticated learning algorithms by thinking about the similarities and [UNK] between sophisticated algorithms and - nn or decision tree baselines. k see ( ), ( ), ( ) or other machine murphy 2012 bishop 2006 hastie et al. 2001 learning textbooks for more material on traditional supervised learning algorithms. 5. 8 unsupervised learning algorithms recall from section that unsupervised algorithms are those that experience 5. 1. 3 only “ features ” but not a supervision signal. the distinction between supervised and unsupervised algorithms is not formally and rigidly deﬁned because there is no objective test for distinguishing


##vised algorithms are those that experience 5. 1. 3 only “ features ” but not a supervision signal. the distinction between supervised and unsupervised algorithms is not formally and rigidly deﬁned because there is no objective test for distinguishing whether a value is a feature or a target provided by a supervisor. informally, unsupervised learning refers to most attempts to extract information from a distribution that do not require human labor to annotate examples. the term is usually associated with density estimation, learning to draw samples from a distribution, learning to denoise data from some distribution, ﬁnding a manifold that the data lies near, or clustering the data into groups of 146


chapter 5. machine learning basics related examples. a classic unsupervised learning task is to ﬁnd the “ best ” representation of the data. by ‘ best ’ we can mean [UNK] things, but generally speaking we are looking for a representation that preserves as much information about x as possible while obeying some penalty or constraint aimed at keeping the representation or simpler more accessible than itself. x there are multiple ways of deﬁning a representation. three of the simpler most common include lower dimensional representations, sparse representations and independent representations. low - dimensional representations attempt to compress as much information about x as possible in a smaller representation. sparse representations (, ;, ; barlow 1989 olshausen and field 1996 hinton and ghahramani 1997, ) embed the dataset into a representation whose entries are mostly zeroes for most inputs. the use of sparse representations typically requires increasing the dimensionality of the representation, so that the representation becoming mostly zeroes does not discard too much information. this results in an overall structure of the representation that tends to distribute data along the axes of the representation space. independent representations attempt to disentangle the sources of variation underlying the data distribution such that the dimensions of the representation are statistically independent. of course these three criteria


overall structure of the representation that tends to distribute data along the axes of the representation space. independent representations attempt to disentangle the sources of variation underlying the data distribution such that the dimensions of the representation are statistically independent. of course these three criteria are certainly not mutually exclusive. low - dimensional representations often yield elements that have fewer or weaker de - pendencies than the original high - dimensional data. this is because one way to reduce the size of a representation is to ﬁnd and remove redundancies. identifying and removing more redundancy allows the dimensionality reduction algorithm to achieve more compression while discarding less information. the notion of representation is one of the central themes of deep learning and therefore one of the central themes in this book. in this section, we develop some simple examples of representation learning algorithms. together, these example algorithms show how to operationalize all three of the criteria above. most of the remaining chapters introduce additional representation learning algorithms that develop these criteria in [UNK] ways or introduce other criteria. 5. 8. 1 principal components analysis in section, we saw that the principal components analysis algorithm provides 2. 12 a means of compressing data. we can also view pca as an unsupervised learning algorithm that learns a representation of data. this


. 1 principal components analysis in section, we saw that the principal components analysis algorithm provides 2. 12 a means of compressing data. we can also view pca as an unsupervised learning algorithm that learns a representation of data. this representation is based on two of the criteria for a simple representation described above. pca learns a 147


chapter 5. machine learning basics − − 20 10 0 10 20 x1 −20 −10 0 10 20 x2 − − 20 10 0 10 20 z1 −20 −10 0 10 20 z2 figure 5. 8 : pca learns a linear projection that aligns the direction of greatest variance with the axes of the new space. ( left ) the original data consists of samples ofx. in this space, the variance might occur along directions that are not axis - aligned. ( right ) the transformed data z = xw now varies most along the axis z1. the direction of second most variance is now along z2. representation that has lower dimensionality than the original input. it also learns a representation whose elements have no linear correlation with each other. this is a ﬁrst step toward the criterion of learning representations whose elements are statistically independent. to achieve full independence, a representation learning algorithm must also remove the nonlinear relationships between variables. pca learns an orthogonal, linear transformation of the data that projects an input x to a representation z as shown in ﬁgure. in section, we saw that 5. 8 2. 12 we could learn a one - dimensional representation that best reconstructs the original data ( in the sense of mean squared error


input x to a representation z as shown in ﬁgure. in section, we saw that 5. 8 2. 12 we could learn a one - dimensional representation that best reconstructs the original data ( in the sense of mean squared error ) and that this representation actually corresponds to the ﬁrst principal component of the data. thus we can use pca as a simple and [UNK] dimensionality reduction method that preserves as much of the information in the data as possible ( again, as measured by least - squares reconstruction error ). in the following, we will study how the pca representation decorrelates the original data representation. x let us consider the m n × - dimensional design matrix x. we will assume that the data has a mean of zero, e [ x ] = 0. if this is not the case, the data can easily be centered by subtracting the mean from all examples in a preprocessing step. the unbiased sample covariance matrix associated with is given by : x var [ ] = x 1 m −1x x. ( 5. 85 ) 148


chapter 5. machine learning basics pca ﬁnds a representation ( through linear transformation ) z = xw where var [ ] z is diagonal. in section, we saw that the principal components of a design matrix 2. 12 x are given by the eigenvectors of xx. from this view, xx w w = λ. ( 5. 86 ) in this section, we exploit an alternative derivation of the principal components. the principal components may also be obtained via the singular value decomposition. speciﬁcally, they are the right singular vectors of x. to see this, let w be the right singular vectors in the decomposition x = u w σ. we then recover the original eigenvector equation with as the eigenvector basis : w x x = u w σ u w σ = wσ2w. ( 5. 87 ) the svd is helpful to show that pca results in a diagonal var [ z ]. using the svd of, we can express the variance of as : x x var [ ] = x 1 m −1xx ( 5. 88 ) = 1 m −1 ( u w σ ) u w σ ( 5. 89 ) = 1 m −1wσu u w σ ( 5.


: x x var [ ] = x 1 m −1xx ( 5. 88 ) = 1 m −1 ( u w σ ) u w σ ( 5. 89 ) = 1 m −1wσu u w σ ( 5. 90 ) = 1 m −1wσ2 w, ( 5. 91 ) where we use the fact that uu = i because the u matrix of the singular value decomposition is deﬁned to be orthogonal. this shows that if we take z = xw, we can ensure that the covariance of is diagonal as required : z var [ ] = z 1 m −1 zz ( 5. 92 ) = 1 m −1 w x xw ( 5. 93 ) = 1 m −1 w wσ2w w ( 5. 94 ) = 1 m −1 σ2, ( 5. 95 ) where this time we use the fact that w w = i, again from the deﬁnition of the svd. 149


chapter 5. machine learning basics the above analysis shows that when we project the data x to z, via the linear transformation w, the resulting representation has a diagonal covariance matrix ( as given by σ2 ) which immediately implies that the individual elements of z are mutually uncorrelated. this ability of pca to transform data into a representation where the elements are mutually uncorrelated is a very important property of pca. it is a simple example of a representation that attempts to disentangle the unknown factors of variation underlying the data. in the case of pca, this disentangling takes the form of ﬁnding a rotation of the input space ( described by w ) that aligns the principal axes of variance with the basis of the new representation space associated with. z while correlation is an important category of dependency between elements of the data, we are also interested in learning representations that disentangle more complicated forms of feature dependencies. for this, we will need more than what can be done with a simple linear transformation. 5. 8. 2 - means clustering k another example of a simple representation learning algorithm isk - means clustering. the k - means clustering algorithm divides the training set intok [UNK] clusters of examples that are near


a simple linear transformation. 5. 8. 2 - means clustering k another example of a simple representation learning algorithm isk - means clustering. the k - means clustering algorithm divides the training set intok [UNK] clusters of examples that are near each other. we can thus think of the algorithm as providing a k - dimensional one - hot code vector h representing an input x. if x belongs to cluster i, then hi = 1 and all other entries of the representation h are zero. the one - hot code provided by k - means clustering is an example of a sparse representation, because the majority of its entries are zero for every input. later, we will develop other algorithms that learn more ﬂexible sparse representations, where more than one entry can be non - zero for each input x. one - hot codes are an extreme example of sparse representations that lose many of the beneﬁts of a distributed representation. the one - hot code still confers some statistical advantages ( it naturally conveys the idea that all examples in the same cluster are similar to each other ) and it confers the computational advantage that the entire representation may be captured by a single integer. the k - means algorithm works by initializingk [UNK] centroids { µ ( 1 ),


all examples in the same cluster are similar to each other ) and it confers the computational advantage that the entire representation may be captured by a single integer. the k - means algorithm works by initializingk [UNK] centroids { µ ( 1 ),..., µ ( ) k } to [UNK] values, then alternating between two [UNK] steps until convergence. in one step, each training example is assigned to cluster i, where i is the index of the nearest centroid µ ( ) i. in the other step, each centroid µ ( ) i is updated to the mean of all training examples x ( ) j assigned to cluster. i 150


chapter 5. machine learning basics one [UNK] pertaining to clustering is that the clustering problem is inherently ill - posed, in the sense that there is no single criterion that measures how well a clustering of the data corresponds to the real world. we can measure properties of the clustering such as the average euclidean distance from a cluster centroid to the members of the cluster. this allows us to tell how well we are able to reconstruct the training data from the cluster assignments. we do not know how well the cluster assignments correspond to properties of the real world. moreover, there may be many [UNK] clusterings that all correspond well to some property of the real world. we may hope to ﬁnd a clustering that relates to one feature but obtain a [UNK], equally valid clustering that is not relevant to our task. for example, suppose that we run two clustering algorithms on a dataset consisting of images of red trucks, images of red cars, images of gray trucks, and images of gray cars. if we ask each clustering algorithm to ﬁnd two clusters, one algorithm may ﬁnd a cluster of cars and a cluster of trucks, while another may ﬁnd a cluster of red vehicles and a cluster of gray vehicles. suppose we also run a third cluster


ask each clustering algorithm to ﬁnd two clusters, one algorithm may ﬁnd a cluster of cars and a cluster of trucks, while another may ﬁnd a cluster of red vehicles and a cluster of gray vehicles. suppose we also run a third clustering algorithm, which is allowed to determine the number of clusters. this may assign the examples to four clusters, red cars, red trucks, gray cars, and gray trucks. this new clustering now at least captures information about both attributes, but it has lost information about similarity. red cars are in a [UNK] cluster from gray cars, just as they are in a [UNK] cluster from gray trucks. the output of the clustering algorithm does not tell us that red cars are more similar to gray cars than they are to gray trucks. they are [UNK] from both things, and that is all we know. these issues illustrate some of the reasons that we may prefer a distributed representation to a one - hot representation. a distributed representation could have two attributes for each vehicle — one representing its color and one representing whether it is a car or a truck. it is still not entirely clear what the optimal distributed representation is ( how can the learning algorithm know whether the two attributes we are interested in are color and car - versus - truck rather than manufacturer


and one representing whether it is a car or a truck. it is still not entirely clear what the optimal distributed representation is ( how can the learning algorithm know whether the two attributes we are interested in are color and car - versus - truck rather than manufacturer and age? ) but having many attributes reduces the burden on the algorithm to guess which single attribute we care about, and allows us to measure similarity between objects in a ﬁne - grained way by comparing many attributes instead of just testing whether one attribute matches. 5. 9 stochastic gradient descent nearly all of deep learning is powered by one very important algorithm : stochastic gradient descent or sgd. stochastic gradient descent is an extension of the 151


chapter 5. machine learning basics gradient descent algorithm introduced in section. 4. 3 a recurring problem in machine learning is that large training sets are necessary for good generalization, but large training sets are also more computationally expensive. the cost function used by a machine learning algorithm often decomposes as a sum over training examples of some per - example loss function. for example, the negative conditional log - likelihood of the training data can be written as j ( ) = θ ex, [UNK], y, ( x θ ) = 1 m m i = 1 l ( x ( ) i, y ( ) i, θ ) ( 5. 96 ) where is the per - example loss l l, y, p y. ( x θ ) = log − ( | x θ ; ) for these additive cost functions, gradient descent requires computing ∇θj ( ) = θ 1 m m i = 1 ∇θl ( x ( ) i, y ( ) i,. θ ) ( 5. 97 ) the computational cost of this operation is o ( m ). as the training set size grows to billions of examples, the time to take a single gradient step becomes prohibitively long. the insight of stochastic gradient descent is that the gradient is an expectation. the


of this operation is o ( m ). as the training set size grows to billions of examples, the time to take a single gradient step becomes prohibitively long. the insight of stochastic gradient descent is that the gradient is an expectation. the expectation may be approximately estimated using a small set of samples. speciﬁcally, on each step of the algorithm, we can sample a minibatch of examples b = { x ( 1 ),..., x ( m ) } drawn uniformly from the training set. the minibatch size mis typically chosen to be a relatively small number of examples, ranging from 1 to a few hundred. crucially, mis usually held ﬁxed as the training set size m grows. we may ﬁt a training set with billions of examples using updates computed on only a hundred examples. the estimate of the gradient is formed as g = 1 m∇θ m i = 1 l ( x ( ) i, y ( ) i,. θ ) ( 5. 98 ) using examples from the minibatch. the stochastic gradient descent algorithm b then follows the estimated gradient downhill : θ θ g ← −, ( 5. 99 ) where is the learning rate. 152


chapter 5. machine learning basics gradient descent in general has often been regarded as slow or unreliable. in the past, the application of gradient descent to non - convex optimization problems was regarded as foolhardy or unprincipled. today, we know that the machine learning models described in part work very well when trained with gradient ii descent. the optimization algorithm may not be guaranteed to arrive at even a local minimum in a reasonable amount of time, but it often ﬁnds a very low value of the cost function quickly enough to be useful. stochastic gradient descent has many important uses outside the context of deep learning. it is the main way to train large linear models on very large datasets. for a ﬁxed model size, the cost per sgd update does not depend on the training set size m. in practice, we often use a larger model as the training set size increases, but we are not forced to do so. the number of updates required to reach convergence usually increases with training set size. however, as m approaches inﬁnity, the model will eventually converge to its best possible test error before sgd has sampled every example in the training set. increasing m further will not extend the amount of training time needed to reach the model ’ s best


, as m approaches inﬁnity, the model will eventually converge to its best possible test error before sgd has sampled every example in the training set. increasing m further will not extend the amount of training time needed to reach the model ’ s best possible test error. from this point of view, one can argue that the asymptotic cost of training a model with sgd is as a function of. o ( 1 ) m prior to the advent of deep learning, the main way to learn nonlinear models was to use the kernel trick in combination with a linear model. many kernel learning algorithms require constructing an m m × matrix gi, j = k ( x ( ) i, x ( ) j ). constructing this matrix has computational cost o ( m2 ), which is clearly undesirable for datasets with billions of examples. in academia, starting in 2006, deep learning was initially interesting because it was able to generalize to new examples better than competing algorithms when trained on medium - sized datasets with tens of thousands of examples. soon after, deep learning garnered additional interest in industry, because it provided a scalable way of training nonlinear models on large datasets. stochastic gradient descent and many enhancements to it are described further in


with tens of thousands of examples. soon after, deep learning garnered additional interest in industry, because it provided a scalable way of training nonlinear models on large datasets. stochastic gradient descent and many enhancements to it are described further in chapter. 8 5. 10 building a machine learning algorithm nearly all deep learning algorithms can be described as particular instances of a fairly simple recipe : combine a speciﬁcation of a dataset, a cost function, an optimization procedure and a model. for example, the linear regression algorithm combines a dataset consisting of 153


chapter 5. machine learning basics x y and, the cost function j, b ( w ) = −ex, [UNK] log pmodel ( ) y | x, ( 5. 100 ) the model speciﬁcation pmodel ( y | x ) = n ( y ; xw + b, 1 ), and, in most cases, the optimization algorithm deﬁned by solving for where the gradient of the cost is zero using the normal equations. by realizing that we can replace any of these components mostly independently from the others, we can obtain a very wide variety of algorithms. the cost function typically includes at least one term that causes the learning process to perform statistical estimation. the most common cost function is the negative log - likelihood, so that minimizing the cost function causes maximum likelihood estimation. the cost function may also include additional terms, such as regularization terms. for example, we can add weight decay to the linear regression cost function to obtain j, b λ ( w ) = | | | | w 2 2 −ex, [UNK] log pmodel ( ) y | x. ( 5. 101 ) this still allows closed - form optimization. if we change the model to be nonlinear, then most cost functions can no longer be optimi


| w 2 2 −ex, [UNK] log pmodel ( ) y | x. ( 5. 101 ) this still allows closed - form optimization. if we change the model to be nonlinear, then most cost functions can no longer be optimized in closed form. this requires us to choose an iterative numerical optimization procedure, such as gradient descent. the recipe for constructing a learning algorithm by combining models, costs, and optimization algorithms supports both supervised and unsupervised learning. the linear regression example shows how to support supervised learning. unsupervised learning can be supported by deﬁning a dataset that contains only x and providing an appropriate unsupervised cost and model. for example, we can obtain the ﬁrst pca vector by specifying that our loss function is j ( ) = w [UNK] | | − | | x r ( ; ) x w 2 2 ( 5. 102 ) while our model is deﬁned to have w with norm one and reconstruction function r ( ) = x wxw. in some cases, the cost function may be a function that we cannot actually evaluate, for computational reasons. in these cases, we can still approximately minimize it using iterative numerical optimization so long as we have some way


) = x wxw. in some cases, the cost function may be a function that we cannot actually evaluate, for computational reasons. in these cases, we can still approximately minimize it using iterative numerical optimization so long as we have some way of approximating its gradients. most machine learning algorithms make use of this recipe, though it may not immediately be obvious. if a machine learning algorithm seems especially unique or 154