mecha - nism. memory networks originally required a supervision signal instructing them how to use their memory cells. graves 2014b et al. ( ) introduced the neural turing machine, which is able to learn to read from and write arbitrary content to memory cells without explicit supervision about which actions to undertake, and allowed end - to - end training without this supervision signal, via the use of a content - based soft attention mechanism ( see ( ) and sec - bahdanau et al. 2015 tion ). this soft addressing mechanism has become standard with other 12. 4. 5. 1 related architectures emulating algorithmic mechanisms in a way that still allows gradient - based optimization (, ; sukhbaatar et al. 2015 joulin and mikolov 2015, ; kumar 2015 vinyals 2015a grefenstette 2015 et al., ; et al., ; et al., ). each memory cell can be thought of as an extension of the memory cells in lstms and grus. the [UNK] is that the network outputs an internal state that chooses which cell to read from or write to, just as memory accesses in a digital computer read from or write to a speciﬁc address. it is [UNK] to opt


grus. the [UNK] is that the network outputs an internal state that chooses which cell to read from or write to, just as memory accesses in a digital computer read from or write to a speciﬁc address. it is [UNK] to optimize functions that produce exact, integer addresses. to alleviate this problem, ntms actually read to or write from many memory cells simultaneously. to read, they take a weighted average of many cells. to write, they modify multiple cells by [UNK] amounts. the [UNK] for these operations are chosen to be focused on a small number of cells, for example, by producing them via a softmax function. using these weights with non - zero derivatives allows the functions controlling access to the memory to be optimized using gradient descent. the gradient on these [UNK] indicates whether each of them should be increased or decreased, but the gradient will typically be large only for those memory addresses receiving a large [UNK]. these memory cells are typically augmented to contain a vector, rather than 418

/home/ricoiban/GEMMA/mnlp_chatsplaining/RAG/Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville (z-lib.org).pdf


chapter 10. sequence modeling : recurrent and recursive nets the single scalar stored by an lstm or gru memory cell. there are two reasons to increase the size of the memory cell. one reason is that we have increased the cost of accessing a memory cell. we pay the computational cost of producing a [UNK] for many cells, but we expect these [UNK] to cluster around a small number of cells. by reading a vector value, rather than a scalar value, we can [UNK] some of this cost. another reason to use vector - valued memory cells is that they allow for content - based addressing, where the weight used to read to or write from a cell is a function of that cell. vector - valued cells allow us to retrieve a complete vector - valued memory if we are able to produce a pattern that matches some but not all of its elements. this is analogous to the way that people can recall the lyrics of a song based on a few words. we can think of a content - based read instruction as saying, “ retrieve the lyrics of the song that has the chorus ‘ we all live in a yellow submarine. ’ ” content - based addressing is more useful when we make the objects to be retrieved large — if every letter of the song was


instruction as saying, “ retrieve the lyrics of the song that has the chorus ‘ we all live in a yellow submarine. ’ ” content - based addressing is more useful when we make the objects to be retrieved large — if every letter of the song was stored in a separate memory cell, we would not be able to ﬁnd them this way. by comparison, location - based addressing is not allowed to refer to the content of the memory. we can think of a location - based read instruction as saying “ retrieve the lyrics of the song in slot 347. ” location - based addressing can often be a perfectly sensible mechanism even when the memory cells are small. if the content of a memory cell is copied ( not forgotten ) at most time steps, then the information it contains can be propagated forward in time and the gradients propagated backward in time without either vanishing or exploding. the explicit memory approach is illustrated in ﬁgure, where we see that 10. 18 a “ task neural network ” is coupled with a memory. although that task neural network could be feedforward or recurrent, the overall system is a recurrent network. the task network can choose to read from or write to speciﬁc memory addresses. explicit memory seems to allow models


a memory. although that task neural network could be feedforward or recurrent, the overall system is a recurrent network. the task network can choose to read from or write to speciﬁc memory addresses. explicit memory seems to allow models to learn tasks that ordinary rnns or lstm rnns cannot learn. one reason for this advantage may be because information and gradients can be propagated ( forward in time or backwards in time, respectively ) for very long durations. as an alternative to back - propagation through weighted averages of memory cells, we can interpret the memory addressing [UNK] as probabilities and stochastically read just one cell ( zaremba and sutskever 2015, ). optimizing models that make discrete decisions requires specialized optimization algorithms, described in section. so far, training these stochastic architectures that make discrete 20. 9. 1 decisions remains harder than training deterministic algorithms that make soft decisions. whether it is soft ( allowing back - propagation ) or stochastic and hard, the 419


chapter 10. sequence modeling : recurrent and recursive nets mechanism for choosing an address is in its form identical to the attention mechanism which had been previously introduced in the context of machine translation (, ) and discussed in section. the idea bahdanau et al. 2015 12. 4. 5. 1 of attention mechanisms for neural networks was introduced even earlier, in the context of handwriting generation ( graves 2013, ), with an attention mechanism that was constrained to move only forward in time through the sequence. in the case of machine translation and memory networks, at each step, the focus of attention can move to a completely [UNK] place, compared to the previous step. recurrent neural networks provide a way to extend deep learning to sequential data. they are the last major tool in our deep learning toolbox. our discussion now moves to how to choose and use these tools and how to apply them to real - world tasks. 420


chapter 11 practical methodology successfully applying deep learning techniques requires more than just a good knowledge of what algorithms exist and the principles that explain how they work. a good machine learning practitioner also needs to know how to choose an algorithm for a particular application and how to monitor and respond to feedback obtained from experiments in order to improve a machine learning system. during day to day development of machine learning systems, practitioners need to decide whether to gather more data, increase or decrease model capacity, add or remove regularizing features, improve the optimization of a model, improve approximate inference in a model, or debug the software implementation of the model. all of these operations are at the very least time - consuming to try out, so it is important to be able to determine the right course of action rather than blindly guessing. most of this book is about [UNK] machine learning models, training algo - rithms, and objective functions. this may give the impression that the most important ingredient to being a machine learning expert is knowing a wide variety of machine learning techniques and being good at [UNK] kinds of math. in prac - tice, one can usually do much better with a correct application of a commonplace algorithm than by sloppily applying an obscure algorithm. correct application of an algorithm depends on


learning techniques and being good at [UNK] kinds of math. in prac - tice, one can usually do much better with a correct application of a commonplace algorithm than by sloppily applying an obscure algorithm. correct application of an algorithm depends on mastering some fairly simple methodology. many of the recommendations in this chapter are adapted from ( ). ng 2015 we recommend the following practical design process : • determine your goals — what error metric to use, and your target value for this error metric. these goals and error metrics should be driven by the problem that the application is intended to solve. • establish a working end - to - end pipeline as soon as possible, including the 421


chapter 11. practical methodology estimation of the appropriate performance metrics. • instrument the system well to determine bottlenecks in performance. diag - nose which components are performing worse than expected and whether it is due to overﬁtting, underﬁtting, or a defect in the data or software. • repeatedly make incremental changes such as gathering new data, adjusting hyperparameters, or changing algorithms, based on speciﬁc ﬁndings from your instrumentation. as a running example, we will use street view address number transcription system (, ). the purpose of this application is to add goodfellow et al. 2014d buildings to google maps. street view cars photograph the buildings and record the gps coordinates associated with each photograph. a convolutional network recognizes the address number in each photograph, allowing the google maps database to add that address in the correct location. the story of how this commercial application was developed gives an example of how to follow the design methodology we advocate. we now describe each of the steps in this process. 11. 1 performance metrics determining your goals, in terms of which error metric to use, is a necessary ﬁrst step because your error metric will guide all of your future actions. you should also have an idea


of the steps in this process. 11. 1 performance metrics determining your goals, in terms of which error metric to use, is a necessary ﬁrst step because your error metric will guide all of your future actions. you should also have an idea of what level of performance you desire. keep in mind that for most applications, it is impossible to achieve absolute zero error. the bayes error deﬁnes the minimum error rate that you can hope to achieve, even if you have inﬁnite training data and can recover the true probability distribution. this is because your input features may not contain complete information about the output variable, or because the system might be intrinsically stochastic. you will also be limited by having a ﬁnite amount of training data. the amount of training data can be limited for a variety of reasons. when your goal is to build the best possible real - world product or service, you can typically collect more data but must determine the value of reducing error further and weigh this against the cost of collecting more data. data collection can require time, money, or human [UNK] ( for example, if your data collection process involves performing invasive medical tests ). when your goal is to answer a scientiﬁc question about which algorithm performs better


of collecting more data. data collection can require time, money, or human [UNK] ( for example, if your data collection process involves performing invasive medical tests ). when your goal is to answer a scientiﬁc question about which algorithm performs better on a ﬁxed benchmark, the benchmark 422


chapter 11. practical methodology speciﬁcation usually determines the training set and you are not allowed to collect more data. how can one determine a reasonable level of performance to expect? typically, in the academic setting, we have some estimate of the error rate that is attainable based on previously published benchmark results. in the real - word setting, we have some idea of the error rate that is necessary for an application to be safe, cost - [UNK], or appealing to consumers. once you have determined your realistic desired error rate, your design decisions will be guided by reaching this error rate. another important consideration besides the target value of the performance metric is the choice of which metric to use. several [UNK] performance metrics may be used to measure the [UNK] of a complete application that includes machine learning components. these performance metrics are usually [UNK] from the cost function used to train the model. as described in section, it is 5. 1. 2 common to measure the accuracy, or equivalently, the error rate, of a system. however, many applications require more advanced metrics. sometimes it is much more costly to make one kind of a mistake than another. for example, an e - mail spam detection system can make two kinds of mistakes : incorrectly classifying a


system. however, many applications require more advanced metrics. sometimes it is much more costly to make one kind of a mistake than another. for example, an e - mail spam detection system can make two kinds of mistakes : incorrectly classifying a legitimate message as spam, and incorrectly allowing a spam message to appear in the inbox. it is much worse to block a legitimate message than to allow a questionable message to pass through. rather than measuring the error rate of a spam classiﬁer, we may wish to measure some form of total cost, where the cost of blocking legitimate messages is higher than the cost of allowing spam messages. sometimes we wish to train a binary classiﬁer that is intended to detect some rare event. for example, we might design a medical test for a rare disease. suppose that only one in every million people has this disease. we can easily achieve 99. 9999 % accuracy on the detection task, by simply hard - coding the classiﬁer to always report that the disease is absent. clearly, accuracy is a poor way to characterize the performance of such a system. one way to solve this problem is to instead measure precision and recall. precision is the fraction of detections reported by the model that


report that the disease is absent. clearly, accuracy is a poor way to characterize the performance of such a system. one way to solve this problem is to instead measure precision and recall. precision is the fraction of detections reported by the model that were correct, while recall is the fraction of true events that were detected. a detector that says no one has the disease would achieve perfect precision, but zero recall. a detector that says everyone has the disease would achieve perfect recall, but precision equal to the percentage of people who have the disease ( 0. 0001 % in our example of a disease that only one people in a million have ). when using precision and recall, it is common to plot a pr curve, with precision on the y - axis and recall on the x - axis. the classiﬁer generates a score that is higher if the event to be detected occurred. for example, a feedforward 423


chapter 11. practical methodology network designed to detect a disease outputs [UNK] = p ( y = 1 | x ), estimating the probability that a person whose medical results are described by features x has the disease. we choose to report a detection whenever this score exceeds some threshold. by varying the threshold, we can trade precision for recall. in many cases, we wish to summarize the performance of the classiﬁer with a single number rather than a curve. to do so, we can convert precision p and recall r into an f - score given by f = 2pr p r +. ( 11. 1 ) another option is to report the total area lying beneath the pr curve. in some applications, it is possible for the machine learning system to refuse to make a decision. this is useful when the machine learning algorithm can estimate how conﬁdent it should be about a decision, especially if a wrong decision can be harmful and if a human operator is able to occasionally take over. the street view transcription system provides an example of this situation. the task is to transcribe the address number from a photograph in order to associate the location where the photo was taken with the correct address in a map. because the value of the map degrades considerably if the


an example of this situation. the task is to transcribe the address number from a photograph in order to associate the location where the photo was taken with the correct address in a map. because the value of the map degrades considerably if the map is inaccurate, it is important to add an address only if the transcription is correct. if the machine learning system thinks that it is less likely than a human being to obtain the correct transcription, then the best course of action is to allow a human to transcribe the photo instead. of course, the machine learning system is only useful if it is able to dramatically reduce the amount of photos that the human operators must process. a natural performance metric to use in this situation is coverage. coverage is the fraction of examples for which the machine learning system is able to produce a response. it is possible to trade coverage for accuracy. one can always obtain 100 % accuracy by refusing to process any example, but this reduces the coverage to 0 %. for the street view task, the goal for the project was to reach human - level transcription accuracy while maintaining 95 % coverage. human - level performance on this task is 98 % accuracy. many other metrics are possible. we can for example, measure click - through rates, collect user


goal for the project was to reach human - level transcription accuracy while maintaining 95 % coverage. human - level performance on this task is 98 % accuracy. many other metrics are possible. we can for example, measure click - through rates, collect user satisfaction surveys, and so on. many specialized application areas have application - speciﬁc criteria as well. what is important is to determine which performance metric to improve ahead of time, then concentrate on improving this metric. without clearly deﬁned goals, it can be [UNK] to tell whether changes to a machine learning system make progress or not. 424


chapter 11. practical methodology 11. 2 default baseline models after choosing performance metrics and goals, the next step in any practical application is to establish a reasonable end - to - end system as soon as possible. in this section, we provide recommendations for which algorithms to use as the ﬁrst baseline approach in various situations. keep in mind that deep learning research progresses quickly, so better default algorithms are likely to become available soon after this writing. depending on the complexity of your problem, you may even want to begin without using deep learning. if your problem has a chance of being solved by just choosing a few linear weights correctly, you may want to begin with a simple statistical model like logistic regression. if you know that your problem falls into an “ ai - complete ” category like object recognition, speech recognition, machine translation, and so on, then you are likely to do well by beginning with an appropriate deep learning model. first, choose the general category of model based on the structure of your data. if you want to perform supervised learning with ﬁxed - size vectors as input, use a feedforward network with fully connected layers. if the input has known topological structure ( for example, if the input is an image ), use a convolutional network.


supervised learning with ﬁxed - size vectors as input, use a feedforward network with fully connected layers. if the input has known topological structure ( for example, if the input is an image ), use a convolutional network. in these cases, you should begin by using some kind of piecewise linear unit ( relus or their generalizations like leaky relus, prelus and maxout ). if your input or output is a sequence, use a gated recurrent net ( lstm or gru ). a reasonable choice of optimization algorithm is sgd with momentum with a decaying learning rate ( popular decay schemes that perform better or worse on [UNK] problems include decaying linearly until reaching a ﬁxed minimum learning rate, decaying exponentially, or decreasing the learning rate by a factor of 2 - 10 each time validation error plateaus ). another very reasonable alternative is adam. batch normalization can have a dramatic [UNK] on optimization performance, especially for convolutional networks and networks with sigmoidal nonlinearities. while it is reasonable to omit batch normalization from the very ﬁrst baseline, it should be introduced quickly if optimization appears to be problematic. unless your training set contains tens of millions of examples or


and networks with sigmoidal nonlinearities. while it is reasonable to omit batch normalization from the very ﬁrst baseline, it should be introduced quickly if optimization appears to be problematic. unless your training set contains tens of millions of examples or more, you should include some mild forms of regularization from the start. early stopping should be used almost universally. dropout is an excellent regularizer that is easy to implement and compatible with many models and training algorithms. batch normalization also sometimes reduces generalization error and allows dropout to be omitted, due to the noise in the estimate of the statistics used to normalize each variable. 425


chapter 11. practical methodology if your task is similar to another task that has been studied extensively, you will probably do well by ﬁrst copying the model and algorithm that is already known to perform best on the previously studied task. you may even want to copy a trained model from that task. for example, it is common to use the features from a convolutional network trained on imagenet to solve other computer vision tasks (, ). girshick et al. 2015 a common question is whether to begin by using unsupervised learning, de - scribed further in part. this is somewhat domain speciﬁc. some domains, such iii as natural language processing, are known to beneﬁt tremendously from unsuper - vised learning techniques such as learning unsupervised word embeddings. in other domains, such as computer vision, current unsupervised learning techniques do not bring a beneﬁt, except in the semi - supervised setting, when the number of labeled examples is very small (, ; kingma et al. 2014 rasmus 2015 et al., ). if your application is in a context where unsupervised learning is known to be important, then include


, when the number of labeled examples is very small (, ; kingma et al. 2014 rasmus 2015 et al., ). if your application is in a context where unsupervised learning is known to be important, then include it in your ﬁrst end - to - end baseline. otherwise, only use unsupervised learning in your ﬁrst attempt if the task you want to solve is unsupervised. you can always try adding unsupervised learning later if you observe that your initial baseline overﬁts. 11. 3 determining whether to gather more data after the ﬁrst end - to - end system is established, it is time to measure the perfor - mance of the algorithm and determine how to improve it. many machine learning novices are tempted to make improvements by trying out many [UNK] algorithms. however, it is often much better to gather more data than to improve the learning algorithm. how does one decide whether to gather more data? first, determine whether the performance on the training set is acceptable. if performance on the training set is poor, the learning algorithm is not using the training data that is already available, so there is no reason to gather more data. instead, try increasing the size of the


the performance on the training set is acceptable. if performance on the training set is poor, the learning algorithm is not using the training data that is already available, so there is no reason to gather more data. instead, try increasing the size of the model by adding more layers or adding more hidden units to each layer. also, try improving the learning algorithm, for example by tuning the learning rate hyperparameter. if large models and carefully tuned optimization algorithms do not work well, then the problem might be the of the training data. the quality data may be too noisy or may not include the right inputs needed to predict the desired outputs. this suggests starting over, collecting cleaner data or collecting a richer set of features. if the performance on the training set is acceptable, then measure the per - 426


chapter 11. practical methodology formance on a test set. if the performance on the test set is also acceptable, then there is nothing left to be done. if test set performance is much worse than training set performance, then gathering more data is one of the most [UNK] solutions. the key considerations are the cost and feasibility of gathering more data, the cost and feasibility of reducing the test error by other means, and the amount of data that is expected to be necessary to improve test set performance signiﬁcantly. at large internet companies with millions or billions of users, it is feasible to gather large datasets, and the expense of doing so can be considerably less than the other alternatives, so the answer is almost always to gather more training data. for example, the development of large labeled datasets was one of the most important factors in solving object recognition. in other contexts, such as medical applications, it may be costly or infeasible to gather more data. a simple alternative to gathering more data is to reduce the size of the model or improve regularization, by adjusting hyperparameters such as weight decay [UNK], or by adding regularization strategies such as dropout. if you ﬁnd that the gap between train and test performance is still unacceptable


to reduce the size of the model or improve regularization, by adjusting hyperparameters such as weight decay [UNK], or by adding regularization strategies such as dropout. if you ﬁnd that the gap between train and test performance is still unacceptable even after tuning the regularization hyperparameters, then gathering more data is advisable. when deciding whether to gather more data, it is also necessary to decide how much to gather. it is helpful to plot curves showing the relationship between training set size and generalization error, like in ﬁgure. by extrapolating such 5. 4 curves, one can predict how much additional training data would be needed to achieve a certain level of performance. usually, adding a small fraction of the total number of examples will not have a noticeable impact on generalization error. it is therefore recommended to experiment with training set sizes on a logarithmic scale, for example doubling the number of examples between consecutive experiments. if gathering much more data is not feasible, the only other way to improve generalization error is to improve the learning algorithm itself. this becomes the domain of research and not the domain of advice for applied practitioners. 11. 4 selecting hyperparameters most deep learning algorithms come with many hyperparameters that


way to improve generalization error is to improve the learning algorithm itself. this becomes the domain of research and not the domain of advice for applied practitioners. 11. 4 selecting hyperparameters most deep learning algorithms come with many hyperparameters that control many aspects of the algorithm ’ s behavior. some of these hyperparameters [UNK] the time and memory cost of running the algorithm. some of these hyperparameters [UNK] the quality of the model recovered by the training process and its ability to infer correct results when deployed on new inputs. there are two basic approaches to choosing these hyperparameters : choosing them manually and choosing them automatically. choosing the hyperparameters 427


chapter 11. practical methodology manually requires understanding what the hyperparameters do and how machine learning models achieve good generalization. automatic hyperparameter selection algorithms greatly reduce the need to understand these ideas, but they are often much more computationally costly. 11. 4. 1 manual hyperparameter tuning to set hyperparameters manually, one must understand the relationship between hyperparameters, training error, generalization error and computational resources ( memory and runtime ). this means establishing a solid foundation on the fun - damental ideas concerning the [UNK] capacity of a learning algorithm from chapter. 5 the goal of manual hyperparameter search is usually to ﬁnd the lowest general - ization error subject to some runtime and memory budget. we do not discuss how to determine the runtime and memory impact of various hyperparameters here because this is highly platform - dependent. the primary goal of manual hyperparameter search is to adjust the [UNK] capacity of the model to match the complexity of the task. [UNK] capacity is constrained by three factors : the representational capacity of the model, the ability of the learning algorithm to successfully minimize the cost function used to train the model, and the degree to which the cost function and training procedure regularize the model. a model with


constrained by three factors : the representational capacity of the model, the ability of the learning algorithm to successfully minimize the cost function used to train the model, and the degree to which the cost function and training procedure regularize the model. a model with more layers and more hidden units per layer has higher representational capacity — it is capable of representing more complicated functions. it can not necessarily actually learn all of these functions though, if the training algorithm cannot discover that certain functions do a good job of minimizing the training cost, or if regularization terms such as weight decay forbid some of these functions. the generalization error typically follows a u - shaped curve when plotted as a function of one of the hyperparameters, as in ﬁgure. at one extreme, the 5. 3 hyperparameter value corresponds to low capacity, and generalization error is high because training error is high. this is the underﬁtting regime. at the other extreme, the hyperparameter value corresponds to high capacity, and the generalization error is high because the gap between training and test error is high. somewhere in the middle lies the optimal model capacity, which achieves the lowest possible generalization error, by adding a medium generalization gap to a medium amount of training error


generalization error is high because the gap between training and test error is high. somewhere in the middle lies the optimal model capacity, which achieves the lowest possible generalization error, by adding a medium generalization gap to a medium amount of training error. for some hyperparameters, overﬁtting occurs when the value of the hyper - parameter is large. the number of hidden units in a layer is one such example, 428


chapter 11. practical methodology because increasing the number of hidden units increases the capacity of the model. for some hyperparameters, overﬁtting occurs when the value of the hyperparame - ter is small. for example, the smallest allowable weight decay [UNK] of zero corresponds to the greatest [UNK] capacity of the learning algorithm. not every hyperparameter will be able to explore the entire u - shaped curve. many hyperparameters are discrete, such as the number of units in a layer or the number of linear pieces in a maxout unit, so it is only possible to visit a few points along the curve. some hyperparameters are binary. usually these hyperparameters are switches that specify whether or not to use some optional component of the learning algorithm, such as a preprocessing step that normalizes the input features by subtracting their mean and dividing by their standard deviation. these hyperparameters can only explore two points on the curve. other hyperparameters have some minimum or maximum value that prevents them from exploring some part of the curve. for example, the minimum weight decay [UNK] is zero. this means that if the model is underﬁtting when weight decay is zero, we can not enter the overﬁtting region


or maximum value that prevents them from exploring some part of the curve. for example, the minimum weight decay [UNK] is zero. this means that if the model is underﬁtting when weight decay is zero, we can not enter the overﬁtting region by modifying the weight decay [UNK]. in other words, some hyperparameters can only subtract capacity. the learning rate is perhaps the most important hyperparameter. if you have time to tune only one hyperparameter, tune the learning rate. it con - trols the [UNK] capacity of the model in a more complicated way than other hyperparameters — the [UNK] capacity of the model is highest when the learning rate is correct for the optimization problem, not when the learning rate is especially large or especially small. the learning rate has a u - shaped curve for training error, illustrated in ﬁgure. when the learning rate is too large, gradient descent 11. 1 can inadvertently increase rather than decrease the training error. in the idealized quadratic case, this occurs if the learning rate is at least twice as large as its optimal value (, ). when the learning rate is too small, training lecun et al. 1998a is not only slower, but may become permanently stuck with a high


this occurs if the learning rate is at least twice as large as its optimal value (, ). when the learning rate is too small, training lecun et al. 1998a is not only slower, but may become permanently stuck with a high training error. this [UNK] is poorly understood ( it would not happen for a convex loss function ). tuning the parameters other than the learning rate requires monitoring both training and test error to diagnose whether your model is overﬁtting or underﬁtting, then adjusting its capacity appropriately. if your error on the training set is higher than your target error rate, you have no choice but to increase capacity. if you are not using regularization and you are conﬁdent that your optimization algorithm is performing correctly, then you must add more layers to your network or add more hidden units. unfortunately, this increases the computational costs associated with the model. if your error on the test set is higher than than your target error rate, you can 429


chapter 11. practical methodology 10−2 10−1 100 learning rate ( logarithmic scale ) 0 1 2 3 4 5 6 7 8 training error figure 11. 1 : typical relationship between the learning rate and the training error. notice the sharp rise in error when the learning is above an optimal value. this is for a ﬁxed training time, as a smaller learning rate may sometimes only slow down training by a factor proportional to the learning rate reduction. generalization error can follow this curve or be complicated by regularization [UNK] arising out of having a too large or too small learning rates, since poor optimization can, to some degree, reduce or prevent overﬁtting, and even points with equivalent training error can have [UNK] generalization error. now take two kinds of actions. the test error is the sum of the training error and the gap between training and test error. the optimal test error is found by trading [UNK] quantities. neural networks typically perform best when the training error is very low ( and thus, when capacity is high ) and the test error is primarily driven by the gap between train and test error. your goal is to reduce this gap without increasing training error faster than the gap decreases. to reduce the gap, change regularization hyperparameters to reduce [UNK] model


) and the test error is primarily driven by the gap between train and test error. your goal is to reduce this gap without increasing training error faster than the gap decreases. to reduce the gap, change regularization hyperparameters to reduce [UNK] model capacity, such as by adding dropout or weight decay. usually the best performance comes from a large model that is regularized well, for example by using dropout. most hyperparameters can be set by reasoning about whether they increase or decrease model capacity. some examples are included in table. 11. 1 while manually tuning hyperparameters, do not lose sight of your end goal : good performance on the test set. adding regularization is only one way to achieve this goal. as long as you have low training error, you can always reduce general - ization error by collecting more training data. the brute force way to practically guarantee success is to continually increase model capacity and training set size until the task is solved. this approach does of course increase the computational cost of training and inference, so it is only feasible given appropriate resources. in 430


chapter 11. practical methodology hyperparameter increases capacity when... reason caveats number of hid - den units increased increasing the number of hidden units increases the representational capacity of the model. increasing the number of hidden units increases both the time and memory cost of essentially every op - eration on the model. learning rate tuned op - timally an improper learning rate, whether too high or too low, results in a model with low [UNK] capacity due to optimization failure convolution ker - nel width increased increasing the kernel width increases the number of pa - rameters in the model a wider kernel results in a narrower output dimen - sion, reducing model ca - pacity unless you use im - plicit zero padding to re - duce this [UNK]. wider kernels require more mem - ory for parameter storage and increase runtime, but a narrower output reduces memory cost. implicit zero padding increased adding implicit zeros be - fore convolution keeps the representation size large increased time and mem - ory cost of most opera - tions. weight decay co - [UNK] decreased decreasing the weight de - cay [UNK] frees the model parameters to be - come larger dropout rate decreased dropping units less often gives


large increased time and mem - ory cost of most opera - tions. weight decay co - [UNK] decreased decreasing the weight de - cay [UNK] frees the model parameters to be - come larger dropout rate decreased dropping units less often gives the units more oppor - tunities to “ conspire ” with each other to ﬁt the train - ing set table 11. 1 : the [UNK] of various hyperparameters on model capacity. 431


chapter 11. practical methodology principle, this approach could fail due to optimization [UNK], but for many problems optimization does not seem to be a signiﬁcant barrier, provided that the model is chosen appropriately. 11. 4. 2 automatic hyperparameter optimization algorithms the ideal learning algorithm just takes a dataset and outputs a function, without requiring hand - tuning of hyperparameters. the popularity of several learning algorithms such as logistic regression and svms stems in part from their ability to perform well with only one or two tuned hyperparameters. neural networks can sometimes perform well with only a small number of tuned hyperparameters, but often beneﬁt signiﬁcantly from tuning of forty or more hyperparameters. manual hyperparameter tuning can work very well when the user has a good starting point, such as one determined by others having worked on the same type of application and architecture, or when the user has months or years of experience in exploring hyperparameter values for neural networks applied to similar tasks. however, for many applications, these starting points are not available. in these cases, automated algorithms can ﬁnd useful values of the hyperparameters. if we think about the way in which the user of a learning algorithm


applied to similar tasks. however, for many applications, these starting points are not available. in these cases, automated algorithms can ﬁnd useful values of the hyperparameters. if we think about the way in which the user of a learning algorithm searches for good values of the hyperparameters, we realize that an optimization is taking place : we are trying to ﬁnd a value of the hyperparameters that optimizes an objective function, such as validation error, sometimes under constraints ( such as a budget for training time, memory or recognition time ). it is therefore possible, in principle, to develop hyperparameter optimization algorithms that wrap a learning algorithm and choose its hyperparameters, thus hiding the hyperparameters of the learning algorithm from the user. unfortunately, hyperparameter optimization algorithms often have their own hyperparameters, such as the range of values that should be explored for each of the learning algorithm ’ s hyperparameters. however, these secondary hyperparameters are usually easier to choose, in the sense that acceptable performance may be achieved on a wide range of tasks using the same secondary hyperparameters for all tasks. 11. 4. 3 grid search when there are three or fewer hyperparameters,


usually easier to choose, in the sense that acceptable performance may be achieved on a wide range of tasks using the same secondary hyperparameters for all tasks. 11. 4. 3 grid search when there are three or fewer hyperparameters, the common practice is to perform grid search. for each hyperparameter, the user selects a small ﬁnite set of values to explore. the grid search algorithm then trains a model for every joint speciﬁcation of hyperparameter values in the cartesian product of the set of values for each individual hyperparameter. the experiment that yields the best validation 432


chapter 11. practical methodology grid random figure 11. 2 : comparison of grid search and random search. for illustration purposes we display two hyperparameters but we are typically interested in having many more. ( left ) to perform grid search, we provide a set of values for each hyperparameter. the search algorithm runs training for every joint hyperparameter setting in the cross product of these sets. to perform random search, we provide a probability distribution over joint ( right ) hyperparameter conﬁgurations. usually most of these hyperparameters are independent from each other. common choices for the distribution over a single hyperparameter include uniform and log - uniform ( to sample from a log - uniform distribution, take theexp of a sample from a uniform distribution ). the search algorithm then randomly samples joint hyperparameter conﬁgurations and runs training with each of them. both grid search and random search evaluate the validation set error and return the best conﬁguration. the ﬁgure illustrates the typical case where only some hyperparameters have a signiﬁcant inﬂuence on the result. in this illustration, only the hyperparameter on the horizontal axis has a signiﬁcant [UNK]. grid search wastes an


case where only some hyperparameters have a signiﬁcant inﬂuence on the result. in this illustration, only the hyperparameter on the horizontal axis has a signiﬁcant [UNK]. grid search wastes an amount of computation that is exponential in the number of non - inﬂuential hyperparameters, while random search tests a unique value of every inﬂuential hyperparameter on nearly every trial. figure reproduced with permission from ( ). bergstra and bengio 2012 433


chapter 11. practical methodology set error is then chosen as having found the best hyperparameters. see the left of ﬁgure for an illustration of a grid of hyperparameter values. 11. 2 how should the lists of values to search over be chosen? in the case of numerical ( ordered ) hyperparameters, the smallest and largest element of each list is chosen conservatively, based on prior experience with similar experiments, to make sure that the optimal value is very likely to be in the selected range. typically, a grid search involves picking values approximately on a logarithmic scale, e. g., a learning rate taken within the set {. 1,. 01, 10−3, 10−4, 10−5 }, or a number of hidden units taken with the set. { } 50 100 200 500 1000 2000,,,,, grid search usually performs best when it is performed repeatedly. for example, suppose that we ran a grid search over a hyperparameter α using values of { −1, 0, 1 }. if the best value found is, then we underestimated the range in which the best 1 α lies and we should shift the grid and run another search with α in, for example, {


values of { −1, 0, 1 }. if the best value found is, then we underestimated the range in which the best 1 α lies and we should shift the grid and run another search with α in, for example, { 1, 2, 3 }. if we ﬁnd that the best value of α is, then we may wish to reﬁne our 0 estimate by zooming in and running a grid search over. { − }.,,. 1 0 1 the obvious problem with grid search is that its computational cost grows exponentially with the number of hyperparameters. if there are m hyperparameters, each taking at most n values, then the number of training and evaluation trials required grows as o ( nm ). the trials may be run in parallel and exploit loose parallelism ( with almost no need for communication between [UNK] machines carrying out the search ) unfortunately, due to the exponential cost of grid search, even parallelization may not provide a satisfactory size of search. 11. 4. 4 random search fortunately, there is an alternative to grid search that is as simple to program, more convenient to use, and converges much faster to good values of the hyperparameters : random search (, ). berg


11. 4. 4 random search fortunately, there is an alternative to grid search that is as simple to program, more convenient to use, and converges much faster to good values of the hyperparameters : random search (, ). bergstra and bengio 2012 a random search proceeds as follows. first we deﬁne a marginal distribution for each hyperparameter, e. g., a bernoulli or multinoulli for binary or discrete hyperparameters, or a uniform distribution on a log - scale for positive real - valued hyperparameters. for example, log learning rate _ _ [UNK] − − u ( 1, 5 ) ( 11. 2 ) learning rate _ = 10log learning rate _ _. ( 11. 3 ) where u ( a, b ) indicates a sample of the uniform distribution in the interval ( a, b ). similarly the log number of hidden units _ _ _ _ may be sampled from u ( log ( 50 ), log ( 2000 ) ). 434


chapter 11. practical methodology unlike in the case of a grid search, one should not discretize or bin the values of the hyperparameters. this allows one to explore a larger set of values, and does not incur additional computational cost. in fact, as illustrated in ﬁgure, a 11. 2 random search can be exponentially more [UNK] than a grid search, when there are several hyperparameters that do not strongly [UNK] the performance measure. this is studied at length in ( ), who found that random bergstra and bengio 2012 search reduces the validation set error much faster than grid search, in terms of the number of trials run by each method. as with grid search, one may often want to run repeated versions of random search, to reﬁne the search based on the results of the ﬁrst run. the main reason why random search ﬁnds good solutions faster than grid search is that there are no wasted experimental runs, unlike in the case of grid search, when two values of a hyperparameter ( given values of the other hyperparameters ) would give the same result. in the case of grid search, the other hyperparameters would have the same values for these two runs, whereas with random search, they


a hyperparameter ( given values of the other hyperparameters ) would give the same result. in the case of grid search, the other hyperparameters would have the same values for these two runs, whereas with random search, they would usually have [UNK] values. hence if the change between these two values does not marginally make much [UNK] in terms of validation set error, grid search will unnecessarily repeat two equivalent experiments while random search will still give two independent explorations of the other hyperparameters. 11. 4. 5 model - based hyperparameter optimization the search for good hyperparameters can be cast as an optimization problem. the decision variables are the hyperparameters. the cost to be optimized is the validation set error that results from training using these hyperparameters. in simpliﬁed settings where it is feasible to compute the gradient of some [UNK] error measure on the validation set with respect to the hyperparameters, we can simply follow this gradient (, ;, ;, bengio et al. 1999 bengio 2000 maclaurin et al. 2015 ). unfortunately, in most practical settings, this gradient is unavailable, either due to its high computation and memory cost, or due


this gradient (, ;, ;, bengio et al. 1999 bengio 2000 maclaurin et al. 2015 ). unfortunately, in most practical settings, this gradient is unavailable, either due to its high computation and memory cost, or due to hyperparameters having intrinsically non - [UNK] interactions with the validation set error, as in the case of discrete - valued hyperparameters. to compensate for this lack of a gradient, we can build a model of the validation set error, then propose new hyperparameter guesses by performing optimization within this model. most model - based algorithms for hyperparameter search use a bayesian regression model to estimate both the expected value of the validation set error for each hyperparameter and the uncertainty around this expectation. opti - mization thus involves a [UNK] exploration ( proposing hyperparameters 435


chapter 11. practical methodology for which there is high uncertainty, which may lead to a large improvement but may also perform poorly ) and exploitation ( proposing hyperparameters which the model is conﬁdent will perform as well as any hyperparameters it has seen so far — usually hyperparameters that are very similar to ones it has seen before ). contemporary approaches to hyperparameter optimization include spearmint (, ), snoek et al. 2012 tpe (, ) and smac (, ). bergstra et al. 2011 hutter et al. 2011 currently, we cannot unambiguously recommend bayesian hyperparameter optimization as an established tool for achieving better deep learning results or for obtaining those results with less [UNK]. bayesian hyperparameter optimization sometimes performs comparably to human experts, sometimes better, but fails catastrophically on other problems. it may be worth trying to see if it works on a particular problem but is not yet [UNK] mature or reliable. that being said, hyperparameter optimization is an important ﬁeld of research that, while often driven primarily by the needs of deep learning, holds the potential to beneﬁt not only the entire ﬁeld of machine learning but the discipline of engineering in general. one


hyperparameter optimization is an important ﬁeld of research that, while often driven primarily by the needs of deep learning, holds the potential to beneﬁt not only the entire ﬁeld of machine learning but the discipline of engineering in general. one drawback common to most hyperparameter optimization algorithms with more sophistication than random search is that they require for a training ex - periment to run to completion before they are able to extract any information from the experiment. this is much less [UNK], in the sense of how much infor - mation can be gleaned early in an experiment, than manual search by a human practitioner, since one can usually tell early on if some set of hyperparameters is completely pathological. ( ) have introduced an early version swersky et al. 2014 of an algorithm that maintains a set of multiple experiments. at various time points, the hyperparameter optimization algorithm can choose to begin a new experiment, to “ freeze ” a running experiment that is not promising, or to “ thaw ” and resume an experiment that was earlier frozen but now appears promising given more information. 11. 5 debugging strategies when a machine learning system performs poorly, it is usually [UNK] to tell whether the poor performance is


promising, or to “ thaw ” and resume an experiment that was earlier frozen but now appears promising given more information. 11. 5 debugging strategies when a machine learning system performs poorly, it is usually [UNK] to tell whether the poor performance is intrinsic to the algorithm itself or whether there is a bug in the implementation of the algorithm. machine learning systems are [UNK] to debug for a variety of reasons. in most cases, we do not know a priori what the intended behavior of the algorithm is. in fact, the entire point of using machine learning is that it will discover useful behavior that we were not able to specify ourselves. if we train a 436


chapter 11. practical methodology neural network on a classiﬁcation task and it achieves 5 % test error, we have new no straightforward way of knowing if this is the expected behavior or sub - optimal behavior. a further [UNK] is that most machine learning models have multiple parts that are each adaptive. if one part is broken, the other parts can adapt and still achieve roughly acceptable performance. for example, suppose that we are training a neural net with several layers parametrized by weights w and biases b. suppose further that we have manually implemented the gradient descent rule for each parameter separately, and we made an error in the update for the biases : b b ← −α ( 11. 4 ) where α is the learning rate. this erroneous update does not use the gradient at all. it causes the biases to constantly become negative throughout learning, which is clearly not a correct implementation of any reasonable learning algorithm. the bug may not be apparent just from examining the output of the model though. depending on the distribution of the input, the weights may be able to adapt to compensate for the negative biases. most debugging strategies for neural nets are designed to get around one or both of these two [UNK]. either we design a case that is so simple


distribution of the input, the weights may be able to adapt to compensate for the negative biases. most debugging strategies for neural nets are designed to get around one or both of these two [UNK]. either we design a case that is so simple that the correct behavior actually can be predicted, or we design a test that exercises one part of the neural net implementation in isolation. some important debugging tests include : visualize the model in action : when training a model to detect objects in images, view some images with the detections proposed by the model displayed superimposed on the image. when training a generative model of speech, listen to some of the speech samples it produces. this may seem obvious, but it is easy to fall into the practice of only looking at quantitative performance measurements like accuracy or log - likelihood. directly observing the machine learning model performing its task will help to determine whether the quantitative performance numbers it achieves seem reasonable. evaluation bugs can be some of the most devastating bugs because they can mislead you into believing your system is performing well when it is not. visualize the worst mistakes : most models are able to output some sort of conﬁdence measure for the task they perform. for example, classiﬁers based on a


##d you into believing your system is performing well when it is not. visualize the worst mistakes : most models are able to output some sort of conﬁdence measure for the task they perform. for example, classiﬁers based on a softmax output layer assign a probability to each class. the probability assigned to the most likely class thus gives an estimate of the conﬁdence the model has in its classiﬁcation decision. typically, maximum likelihood training results in these values being overestimates rather than accurate probabilities of correct prediction, 437


chapter 11. practical methodology but they are somewhat useful in the sense that examples that are actually less likely to be correctly labeled receive smaller probabilities under the model. by viewing the training set examples that are the hardest to model correctly, one can often discover problems with the way the data has been preprocessed or labeled. for example, the street view transcription system originally had a problem where the address number detection system would crop the image too tightly and omit some of the digits. the transcription network then assigned very low probability to the correct answer on these images. sorting the images to identify the most conﬁdent mistakes showed that there was a systematic problem with the cropping. modifying the detection system to crop much wider images resulted in much better performance of the overall system, even though the transcription network needed to be able to process greater variation in the position and scale of the address numbers. reasoning about software using train and test error : it is often [UNK] to determine whether the underlying software is correctly implemented. some clues can be obtained from the train and test error. if training error is low but test error is high, then it is likely that that the training procedure works correctly, and the model is overﬁtting for fundamental algorithmic reasons. an alternative possibility is that the


obtained from the train and test error. if training error is low but test error is high, then it is likely that that the training procedure works correctly, and the model is overﬁtting for fundamental algorithmic reasons. an alternative possibility is that the test error is measured incorrectly due to a problem with saving the model after training then reloading it for test set evaluation, or if the test data was prepared [UNK] from the training data. if both train and test error are high, then it is [UNK] to determine whether there is a software defect or whether the model is underﬁtting due to fundamental algorithmic reasons. this scenario requires further tests, described next. fit a tiny dataset : if you have high error on the training set, determine whether it is due to genuine underﬁtting or due to a software defect. usually even small models can be guaranteed to be able ﬁt a [UNK] small dataset. for example, a classiﬁcation dataset with only one example can be ﬁt just by setting the biases of the output layer correctly. usually if you cannot train a classiﬁer to correctly label a single example, an autoencoder to successfully reproduce a single example with high ﬁdelity, or a generative model to consistently emit


##es of the output layer correctly. usually if you cannot train a classiﬁer to correctly label a single example, an autoencoder to successfully reproduce a single example with high ﬁdelity, or a generative model to consistently emit samples resembling a single example, there is a software defect preventing successful optimization on the training set. this test can be extended to a small dataset with few examples. compare back - propagated derivatives to numerical derivatives : if you are using a software framework that requires you to implement your own gradient com - putations, or if you are adding a new operation to a [UNK] library and must deﬁne its bprop method, then a common source of error is implementing this gradient expression incorrectly. one way to verify that these derivatives are correct 438


chapter 11. practical methodology is to compare the derivatives computed by your implementation of automatic [UNK] to the derivatives computed by a. because ﬁnite [UNK] f ( ) = lim x →0 f x f x ( + ) − ( ), ( 11. 5 ) we can approximate the derivative by using a small, ﬁnite : f ( ) x ≈f x f x ( + ) − ( ). ( 11. 6 ) we can improve the accuracy of the approximation by using the centered [UNK] - ence : f ( ) x ≈f x ( + 1 2 f x ) − ( −1 2 ). ( 11. 7 ) the perturbation size must chosen to be large enough to ensure that the pertur - bation is not rounded down too much by ﬁnite - precision numerical computations. usually, we will want to test the gradient or jacobian of a vector - valued function g : rm →rn. unfortunately, ﬁnite [UNK] only allows us to take a single derivative at a time. we can either run ﬁnite [UNK] mn times to evaluate all of the partial derivatives of g, or we can apply the test to a new function that uses random projections at both the input and output of g. for example, we


a time. we can either run ﬁnite [UNK] mn times to evaluate all of the partial derivatives of g, or we can apply the test to a new function that uses random projections at both the input and output of g. for example, we can apply our test of the implementation of the derivatives to f ( x ) where f ( x ) = ut g ( vx ), where u and v are randomly chosen vectors. computing f ( x ) correctly requires being able to back - propagate through g correctly, yet is [UNK] to do with ﬁnite [UNK] because f has only a single input and a single output. it is usually a good idea to repeat this test for more than one value of u and v to reduce the chance that the test overlooks mistakes that are orthogonal to the random projection. if one has access to numerical computation on complex numbers, then there is a very [UNK] way to numerically estimate the gradient by using complex numbers as input to the function ( squire and trapp 1998, ). the method is based on the observation that f x i f x if ( + ) = ( ) + ( ) + ( x o 2 ) ( 11. 8 ) real ( ( + ) ) = ( ) + ( f x i f


. the method is based on the observation that f x i f x if ( + ) = ( ) + ( ) + ( x o 2 ) ( 11. 8 ) real ( ( + ) ) = ( ) + ( f x i f x o 2 ) imag (, f x i ( + ) ) = f ( ) + ( x o 2 ), ( 11. 9 ) where i = √−1. unlike in the real - valued case above, there is no cancellation [UNK] due to taking the [UNK] between the value of f at [UNK] points. this allows the use of tiny values of like = 10 −150, which make the o ( 2 ) error insigniﬁcant for all practical purposes. 439


chapter 11. practical methodology monitor histograms of activations and gradient : it is often useful to visualize statistics of neural network activations and gradients, collected over a large amount of training iterations ( maybe one epoch ). the pre - activation value of hidden units can tell us if the units saturate, or how often they do. for example, for rectiﬁers, how often are they [UNK]? are there units that are always [UNK]? for tanh units, the average of the absolute value of the pre - activations tells us how saturated the unit is. in a deep network where the propagated gradients quickly grow or quickly vanish, optimization may be hampered. finally, it is useful to compare the magnitude of parameter gradients to the magnitude of the parameters themselves. as suggested by ( ), we would like the magnitude of parameter updates bottou 2015 over a minibatch to represent something like 1 % of the magnitude of the parameter, not 50 % or 0. 001 % ( which would make the parameters move too slowly ). it may be that some groups of parameters are moving at a good pace while others are stalled. when the data is sparse ( like in natural language ), some parameters may be very rarely updated,


( which would make the parameters move too slowly ). it may be that some groups of parameters are moving at a good pace while others are stalled. when the data is sparse ( like in natural language ), some parameters may be very rarely updated, and this should be kept in mind when monitoring their evolution. finally, many deep learning algorithms provide some sort of guarantee about the results produced at each step. for example, in part, we will see some approx - iii imate inference algorithms that work by using algebraic solutions to optimization problems. typically these can be debugged by testing each of their guarantees. some guarantees that some optimization algorithms [UNK] include that the objective function will never increase after one step of the algorithm, that the gradient with respect to some subset of variables will be zero after each step of the algorithm, and that the gradient with respect to all variables will be zero at convergence. usually due to rounding error, these conditions will not hold exactly in a digital computer, so the debugging test should include some tolerance parameter. 11. 6 example : multi - digit number recognition to provide an end - to - end description of how to apply our design methodology in practice, we present a brief account of the street view transcription system, from the point of view of designing


parameter. 11. 6 example : multi - digit number recognition to provide an end - to - end description of how to apply our design methodology in practice, we present a brief account of the street view transcription system, from the point of view of designing the deep learning components. obviously, many other components of the complete system, such as the street view cars, the database infrastructure, and so on, were of paramount importance. from the point of view of the machine learning task, the process began with data collection. the cars collected the raw data and human operators provided labels. the transcription task was preceded by a signiﬁcant amount of dataset curation, including using other machine learning techniques to detect the house 440


chapter 11. practical methodology numbers prior to transcribing them. the transcription project began with a choice of performance metrics and desired values for these metrics. an important general principle is to tailor the choice of metric to the business goals for the project. because maps are only useful if they have high accuracy, it was important to set a high accuracy requirement for this project. speciﬁcally, the goal was to obtain human - level, 98 % accuracy. this level of accuracy may not always be feasible to obtain. in order to reach this level of accuracy, the street view transcription system sacriﬁces coverage. coverage thus became the main performance metric optimized during the project, with accuracy held at 98 %. as the convolutional network improved, it became possible to reduce the conﬁdence threshold below which the network refuses to transcribe the input, eventually exceeding the goal of 95 % coverage. after choosing quantitative goals, the next step in our recommended methodol - ogy is to rapidly establish a sensible baseline system. for vision tasks, this means a convolutional network with rectiﬁed linear units. the transcription project began with such a model. at the time, it was not common for a con


is to rapidly establish a sensible baseline system. for vision tasks, this means a convolutional network with rectiﬁed linear units. the transcription project began with such a model. at the time, it was not common for a convolutional network to output a sequence of predictions. in order to begin with the simplest possible baseline, the ﬁrst implementation of the output layer of the model consisted of n [UNK] softmax units to predict a sequence of n characters. these softmax units were trained exactly the same as if the task were classiﬁcation, with each softmax unit trained independently. our recommended methodology is to iteratively reﬁne the baseline and test whether each change makes an improvement. the ﬁrst change to the street view transcription system was motivated by a theoretical understanding of the coverage metric and the structure of the data. speciﬁcally, the network refuses to classify an input x whenever the probability of the output sequence p ( y x | ) < t for some threshold t. initially, the deﬁnition of p ( y x | ) was ad - hoc, based on simply multiplying all of the softmax outputs together. this motivated the development of a specialized output layer and cost function that actually computed


t. initially, the deﬁnition of p ( y x | ) was ad - hoc, based on simply multiplying all of the softmax outputs together. this motivated the development of a specialized output layer and cost function that actually computed a principled log - likelihood. this approach allowed the example rejection mechanism to function much more [UNK]. at this point, coverage was still below 90 %, yet there were no obvious theoretical problems with the approach. our methodology therefore suggests to instrument the train and test set performance in order to determine whether the problem is underﬁtting or overﬁtting. in this case, train and test set error were nearly identical. indeed, the main reason this project proceeded so smoothly was the availability of a dataset with tens of millions of labeled examples. because train and test set error were so similar, this suggested that the problem was either due 441


chapter 11. practical methodology to underﬁtting or due to a problem with the training data. one of the debugging strategies we recommend is to visualize the model ’ s worst errors. in this case, that meant visualizing the incorrect training set transcriptions that the model gave the highest conﬁdence. these proved to mostly consist of examples where the input image had been cropped too tightly, with some of the digits of the address being removed by the cropping operation. for example, a photo of an address “ 1849 ” might be cropped too tightly, with only the “ 849 ” remaining visible. this problem could have been resolved by spending weeks improving the accuracy of the address number detection system responsible for determining the cropping regions. instead, the team took a much more practical decision, to simply expand the width of the crop region to be systematically wider than the address number detection system predicted. this single change added ten percentage points to the transcription system ’ s coverage. finally, the last few percentage points of performance came from adjusting hyperparameters. this mostly consisted of making the model larger while main - taining some restrictions on its computational cost. because train and test error remained roughly equal, it was always clear that any performance deﬁcits


of performance came from adjusting hyperparameters. this mostly consisted of making the model larger while main - taining some restrictions on its computational cost. because train and test error remained roughly equal, it was always clear that any performance deﬁcits were due to underﬁtting, as well as due to a few remaining problems with the dataset itself. overall, the transcription project was a great success, and allowed hundreds of millions of addresses to be transcribed both faster and at lower cost than would have been possible via human [UNK]. we hope that the design principles described in this chapter will lead to many other similar successes. 442


chapter 12 applications in this chapter, we describe how to use deep learning to solve applications in com - puter vision, speech recognition, natural language processing, and other application areas of commercial interest. we begin by discussing the large scale neural network implementations required for most serious ai applications. next, we review several speciﬁc application areas that deep learning has been used to solve. while one goal of deep learning is to design algorithms that are capable of solving a broad variety of tasks, so far some degree of specialization is needed. for example, vision tasks require processing a large number of input features ( pixels ) per example. language tasks require modeling a large number of possible values ( words in the vocabulary ) per input feature. 12. 1 large - scale deep learning deep learning is based on the philosophy of connectionism : while an individual biological neuron or an individual feature in a machine learning model is not intelligent, a large population of these neurons or features acting together can exhibit intelligent behavior. it truly is important to emphasize the fact that the number of neurons must be large. one of the key factors responsible for the improvement in neural network ’ s accuracy and the improvement of the complexity of tasks they can solve between the 1980s and today is the dramatic increase in the size of the networks


that the number of neurons must be large. one of the key factors responsible for the improvement in neural network ’ s accuracy and the improvement of the complexity of tasks they can solve between the 1980s and today is the dramatic increase in the size of the networks we use. as we saw in section, network sizes have 1. 2. 3 grown exponentially for the past three decades, yet artiﬁcial neural networks are only as large as the nervous systems of insects. because the size of neural networks is of paramount importance, deep learning 443


chapter 12. applications requires high performance hardware and software infrastructure. 12. 1. 1 fast cpu implementations traditionally, neural networks were trained using the cpu of a single machine. today, this approach is generally considered [UNK]. we now mostly use gpu computing or the cpus of many machines networked together. before moving to these expensive setups, researchers worked hard to demonstrate that cpus could not manage the high computational workload required by neural networks. a description of how to implement [UNK] numerical cpu code is beyond the scope of this book, but we emphasize here that careful implementation for speciﬁc cpu families can yield large improvements. for example, in 2011, the best cpus available could run neural network workloads faster when using ﬁxed - point arithmetic rather than ﬂoating - point arithmetic. by creating a carefully tuned ﬁxed - point implementation, vanhoucke 2011 et al. ( ) obtained a threefold speedup over a strong ﬂoating - point system. each new model of cpu has [UNK] performance characteristics, so sometimes ﬂoating - point implementations can be faster too. the important principle is that careful specialization of numerical computation routines can yield a large [UNK]. other strategies, besides choosing whether to use ﬁxed or ﬂoating point, include


performance characteristics, so sometimes ﬂoating - point implementations can be faster too. the important principle is that careful specialization of numerical computation routines can yield a large [UNK]. other strategies, besides choosing whether to use ﬁxed or ﬂoating point, include optimizing data structures to avoid cache misses and using vector instructions. many machine learning researchers neglect these implementation details, but when the performance of an implementation restricts the size of the model, the accuracy of the model [UNK]. 12. 1. 2 gpu implementations most modern neural network implementations are based on graphics processing units. graphics processing units ( gpus ) are specialized hardware components that were originally developed for graphics applications. the consumer market for video gaming systems spurred development of graphics processing hardware. the performance characteristics needed for good video gaming systems turn out to be beneﬁcial for neural networks as well. video game rendering requires performing many operations in parallel quickly. models of characters and environments are speciﬁed in terms of lists of 3 - d coordinates of vertices. graphics cards must perform matrix multiplication and division on many vertices in parallel to convert these 3 - d coordinates into 2 - d on - screen coordinates. the graphics card must then perform many computations at each pixel in parallel to determine the color of each pixel. in


must perform matrix multiplication and division on many vertices in parallel to convert these 3 - d coordinates into 2 - d on - screen coordinates. the graphics card must then perform many computations at each pixel in parallel to determine the color of each pixel. in both cases, the 444


chapter 12. applications computations are fairly simple and do not involve much branching compared to the computational workload that a cpu usually encounters. for example, each vertex in the same rigid object will be multiplied by the same matrix ; there is no need to evaluate an if statement per - vertex to determine which matrix to multiply by. the computations are also entirely independent of each other, and thus may be parallelized easily. the computations also involve processing massive [UNK] of memory, containing bitmaps describing the texture ( color pattern ) of each object to be rendered. together, this results in graphics cards having been designed to have a high degree of parallelism and high memory bandwidth, at the cost of having a lower clock speed and less branching capability relative to traditional cpus. neural network algorithms require the same performance characteristics as the real - time graphics algorithms described above. neural networks usually involve large and numerous [UNK] of parameters, activation values, and gradient values, each of which must be completely updated during every step of training. these [UNK] are large enough to fall outside the cache of a traditional desktop computer so the memory bandwidth of the system often becomes the rate limiting factor. gpus [UNK] a compelling advantage over cpus due to their high memory bandwidth. neural network training algorithms typically do


[UNK] are large enough to fall outside the cache of a traditional desktop computer so the memory bandwidth of the system often becomes the rate limiting factor. gpus [UNK] a compelling advantage over cpus due to their high memory bandwidth. neural network training algorithms typically do not involve much branching or sophisticated control, so they are appropriate for gpu hardware. since neural networks can be divided into multiple individual “ neurons ” that can be processed independently from the other neurons in the same layer, neural networks easily beneﬁt from the parallelism of gpu computing. gpu hardware was originally so specialized that it could only be used for graphics tasks. over time, gpu hardware became more ﬂexible, allowing custom subroutines to be used to transform the coordinates of vertices or assign colors to pixels. in principle, there was no requirement that these pixel values actually be based on a rendering task. these gpus could be used for scientiﬁc computing by writing the output of a computation to a [UNK] of pixel values. steinkrau et al. ( ) implemented a two - layer fully connected neural network on a gpu and 2005 reported a threefold speedup over their cpu - based baseline. shortly thereafter, chellapilla 2006 et al. ( ) demonstrated that


##krau et al. ( ) implemented a two - layer fully connected neural network on a gpu and 2005 reported a threefold speedup over their cpu - based baseline. shortly thereafter, chellapilla 2006 et al. ( ) demonstrated that the same technique could be used to accelerate supervised convolutional networks. the popularity of graphics cards for neural network training exploded after the advent of general purpose gpus. these gp - gpus could execute arbitrary code, not just rendering subroutines. nvidia ’ s cuda programming language provided a way to write this arbitrary code in a c - like language. with their relatively convenient programming model, massive parallelism, and high memory 445


chapter 12. applications bandwidth, gp - gpus now [UNK] an ideal platform for neural network programming. this platform was rapidly adopted by deep learning researchers soon after it became available (, ;, ). raina et al. 2009 ciresan et al. 2010 writing [UNK] code for gp - gpus remains a [UNK] task best left to spe - cialists. the techniques required to obtain good performance on gpu are very [UNK] from those used on cpu. for example, good cpu - based code is usually designed to read information from the cache as much as possible. on gpu, most writable memory locations are not cached, so it can actually be faster to compute the same value twice, rather than compute it once and read it back from memory. gpu code is also inherently multi - threaded and the [UNK] threads must be coordinated with each other carefully. for example, memory operations are faster if they can be coalesced. coalesced reads or writes occur when several threads can each read or write a value that they need simultaneously, as part of a single memory transaction. [UNK] models of gpus are able to coalesce [UNK] kinds of read or write patterns. typically, memory operations are easier to coalesce if among n threads, thread i


value that they need simultaneously, as part of a single memory transaction. [UNK] models of gpus are able to coalesce [UNK] kinds of read or write patterns. typically, memory operations are easier to coalesce if among n threads, thread i accesses byte i + j of memory, and j is a multiple of some power of 2. the exact speciﬁcations [UNK] between models of gpu. another common consideration for gpus is making sure that each thread in a group executes the same instruction simultaneously. this means that branching can be [UNK] on gpu. threads are divided into small groups called warps. each thread in a warp executes the same instruction during each cycle, so if [UNK] threads within the same warp need to execute [UNK] code paths, these [UNK] code paths must be traversed sequentially rather than in parallel. due to the [UNK] of writing high performance gpu code, researchers should structure their workﬂow to avoid needing to write new gpu code in order to test new models or algorithms. typically, one can do this by building a software library of high performance operations like convolution and matrix multiplication, then specifying models in terms of calls to this library of operations. for example, the machine learning library pylearn


. typically, one can do this by building a software library of high performance operations like convolution and matrix multiplication, then specifying models in terms of calls to this library of operations. for example, the machine learning library pylearn2 ( goodfellow 2013c et al., ) speciﬁes all of its machine learning algorithms in terms of calls to theano (, ; bergstra et al. 2010 bastien 2012 et al., ) and cuda - convnet (, ), which provide these krizhevsky 2010 high - performance operations. this factored approach can also ease support for multiple kinds of hardware. for example, the same theano program can run on either cpu or gpu, without needing to change any of the calls to theano itself. other libraries like tensorflow (, ) and torch (, abadi et al. 2015 collobert et al. 2011b ) provide similar features. 446


chapter 12. applications 12. 1. 3 large - scale distributed implementations in many cases, the computational resources available on a single machine are [UNK]. we therefore want to distribute the workload of training and inference across many machines. distributing inference is simple, because each input example we want to process can be run by a separate machine. this is known as. data parallelism it is also possible to get model parallelism, where multiple machines work together on a single datapoint, with each machine running a [UNK] part of the model. this is feasible for both inference and training. data parallelism during training is somewhat harder. we can increase the size of the minibatch used for a single sgd step, but usually we get less than linear returns in terms of optimization performance. it would be better to allow multiple machines to compute multiple gradient descent steps in parallel. unfortunately, the standard deﬁnition of gradient descent is as a completely sequential algorithm : the gradient at step is a function of the parameters produced by step. t t −1 this can be solved using asynchronous stochastic gradient descent ( ben - gio 2001 recht 2011 et al., ; et al., ). in this approach, several processor cores share the memory representing the


t −1 this can be solved using asynchronous stochastic gradient descent ( ben - gio 2001 recht 2011 et al., ; et al., ). in this approach, several processor cores share the memory representing the parameters. each core reads parameters without a lock, then computes a gradient, then increments the parameters without a lock. this reduces the average amount of improvement that each gradient descent step yields, because some of the cores overwrite each other ’ s progress, but the increased rate of production of steps causes the learning process to be faster overall. dean et al. ( ) pioneered the multi - machine implementation of this lock - free approach 2012 to gradient descent, where the parameters are managed by a parameter server rather than stored in shared memory. distributed asynchronous gradient descent remains the primary strategy for training large deep networks and is used by most major deep learning groups in industry (, ; chilimbi et al. 2014 wu et al., 2015 ). academic deep learning researchers typically cannot [UNK] the same scale of distributed learning systems but some research has focused on how to build distributed networks with relatively low - cost hardware available in the university setting (, ). coates et al. 2013 12. 1. 4


deep learning researchers typically cannot [UNK] the same scale of distributed learning systems but some research has focused on how to build distributed networks with relatively low - cost hardware available in the university setting (, ). coates et al. 2013 12. 1. 4 model compression in many commercial applications, it is much more important that the time and memory cost of running inference in a machine learning model be low than that the time and memory cost of training be low. for applications that do not require 447


chapter 12. applications personalization, it is possible to train a model once, then deploy it to be used by billions of users. in many cases, the end user is more resource - constrained than the developer. for example, one might train a speech recognition network with a powerful computer cluster, then deploy it on mobile phones. a key strategy for reducing the cost of inference is model compression ( bu - [UNK] 2006 et al., ). the basic idea of model compression is to replace the original, expensive model with a smaller model that requires less memory and runtime to store and evaluate. model compression is applicable when the size of the original model is driven primarily by a need to prevent overﬁtting. in most cases, the model with the lowest generalization error is an ensemble of several independently trained models. evaluating all n ensemble members is expensive. sometimes, even a single model generalizes better if it is large ( for example, if it is regularized with dropout ). these large models learn some function f ( x ), but do so using many more parameters than are necessary for the task. their size is necessary only due to the limited number of training examples. as soon as we have ﬁt this function f ( x ), we can generate a training


x ), but do so using many more parameters than are necessary for the task. their size is necessary only due to the limited number of training examples. as soon as we have ﬁt this function f ( x ), we can generate a training set containing inﬁnitely many examples, simply by applying f to randomly sampled points x. we then train the new, smaller, model to match f ( x ) on these points. in order to most [UNK] use the capacity of the new, small model, it is best to sample the new x points from a distribution resembling the actual test inputs that will be supplied to the model later. this can be done by corrupting training examples or by drawing points from a generative model trained on the original training set. alternatively, one can train the smaller model only on the original training points, but train it to copy other features of the model, such as its posterior distribution over the incorrect classes ( hinton 2014 2015 et al.,, ). 12. 1. 5 dynamic structure one strategy for accelerating data processing systems in general is to build systems that have dynamic structure in the graph describing the computation needed to process an input. data processing systems can dynamically determine which subset of many neural networks should be run on a


5 dynamic structure one strategy for accelerating data processing systems in general is to build systems that have dynamic structure in the graph describing the computation needed to process an input. data processing systems can dynamically determine which subset of many neural networks should be run on a given input. individual neural networks can also exhibit dynamic structure internally by determining which subset of features ( hidden units ) to compute given information from the input. this form of dynamic structure inside neural networks is sometimes called conditional computation (, ;, ). since many components of bengio 2013 bengio et al. 2013b the architecture may be relevant only for a small amount of possible inputs, the 448


chapter 12. applications system can run faster by computing these features only when they are needed. dynamic structure of computations is a basic computer science principle applied generally throughout the software engineering discipline. the simplest versions of dynamic structure applied to neural networks are based on determining which subset of some group of neural networks ( or other machine learning models ) should be applied to a particular input. a venerable strategy for accelerating inference in a classiﬁer is to use a cascade of classiﬁers. the cascade strategy may be applied when the goal is to detect the presence of a rare object ( or event ). to know for sure that the object is present, we must use a sophisticated classiﬁer with high capacity, that is expensive to run. however, because the object is rare, we can usually use much less computation to reject inputs as not containing the object. in these situations, we can train a sequence of classiﬁers. the ﬁrst classiﬁers in the sequence have low capacity, and are trained to have high recall. in other words, they are trained to make sure we do not wrongly reject an input when the object is present. the ﬁnal classiﬁer is trained to have high precision. at test time, we run inference by running


have high recall. in other words, they are trained to make sure we do not wrongly reject an input when the object is present. the ﬁnal classiﬁer is trained to have high precision. at test time, we run inference by running the classiﬁers in a sequence, abandoning any example as soon as any one element in the cascade rejects it. overall, this allows us to verify the presence of objects with high conﬁdence, using a high capacity model, but does not force us to pay the cost of full inference for every example. there are two [UNK] ways that the cascade can achieve high capacity. one way is to make the later members of the cascade individually have high capacity. in this case, the system as a whole obviously has high capacity, because some of its individual members do. it is also possible to make a cascade in which every individual model has low capacity but the system as a whole has high capacity due to the combination of many small models. viola and jones 2001 ( ) used a cascade of boosted decision trees to implement a fast and robust face detector suitable for use in handheld digital cameras. their classiﬁer localizes a face using essentially a sliding window approach in which many windows are examined and rejected if they do not contain


a cascade of boosted decision trees to implement a fast and robust face detector suitable for use in handheld digital cameras. their classiﬁer localizes a face using essentially a sliding window approach in which many windows are examined and rejected if they do not contain faces. another version of cascades uses the earlier models to implement a sort of hard attention mechanism : the early members of the cascade localize an object and later members of the cascade perform further processing given the location of the object. for example, google transcribes address numbers from street view imagery using a two - step cascade that ﬁrst locates the address number with one machine learning model and then transcribes it with another ( goodfellow 2014d et al., ). decision trees themselves are an example of dynamic structure, because each node in the tree determines which of its subtrees should be evaluated for each input. a simple way to accomplish the union of deep learning and dynamic structure 449


chapter 12. applications is to train a decision tree in which each node uses a neural network to make the splitting decision (, ), though this has typically not been guo and gelfand 1992 done with the primary goal of accelerating inference computations. in the same spirit, one can use a neural network, called the gater to select which one out of several expert networks will be used to compute the output, given the current input. the ﬁrst version of this idea is called the mixture of experts ( nowlan 1990 jacobs 1991, ; et al., ), in which the gater outputs a set of probabilities or weights ( obtained via a softmax nonlinearity ), one per expert, and the ﬁnal output is obtained by the weighted combination of the output of the experts. in that case, the use of the gater does not [UNK] a reduction in computational cost, but if a single expert is chosen by the gater for each example, we obtain the hard mixture of experts (,, ), which collobert et al. 2001 2002 can considerably accelerate training and inference time. this strategy works well when the number of gating decisions is small because it is not combinatorial. but when we want to select [UNK] subsets of units


, which collobert et al. 2001 2002 can considerably accelerate training and inference time. this strategy works well when the number of gating decisions is small because it is not combinatorial. but when we want to select [UNK] subsets of units or parameters, it is not possible to use a “ soft switch ” because it requires enumerating ( and computing outputs for ) all the gater conﬁgurations. to deal with this problem, several approaches have been explored to train combinatorial gaters. ( ) experiment with bengio et al. 2013b several estimators of the gradient on the gating probabilities, while bacon et al. ( ) and ( ) use reinforcement learning techniques ( policy 2015 bengio et al. 2015a gradient ) to learn a form of conditional dropout on blocks of hidden units and get an actual reduction in computational cost without impacting negatively on the quality of the approximation. another kind of dynamic structure is a switch, where a hidden unit can receive input from [UNK] units depending on the context. this dynamic routing approach can be interpreted as an attention mechanism (, ). olshausen et al. 1993 so far, the use of a hard switch has not proven [UNK] on large - scale applications. contemporary approaches


units depending on the context. this dynamic routing approach can be interpreted as an attention mechanism (, ). olshausen et al. 1993 so far, the use of a hard switch has not proven [UNK] on large - scale applications. contemporary approaches instead use a weighted average over many possible inputs, and thus do not achieve all of the possible computational beneﬁts of dynamic structure. contemporary attention mechanisms are described in section. 12. 4. 5. 1 one major obstacle to using dynamically structured systems is the decreased degree of parallelism that results from the system following [UNK] code branches for [UNK] inputs. this means that few operations in the network can be described as matrix multiplication or batch convolution on a minibatch of examples. we can write more specialized sub - routines that convolve each example with [UNK] kernels or multiply each row of a design matrix by a [UNK] set of columns of weights. unfortunately, these more specialized subroutines are [UNK] to implement [UNK]. cpu implementations will be slow due to the lack of cache 450


chapter 12. applications coherence and gpu implementations will be slow due to the lack of coalesced memory transactions and the need to serialize warps when members of a warp take [UNK] branches. in some cases, these issues can be mitigated by partitioning the examples into groups that all take the same branch, and processing these groups of examples simultaneously. this can be an acceptable strategy for minimizing the time required to process a ﬁxed amount of examples in an [UNK] setting. in a real - time setting where examples must be processed continuously, partitioning the workload can result in load - balancing issues. for example, if we assign one machine to process the ﬁrst step in a cascade and another machine to process the last step in a cascade, then the ﬁrst will tend to be overloaded and the last will tend to be underloaded. similar issues arise if each machine is assigned to implement [UNK] nodes of a neural decision tree. 12. 1. 6 specialized hardware implementations of deep networks since the early days of neural networks research, hardware designers have worked on specialized hardware implementations that could speed up training and / or inference of neural network algorithms. see early and more recent reviews of specialized hardware for deep networks (, ;, lindsey and lindblad


early days of neural networks research, hardware designers have worked on specialized hardware implementations that could speed up training and / or inference of neural network algorithms. see early and more recent reviews of specialized hardware for deep networks (, ;, lindsey and lindblad 1994 beiu et al. 2003 misra and saha 2010 ;, ). [UNK] forms of specialized hardware ( graf and jackel 1989 mead and, ; ismail 2012 kim 2009 pham 2012 chen 2014a b, ; et al., ; et al., ; et al.,, ) have been developed over the last decades, either with asics ( application - speciﬁc inte - grated circuit ), either with digital ( based on binary representations of numbers ), analog ( graf and jackel 1989 mead and ismail 2012, ;, ) ( based on physical imple - mentations of continuous values as voltages or currents ) or hybrid implementations ( combining digital and analog components ). in recent years more ﬂexible fpga ( ﬁeld programmable gated array ) implementations ( where the particulars of the circuit can be written on the chip after it has been built ) have been developed. though software implementations on general - purpose processing units ( cpus and gpus ) typically


ﬁeld programmable gated array ) implementations ( where the particulars of the circuit can be written on the chip after it has been built ) have been developed. though software implementations on general - purpose processing units ( cpus and gpus ) typically use 32 or 64 bits of precision to represent ﬂoating point numbers, it has long been known that it was possible to use less precision, at least at inference time ( holt and baker 1991 holi and hwang 1993 presley, ;, ; and haggard 1994 simard and graf 1994 wawrzynek 1996 savich, ;, ; et al., ; et al., 2007 ). this has become a more pressing issue in recent years as deep learning has gained in popularity in industrial products, and as the great impact of faster hardware was demonstrated with gpus. another factor that motivates current research on specialized hardware for deep networks is that the rate of progress of a single cpu or gpu core has slowed down, and most recent improvements in 451