chapter 14. autoencoders many forms of dimensionality reduction place semantically related examples near each other, as observed by salakhutdinov and hinton 2007b torralba ( ) and et al. ( ). the hints provided by the mapping to the lower - dimensional space aid 2008 generalization. one task that beneﬁts even more than usual from dimensionality reduction is information retrieval, the task of ﬁnding entries in a database that resemble a query entry. this task derives the usual beneﬁts from dimensionality reduction that other tasks do, but also derives the additional beneﬁt that search can become extremely [UNK] in certain kinds of low dimensional spaces. speciﬁcally, if we train the dimensionality reduction algorithm to produce a code that is low - dimensional and, then we can store all database entries in a hash table binary mapping binary code vectors to entries. this hash table allows us to perform information retrieval by returning all database entries that have the same binary code as the query. we can also search over slightly less similar entries very [UNK], just by ﬂipping individual bits from the encoding of the query. this approach to information retrieval via dimensionality reduction and binarization is called semantic hashing ( salakhutdin


. we can also search over slightly less similar entries very [UNK], just by ﬂipping individual bits from the encoding of the query. this approach to information retrieval via dimensionality reduction and binarization is called semantic hashing ( salakhutdinov and hinton 2007b 2009b,, ), and has been applied to both textual input ( salakhutdinov and hinton 2007b 2009b,, ) and images ( torralba 2008 weiss 2008 krizhevsky and hinton 2011 et al., ; et al., ;, ). to produce binary codes for semantic hashing, one typically uses an encoding function with sigmoids on the ﬁnal layer. the sigmoid units must be trained to be saturated to nearly 0 or nearly 1 for all input values. one trick that can accomplish this is simply to inject additive noise just before the sigmoid nonlinearity during training. the magnitude of the noise should increase over time. to ﬁght that noise and preserve as much information as possible, the network must increase the magnitude of the inputs to the sigmoid function, until saturation occurs. the idea of learning a hashing function has been further explored in several directions, including the idea of training the representations so as

/home/ricoiban/GEMMA/mnlp_chatsplaining/RAG/Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville (z-lib.org).pdf


information as possible, the network must increase the magnitude of the inputs to the sigmoid function, until saturation occurs. the idea of learning a hashing function has been further explored in several directions, including the idea of training the representations so as to optimize a loss more directly linked to the task of ﬁnding nearby examples in the hash table (, ). norouzi and fleet 2011 525


chapter 15 representation learning in this chapter, we ﬁrst discuss what it means to learn representations and how the notion of representation can be useful to design deep architectures. we discuss how learning algorithms share statistical strength across [UNK] tasks, including using information from unsupervised tasks to perform supervised tasks. shared representations are useful to handle multiple modalities or domains, or to transfer learned knowledge to tasks for which few or no examples are given but a task representation exists. finally, we step back and argue about the reasons for the success of representation learning, starting with the theoretical advantages of distributed representations ( hinton 1986 et al., ) and deep representations and ending with the more general idea of underlying assumptions about the data generating process, in particular about underlying causes of the observed data. many information processing tasks can be very easy or very [UNK] depending on how the information is represented. this is a general principle applicable to daily life, computer science in general, and to machine learning. for example, it is straightforward for a person to divide 210 by 6 using long division. the task becomes considerably less straightforward if it is instead posed using the roman numeral representation of the numbers. most modern people asked to divide ccx by vi would begin by converting the numbers to the arabic numeral


divide 210 by 6 using long division. the task becomes considerably less straightforward if it is instead posed using the roman numeral representation of the numbers. most modern people asked to divide ccx by vi would begin by converting the numbers to the arabic numeral representation, permitting long division procedures that make use of the place value system. more concretely, we can quantify the asymptotic runtime of various operations using appropriate or inappropriate representations. for example, inserting a number into the correct position in a sorted list of numbers is an o ( n ) operation if the list is represented as a linked list, but only o ( log n ) if the list is represented as a red - black tree. in the context of machine learning, what makes one representation better than 526


chapter 15. representation learning another? generally speaking, a good representation is one that makes a subsequent learning task easier. the choice of representation will usually depend on the choice of the subsequent learning task. we can think of feedforward networks trained by supervised learning as per - forming a kind of representation learning. speciﬁcally, the last layer of the network is typically a linear classiﬁer, such as a softmax regression classiﬁer. the rest of the network learns to provide a representation to this classiﬁer. training with a supervised criterion naturally leads to the representation at every hidden layer ( but more so near the top hidden layer ) taking on properties that make the classiﬁcation task easier. for example, classes that were not linearly separable in the input features may become linearly separable in the last hidden layer. in principle, the last layer could be another kind of model, such as a nearest neighbor classiﬁer ( salakhutdinov and hinton 2007a, ). the features in the penultimate layer should learn [UNK] properties depending on the type of the last layer. supervised training of feedforward networks does not involve explicitly imposing any condition on the learned intermediate features. other kinds of representation learning algorithms are


##a, ). the features in the penultimate layer should learn [UNK] properties depending on the type of the last layer. supervised training of feedforward networks does not involve explicitly imposing any condition on the learned intermediate features. other kinds of representation learning algorithms are often explicitly designed to shape the representation in some particular way. for example, suppose we want to learn a representation that makes density estimation easier. distributions with more independences are easier to model, so we could design an objective function that encourages the elements of the representation vector h to be independent. just like supervised networks, unsupervised deep learning algorithms have a main training objective but also learn a representation as a side [UNK]. regardless of how a representation was obtained, it can be used for another task. alternatively, multiple tasks ( some supervised, some unsupervised ) can be learned together with some shared internal representation. most representation learning problems face a [UNK] preserving as much information about the input as possible and attaining nice properties ( such as independence ). representation learning is particularly interesting because it provides one way to perform unsupervised and semi - supervised learning. we often have very large amounts of unlabeled training data and relatively little labeled training data. training with supervised learning techniques on the labeled subset often


particularly interesting because it provides one way to perform unsupervised and semi - supervised learning. we often have very large amounts of unlabeled training data and relatively little labeled training data. training with supervised learning techniques on the labeled subset often results in severe overﬁtting. semi - supervised learning [UNK] the chance to resolve this overﬁtting problem by also learning from the unlabeled data. speciﬁcally, we can learn good representations for the unlabeled data, and then use these representations to solve the supervised learning task. humans and animals are able to learn from very few labeled examples. we do 527


chapter 15. representation learning not yet know how this is possible. many factors could explain improved human performance — for example, the brain may use very large ensembles of classiﬁers or bayesian inference techniques. one popular hypothesis is that the brain is able to leverage unsupervised or semi - supervised learning. there are many ways to leverage unlabeled data. in this chapter, we focus on the hypothesis that the unlabeled data can be used to learn a good representation. 15. 1 greedy layer - wise unsupervised pretraining unsupervised learning played a key historical role in the revival of deep neural networks, enabling researchers for the ﬁrst time to train a deep supervised network without requiring architectural specializations like convolution or recurrence. we call this procedure unsupervised pretraining, or more precisely, greedy layer - wise unsupervised pretraining. this procedure is a canonical example of how a representation learned for one task ( unsupervised learning, trying to capture the shape of the input distribution ) can sometimes be useful for another task ( supervised learning with the same input domain ). greedy layer - wise unsupervised pretraining relies on


task ( unsupervised learning, trying to capture the shape of the input distribution ) can sometimes be useful for another task ( supervised learning with the same input domain ). greedy layer - wise unsupervised pretraining relies on a single - layer represen - tation learning algorithm such as an rbm, a single - layer autoencoder, a sparse coding model, or another model that learns latent representations. each layer is pretrained using unsupervised learning, taking the output of the previous layer and producing as output a new representation of the data, whose distribution ( or its relation to other variables such as categories to predict ) is hopefully simpler. see algorithm for a formal description. 15. 1 greedy layer - wise training procedures based on unsupervised criteria have long been used to sidestep the [UNK] of jointly training the layers of a deep neural net for a supervised task. this approach dates back at least as far as the neocognitron ( fukushima 1975, ). the deep learning renaissance of 2006 began with the discovery that this greedy learning procedure could be used to ﬁnd a good initialization for a joint learning procedure over all the layers, and that this approach could be used to successfully


fukushima 1975, ). the deep learning renaissance of 2006 began with the discovery that this greedy learning procedure could be used to ﬁnd a good initialization for a joint learning procedure over all the layers, and that this approach could be used to successfully train even fully connected architectures ( hinton 2006 hinton et al., ; and salakhutdinov 2006 hinton 2006 bengio 2007 ranzato 2007a, ;, ; et al., ; et al., ). prior to this discovery, only convolutional deep networks or networks whose depth resulted from recurrence were regarded as feasible to train. today, we now know that greedy layer - wise pretraining is not required to train fully connected deep architectures, but the unsupervised pretraining approach was the ﬁrst method to succeed. greedy layer - wise pretraining is called greedy because it is a greedy algo - 528


chapter 15. representation learning rithm, meaning that it optimizes each piece of the solution independently, one piece at a time, rather than jointly optimizing all pieces. it is called layer - wise because these independent pieces are the layers of the network. speciﬁcally, greedy layer - wise pretraining proceeds one layer at a time, training the k - th layer while keeping the previous ones ﬁxed. in particular, the lower layers ( which are trained ﬁrst ) are not adapted after the upper layers are introduced. it is called unsuper - vised because each layer is trained with an unsupervised representation learning algorithm. however it is also called pretraining, because it is supposed to be only a ﬁrst step before a joint training algorithm is applied to ﬁne - tune all the layers together. in the context of a supervised learning task, it can be viewed as a regularizer ( in some experiments, pretraining decreases test error without decreasing training error ) and a form of parameter initialization. it is common to use the word “ pretraining ” to refer not only to the pretraining stage itself but to the entire two phase protocol that combines the pretraining phase and a supervised learning phase


and a form of parameter initialization. it is common to use the word “ pretraining ” to refer not only to the pretraining stage itself but to the entire two phase protocol that combines the pretraining phase and a supervised learning phase. the supervised learning phase may involve training a simple classiﬁer on top of the features learned in the pretraining phase, or it may involve supervised ﬁne - tuning of the entire network learned in the pretraining phase. no matter what kind of unsupervised learning algorithm or what model type is employed, in the vast majority of cases, the overall training scheme is nearly the same. while the choice of unsupervised learning algorithm will obviously impact the details, most applications of unsupervised pretraining follow this basic protocol. greedy layer - wise unsupervised pretraining can also be used as initialization for other unsupervised learning algorithms, such as deep autoencoders ( hinton and salakhutdinov 2006, ) and probabilistic models with many layers of latent variables. such models include deep belief networks (, ) and deep hinton et al. 2006 boltzmann machines ( salakhutdinov and


salakhutdinov 2006, ) and probabilistic models with many layers of latent variables. such models include deep belief networks (, ) and deep hinton et al. 2006 boltzmann machines ( salakhutdinov and hinton 2009a, ). these deep generative models will be described in chapter. 20 as discussed in section, it is also possible to have greedy layer - wise 8. 7. 4 supervised pretraining. this builds on the premise that training a shallow network is easier than training a deep one, which seems to have been validated in several contexts (, ). erhan et al. 2010 15. 1. 1 when and why does unsupervised pretraining work? on many tasks, greedy layer - wise unsupervised pretraining can yield substantial improvements in test error for classiﬁcation tasks. this observation was responsible for the renewed interested in deep neural networks starting in 2006 ( hinton et al., 529


chapter 15. representation learning algorithm 15. 1 greedy layer - wise unsupervised pretraining protocol. given the following : unsupervised feature learning algorithm l, which takes a training set of examples and returns an encoder or feature function f. the raw input data is x, with one row per example and f ( 1 ) ( x ) is the output of the ﬁrst stage encoder on x. in the case where ﬁne - tuning is performed, we use a learner t which takes an initial function f, input examples x ( and in the supervised ﬁne - tuning case, associated targets y ), and returns a tuned function. the number of stages is. m f ←identity function [UNK] x x = for do k,..., m = 1 f ( ) k = ( l [UNK] x ) f f ← ( ) k [UNK] [UNK] x ←f ( ) k ( [UNK] ) end for if ﬁne - tuning then f f,, ←t ( x y ) end if return f 2006 bengio 2007 ranzato 2007a ; et al., ; et al., ). on many other tasks, however, unsupervised pretraining either does not confer


##t ( x y ) end if return f 2006 bengio 2007 ranzato 2007a ; et al., ; et al., ). on many other tasks, however, unsupervised pretraining either does not confer a beneﬁt or even causes noticeable harm. ( ) studied the [UNK] of pretraining on machine learning ma et al. 2015 models for chemical activity prediction and found that, on average, pretraining was slightly harmful, but for many tasks was signiﬁcantly helpful. because unsupervised pretraining is sometimes helpful but often harmful it is important to understand when and why it works in order to determine whether it is applicable to a particular task. at the outset, it is important to clarify that most of this discussion is restricted to greedy unsupervised pretraining in particular. there are other, completely [UNK] paradigms for performing semi - supervised learning with neural networks, such as virtual adversarial training described in section. it is also possible to 7. 13 train an autoencoder or generative model at the same time as the supervised model. examples of this single - stage approach include the discriminative rbm ( larochelle and bengio


it is also possible to 7. 13 train an autoencoder or generative model at the same time as the supervised model. examples of this single - stage approach include the discriminative rbm ( larochelle and bengio 2008, ) and the ladder network (, ), in which the total rasmus et al. 2015 objective is an explicit sum of the two terms ( one using the labels and one only using the input ). unsupervised pretraining combines two [UNK] ideas. first, it makes use of 530


chapter 15. representation learning the idea that the choice of initial parameters for a deep neural network can have a signiﬁcant regularizing [UNK] on the model ( and, to a lesser extent, that it can improve optimization ). second, it makes use of the more general idea that learning about the input distribution can help to learn about the mapping from inputs to outputs. both of these ideas involve many complicated interactions between several parts of the machine learning algorithm that are not entirely understood. the ﬁrst idea, that the choice of initial parameters for a deep neural network can have a strong regularizing [UNK] on its performance, is the least well understood. at the time that pretraining became popular, it was understood as initializing the model in a location that would cause it to approach one local minimum rather than another. today, local minima are no longer considered to be a serious problem for neural network optimization. we now know that our standard neural network training procedures usually do not arrive at a critical point of any kind. it remains possible that pretraining initializes the model in a location that would otherwise be inaccessible — for example, a region that is surrounded by areas where the cost function varies so much from one example to another that minibatches give only a very noisy


remains possible that pretraining initializes the model in a location that would otherwise be inaccessible — for example, a region that is surrounded by areas where the cost function varies so much from one example to another that minibatches give only a very noisy estimate of the gradient, or a region surrounded by areas where the hessian matrix is so poorly conditioned that gradient descent methods must use very small steps. however, our ability to characterize exactly what aspects of the pretrained parameters are retained during the supervised training stage is limited. this is one reason that modern approaches typically use simultaneous unsupervised learning and supervised learning rather than two sequential stages. one may also avoid struggling with these complicated ideas about how optimization in the supervised learning stage preserves information from the unsupervised learning stage by simply freezing the parameters for the feature extractors and using supervised learning only to add a classiﬁer on top of the learned features. the other idea, that a learning algorithm can use information learned in the unsupervised phase to perform better in the supervised learning stage, is better understood. the basic idea is that some features that are useful for the unsupervised task may also be useful for the supervised learning task. for example, if we train a


##ed phase to perform better in the supervised learning stage, is better understood. the basic idea is that some features that are useful for the unsupervised task may also be useful for the supervised learning task. for example, if we train a generative model of images of cars and motorcycles, it will need to know about wheels, and about how many wheels should be in an image. if we are fortunate, the representation of the wheels will take on a form that is easy for the supervised learner to access. this is not yet understood at a mathematical, theoretical level, so it is not always possible to predict which tasks will beneﬁt from unsupervised learning in this way. many aspects of this approach are highly dependent on the speciﬁc models used. for example, if we wish to add a linear classiﬁer on 531


chapter 15. representation learning top of pretrained features, the features must make the underlying classes linearly separable. these properties often occur naturally but do not always do so. this is another reason that simultaneous supervised and unsupervised learning can be preferable — the constraints imposed by the output layer are naturally included from the start. from the point of view of unsupervised pretraining as learning a representation, we can expect unsupervised pretraining to be more [UNK] when the initial representation is poor. one key example of this is the use of word embeddings. words represented by one - hot vectors are not very informative because every two distinct one - hot vectors are the same distance from each other ( squaredl2 distance of ). learned word embeddings naturally encode similarity between words by their 2 distance from each other. because of this, unsupervised pretraining is especially useful when processing words. it is less useful when processing images, perhaps because images already lie in a rich vector space where distances provide a low quality similarity metric. from the point of view of unsupervised pretraining as a regularizer, we can expect unsupervised pretraining


, perhaps because images already lie in a rich vector space where distances provide a low quality similarity metric. from the point of view of unsupervised pretraining as a regularizer, we can expect unsupervised pretraining to be most helpful when the number of labeled examples is very small. because the source of information added by unsupervised pretraining is the unlabeled data, we may also expect unsupervised pretraining to perform best when the number of unlabeled examples is very large. the advantage of semi - supervised learning via unsupervised pretraining with many unlabeled examples and few labeled examples was made particularly clear in 2011 with unsupervised pretraining winning two international transfer learning competitions (, ;, ), in settings where the mesnil et al. 2011 goodfellow et al. 2011 number of labeled examples in the target task was small ( from a handful to dozens of examples per class ). these [UNK] were also documented in carefully controlled experiments by paine 2014 et al. ( ). other factors are likely to be involved. for example, unsupervised pretraining is likely to be most useful when the


per class ). these [UNK] were also documented in carefully controlled experiments by paine 2014 et al. ( ). other factors are likely to be involved. for example, unsupervised pretraining is likely to be most useful when the function to be learned is extremely complicated. unsupervised learning [UNK] from regularizers like weight decay because it does not bias the learner toward discovering a simple function but rather toward discovering feature functions that are useful for the unsupervised learning task. if the true underlying functions are complicated and shaped by regularities of the input distribution, unsupervised learning can be a more appropriate regularizer. these caveats aside, we now analyze some success cases where unsupervised pretraining is known to cause an improvement, and explain what is known about why this improvement occurs. unsupervised pretraining has usually been used to improve classiﬁers, and is usually most interesting from the point of view of 532


chapter 15. representation learning figure 15. 1 : visualization via nonlinear projection of the learning trajectories of [UNK] neural networks in function space ( not parameter space, to avoid the issue of many - to - one mappings from parameter vectors to functions ), with [UNK] random initializations and with or without unsupervised pretraining. each point corresponds to a [UNK] neural network, at a particular time during its training process. this ﬁgure is adapted with permission from ( ). a coordinate in function space is an inﬁnite - erhan et al. 2010 dimensional vector associating every input x with an output y. ( ) made erhan et al. 2010 a linear projection to high - dimensional space by concatenating they for many speciﬁc x points. they then made a further nonlinear projection to 2 - d by isomap ( tenenbaum et al., ). color indicates time. all networks are initialized near the center of the plot 2000 ( corresponding to the region of functions that produce approximately uniform distributions over the class y for most inputs ). over time, learning moves the function outward, to points that make strong predictions. training consistently terminates in one region when using pretraining and in another


corresponding to the region of functions that produce approximately uniform distributions over the class y for most inputs ). over time, learning moves the function outward, to points that make strong predictions. training consistently terminates in one region when using pretraining and in another, non - overlapping region when not using pretraining. isomap tries to preserve global relative distances ( and hence volumes ) so the small region corresponding to pretrained models may indicate that the pretraining - based estimator has reduced variance. 533


chapter 15. representation learning reducing test set error. however, unsupervised pretraining can help tasks other than classiﬁcation, and can act to improve optimization rather than being merely a regularizer. for example, it can improve both train and test reconstruction error for deep autoencoders ( hinton and salakhutdinov 2006, ). erhan 2010 et al. ( ) performed many experiments to explain several successes of unsupervised pretraining. both improvements to training error and improvements to test error may be explained in terms of unsupervised pretraining taking the parameters into a region that would otherwise be inaccessible. neural network training is non - deterministic, and converges to a [UNK] function every time it is run. training may halt at a point where the gradient becomes small, a point where early stopping ends training to prevent overﬁtting, or at a point where the gradient is large but it is [UNK] to ﬁnd a downhill step due to problems such as stochasticity or poor conditioning of the hessian. neural networks that receive unsupervised pretraining consistently halt in the same region of function space, while neural networks without pretraining consistently halt in another region. see


such as stochasticity or poor conditioning of the hessian. neural networks that receive unsupervised pretraining consistently halt in the same region of function space, while neural networks without pretraining consistently halt in another region. see ﬁgure for a visualization of this phenomenon. the region where pretrained 15. 1 networks arrive is smaller, suggesting that pretraining reduces the variance of the estimation process, which can in turn reduce the risk of severe over - ﬁtting. in other words, unsupervised pretraining initializes neural network parameters into a region that they do not escape, and the results following this initialization are more consistent and less likely to be very bad than without this initialization. erhan 2010 et al. ( ) also provide some answers as to pretraining works when best — the mean and variance of the test error were most reduced by pretraining for deeper networks. keep in mind that these experiments were performed before the invention and popularization of modern techniques for training very deep networks ( rectiﬁed linear units, dropout and batch normalization ) so less is known about the [UNK] of unsupervised pretraining in conjunction with contemporary approaches. an important question is how un


modern techniques for training very deep networks ( rectiﬁed linear units, dropout and batch normalization ) so less is known about the [UNK] of unsupervised pretraining in conjunction with contemporary approaches. an important question is how unsupervised pretraining can act as a regularizer. one hypothesis is that pretraining encourages the learning algorithm to discover features that relate to the underlying causes that generate the observed data. this is an important idea motivating many other algorithms besides unsupervised pretraining, and is described further in section. 15. 3 compared to other forms of unsupervised learning, unsupervised pretraining has the disadvantage that it operates with two separate training phases. many regularization strategies have the advantage of allowing the user to control the strength of the regularization by adjusting the value of a single hyperparameter. unsupervised pretraining does not [UNK] a clear way to adjust the the strength of the regularization arising from the unsupervised stage. instead, there are 534


chapter 15. representation learning very many hyperparameters, whose [UNK] may be measured after the fact but is often [UNK] to predict ahead of time. when we perform unsupervised and supervised learning simultaneously, instead of using the pretraining strategy, there is a single hyperparameter, usually a [UNK] attached to the unsupervised cost, that determines how strongly the unsupervised objective will regularize the supervised model. one can always predictably obtain less regularization by decreasing this [UNK]. in the case of unsupervised pretraining, there is not a way of ﬂexibly adapting the strength of the regularization — either the supervised model is initialized to pretrained parameters, or it is not. another disadvantage of having two separate training phases is that each phase has its own hyperparameters. the performance of the second phase usually cannot be predicted during the ﬁrst phase, so there is a long delay between proposing hyperparameters for the ﬁrst phase and being able to update them using feedback from the second phase. the most principled approach is to use validation set error in the supervised phase in order to select the hyperparameters of the pretraining phase, as discussed in ( ).


and being able to update them using feedback from the second phase. the most principled approach is to use validation set error in the supervised phase in order to select the hyperparameters of the pretraining phase, as discussed in ( ). in practice, some hyperparameters, larochelle et al. 2009 like the number of pretraining iterations, are more conveniently set during the pretraining phase, using early stopping on the unsupervised objective, which is not ideal but computationally much cheaper than using the supervised objective. today, unsupervised pretraining has been largely abandoned, except in the ﬁeld of natural language processing, where the natural representation of words as one - hot vectors conveys no similarity information and where very large unlabeled sets are available. in that case, the advantage of pretraining is that one can pretrain once on a huge unlabeled set ( for example with a corpus containing billions of words ), learn a good representation ( typically of words, but also of sentences ), and then use this representation or ﬁne - tune it for a supervised task for which the training set contains substantially fewer examples. this approach was pioneered by by collobert and


learn a good representation ( typically of words, but also of sentences ), and then use this representation or ﬁne - tune it for a supervised task for which the training set contains substantially fewer examples. this approach was pioneered by by collobert and weston 2008b turian 2010 collobert ( ), et al. ( ), and et al. ( ) and remains in common use today. 2011a deep learning techniques based on supervised learning, regularized with dropout or batch normalization, are able to achieve human - level performance on very many tasks, but only with extremely large labeled datasets. these same techniques out - perform unsupervised pretraining on medium - sized datasets such as cifar - 10 and mnist, which have roughly 5, 000 labeled examples per class. on extremely small datasets, such as the alternative splicing dataset, bayesian methods outperform methods based on unsupervised pretraining ( srivastava 2013, ). for these reasons, the popularity of unsupervised pretraining has declined. nevertheless, unsupervised pretraining remains an important milestone in the history of deep learning research 535


chapter 15. representation learning and continues to inﬂuence contemporary approaches. the idea of pretraining has been generalized to supervised pretraining discussed in section, as a very 8. 7. 4 common approach for transfer learning. supervised pretraining for transfer learning is popular (, ; oquab et al. 2014 yosinski 2014 et al., ) for use with convolutional networks pretrained on the imagenet dataset. practitioners publish the parameters of these trained networks for this purpose, just like pretrained word vectors are published for natural language tasks (, ; collobert et al. 2011a mikolov 2013a et al., ). 15. 2 transfer learning and domain adaptation transfer learning and domain adaptation refer to the situation where what has been learned in one setting ( i. e., distribution p1 ) is exploited to improve generalization in another setting ( say distribution p2 ). this generalizes the idea presented in the previous section, where we transferred representations between an unsupervised learning task and a supervised learning task. in transfer learning, the learner must perform two or more [UNK] tasks, but we assume that many of the factors that explain the variations in p1 are relevant to the


representations between an unsupervised learning task and a supervised learning task. in transfer learning, the learner must perform two or more [UNK] tasks, but we assume that many of the factors that explain the variations in p1 are relevant to the variations that need to be captured for learning p2. this is typically understood in a supervised learning context, where the input is the same but the target may be of a [UNK] nature. for example, we may learn about one set of visual categories, such as cats and dogs, in the ﬁrst setting, then learn about a [UNK] set of visual categories, such as ants and wasps, in the second setting. if there is signiﬁcantly more data in the ﬁrst setting ( sampled from p1 ), then that may help to learn representations that are useful to quickly generalize from only very few examples drawn from p2. many visual categories share low - level notions of edges and visual shapes, the [UNK] of geometric changes, changes in lighting, etc. in general, transfer learning, multi - task learning ( section ), and domain 7. 7 adaptation can be achieved via representation learning when there exist features that are useful for the [UNK] settings or tasks, corresponding to underlying factors that appear in more than one setting


general, transfer learning, multi - task learning ( section ), and domain 7. 7 adaptation can be achieved via representation learning when there exist features that are useful for the [UNK] settings or tasks, corresponding to underlying factors that appear in more than one setting. this is illustrated in ﬁgure, with 7. 2 shared lower layers and task - dependent upper layers. however, sometimes, what is shared among the [UNK] tasks is not the semantics of the input but the semantics of the output. for example, a speech recognition system needs to produce valid sentences at the output layer, but the earlier layers near the input may need to recognize very [UNK] versions of the same phonemes or sub - phonemic vocalizations depending on which person is speaking. in cases like these, it makes more sense to share the upper layers ( near the output ) of the neural network, and have a task - speciﬁc preprocessing, as 536


chapter 15. representation learning illustrated in ﬁgure. 15. 2 selection switch h ( 1 ) h ( 1 ) h ( 2 ) h ( 2 ) h ( 3 ) h ( 3 ) y h ( shared ) h ( shared ) x ( 1 ) x ( 1 ) x ( 2 ) x ( 2 ) x ( 3 ) x ( 3 ) figure 15. 2 : example architecture for multi - task or transfer learning when the output variable has the same semantics for all tasks while the input variable has a [UNK] y x meaning ( and possibly even a [UNK] dimension ) for each task ( or, for example, each user ), called x ( 1 ), x ( 2 ) and x ( 3 ) for three tasks. the lower levels ( up to the selection switch ) are task - speciﬁc, while the upper levels are shared. the lower levels learn to translate their task - speciﬁc input into a generic set of features. in the related case of domain adaptation, the task ( and the optimal input - to - output mapping ) remains the same between each setting, but the input distribution is slightly [UNK]. for example, consider the task of sentiment analysis, which consists of determining whether a comment expresses positive or negative sentiment. comments posted on


the optimal input - to - output mapping ) remains the same between each setting, but the input distribution is slightly [UNK]. for example, consider the task of sentiment analysis, which consists of determining whether a comment expresses positive or negative sentiment. comments posted on the web come from many categories. a domain adaptation scenario can arise when a sentiment predictor trained on customer reviews of media content such as books, videos and music is later used to analyze comments about consumer electronics such as televisions or smartphones. one can imagine that there is an underlying function that tells whether any statement is positive, neutral or negative, but of course the vocabulary and style may vary from one domain to another, making it more [UNK] to generalize across domains. simple unsupervised pretraining ( with denoising autoencoders ) has been found to be very successful for sentiment analysis with domain adaptation (, ). glorot et al. 2011b a related problem is that of concept drift, which we can view as a form of transfer learning due to gradual changes in the data distribution over time. both concept drift and transfer learning can be viewed as particular forms of 537


chapter 15. representation learning multi - task learning. while the phrase “ multi - task learning ” typically refers to supervised learning tasks, the more general notion of transfer learning is applicable to unsupervised learning and reinforcement learning as well. in all of these cases, the objective is to take advantage of data from the ﬁrst setting to extract information that may be useful when learning or even when directly making predictions in the second setting. the core idea of representation learning is that the same representation may be useful in both settings. using the same representation in both settings allows the representation to beneﬁt from the training data that is available for both tasks. as mentioned before, unsupervised deep learning for transfer learning has found success in some machine learning competitions (, ; mesnil et al. 2011 goodfellow et al., ). in the ﬁrst of these competitions, the experimental setup is the 2011 following. each participant is ﬁrst given a dataset from the ﬁrst setting ( from distribution p1 ), illustrating examples of some set of categories. the participants must use this to learn a good feature space ( mapping the raw input to some representation ), such that when we apply this learned transformation to inputs from the transfer setting ( distribution p


p1 ), illustrating examples of some set of categories. the participants must use this to learn a good feature space ( mapping the raw input to some representation ), such that when we apply this learned transformation to inputs from the transfer setting ( distribution p2 ), a linear classiﬁer can be trained and generalize well from very few labeled examples. one of the most striking results found in this competition is that as an architecture makes use of deeper and deeper representations ( learned in a purely unsupervised way from data collected in the ﬁrst setting, p1 ), the learning curve on the new categories of the second ( transfer ) setting p2 becomes much better. for deep representations, fewer labeled examples of the transfer tasks are necessary to achieve the apparently asymptotic generalization performance. two extreme forms of transfer learning are one - shot learning and zero - shot learning, sometimes also called zero - data learning. only one labeled example of the transfer task is given for one - shot learning, while no labeled examples are given at all for the zero - shot learning task. one - shot learning ( fei - fei 2006 et al., ) is possible because the representation learns to cleanly separate the underlying classes during the ﬁrst stage. during the transfer


labeled examples are given at all for the zero - shot learning task. one - shot learning ( fei - fei 2006 et al., ) is possible because the representation learns to cleanly separate the underlying classes during the ﬁrst stage. during the transfer learning stage, only one labeled example is needed to infer the label of many possible test examples that all cluster around the same point in representation space. this works to the extent that the factors of variation corresponding to these invariances have been cleanly separated from the other factors, in the learned representation space, and we have somehow learned which factors do and do not matter when discriminating objects of certain categories. as an example of a zero - shot learning setting, consider the problem of having a learner read a large collection of text and then solve object recognition problems. 538


chapter 15. representation learning it may be possible to recognize a speciﬁc object class even without having seen an image of that object, if the text describes the object well enough. for example, having read that a cat has four legs and pointy ears, the learner might be able to guess that an image is a cat, without having seen a cat before. zero - data learning ( larochelle 2008 palatucci et al., ) and zero - shot learning ( et al., ; 2009 socher 2013b et al., ) are only possible because additional information has been exploited during training. we can think of the zero - data learning scenario as including three random variables : the traditional inputs x, the traditional outputs or targets y, and an additional random variable describing the task, t. the model is trained to estimate the conditional distribution p ( y x |, t ) where t is a description of the task we wish the model to perform. in our example of recognizing cats after having read about cats, the output is a binary variable y with y = 1 indicating “ yes ” and y = 0 indicating “ no. ” the task variable t then represents questions to be answered such as “ is there a cat in this image? ” if


about cats, the output is a binary variable y with y = 1 indicating “ yes ” and y = 0 indicating “ no. ” the task variable t then represents questions to be answered such as “ is there a cat in this image? ” if we have a training set containing unsupervised examples of objects that live in the same space as t, we may be able to infer the meaning of unseen instances of t. in our example of recognizing cats without having seen an image of the cat, it is important that we have had unlabeled text data containing sentences such as “ cats have four legs ” or “ cats have pointy ears. ” zero - shot learning requires t to be represented in a way that allows some sort of generalization. for example, t cannot be just a one - hot code indicating an object category. ( ) provide instead a distributed representation socher et al. 2013b of object categories by using a learned word embedding for the word associated with each category. a similar phenomenon happens in machine translation ( klementiev 2012 et al., ; mikolov 2013b gouws 2014 et al., ; et al., ) : we have words in one language, and the relationships between words


similar phenomenon happens in machine translation ( klementiev 2012 et al., ; mikolov 2013b gouws 2014 et al., ; et al., ) : we have words in one language, and the relationships between words can be learned from unilingual corpora ; on the other hand, we have translated sentences which relate words in one language with words in the other. even though we may not have labeled examples translating word a in language x to word b in language y, we can generalize and guess a translation for word a because we have learned a distributed representation for words in language x, a distributed representation for words in language y, and created a link ( possibly two - way ) relating the two spaces, via training examples consisting of matched pairs of sentences in both languages. this transfer will be most successful if all three ingredients ( the two representations and the relations between them ) are learned jointly. zero - shot learning is a particular form of transfer learning. the same principle explains how one can perform multi - modal learning, capturing a representation 539


chapter 15. representation learning hx = fx ( ) x xtest ytest hy = fy ( ) y y−space relationship between embedded points within one of the domains maps between representation spaces fx fy x−space ( ) pairs in the training set x y, fx : encoder function for x fy : encoder function for y figure 15. 3 : transfer learning between two domains x and y enables zero - shot learning. labeled or unlabeled examples of x allow one to learn a representation function fx and similarly with examples of y to learn fy. each application of the f x and f y functions appears as an upward arrow, with the style of the arrows indicating which function is applied. distance in hx space provides a similarity metric between any pair of points in x space that may be more meaningful than distance in x space. likewise, distance in hy space provides a similarity metric between any pair of points in y space. both of these similarity functions are indicated with dotted bidirectional arrows. labeled examples ( dashed horizontal lines ) are pairs ( x y, ) which allow one to learn a one - way or two - way map ( solid bidirectional arrow ) between the representationsfx ( x ) and


##irectional arrows. labeled examples ( dashed horizontal lines ) are pairs ( x y, ) which allow one to learn a one - way or two - way map ( solid bidirectional arrow ) between the representationsfx ( x ) and the representations f y ( y ) and anchor these representations to each other. zero - data learning is then enabled as follows. one can associate an image xtest to a word ytest, even if no image of that word was ever presented, simply because word - representationsfy ( ytest ) and image - representations fx ( xtest ) can be related to each other via the maps between representation spaces. it works because, although that image and that word were never paired, their respective feature vectors fx ( xtest ) and fy ( ytest ) have been related to each other. figure inspired from suggestion by hrant khachatrian. 540


chapter 15. representation learning in one modality, a representation in the other, and the relationship ( in general a joint distribution ) between pairs ( x y, ) consisting of one observation x in one modality and another observation y in the other modality ( srivastava and salakhutdinov, 2012 ). by learning all three sets of parameters ( from x to its representation, from y to its representation, and the relationship between the two representations ), concepts in one representation are anchored in the other, and vice - versa, allowing one to meaningfully generalize to new pairs. the procedure is illustrated in ﬁgure. 15. 3 15. 3 semi - supervised disentangling of causal factors an important question about representation learning is “ what makes one repre - sentation better than another? ” one hypothesis is that an ideal representation is one in which the features within the representation correspond to the under - lying causes of the observed data, with separate features or directions in feature space corresponding to [UNK] causes, so that the representation disentangles the causes from one another. this hypothesis motivates approaches in which we ﬁrst seek a good representation for p ( x ). such a representation may also be a good representation for computing p (


causes, so that the representation disentangles the causes from one another. this hypothesis motivates approaches in which we ﬁrst seek a good representation for p ( x ). such a representation may also be a good representation for computing p ( y x | ) if y is among the most salient causes of x. this idea has guided a large amount of deep learning research since at least the 1990s ( becker and hinton 1992 hinton and sejnowski 1999, ;, ), in more detail. for other arguments about when semi - supervised learning can outperform pure supervised learning, we refer the reader to section 1. 2 of ( ). chapelle et al. 2006 in other approaches to representation learning, we have often been concerned with a representation that is easy to model — for example, one whose entries are sparse, or independent from each other. a representation that cleanly separates the underlying causal factors may not necessarily be one that is easy to model. however, a further part of the hypothesis motivating semi - supervised learning via unsupervised representation learning is that for many ai tasks, these two properties coincide : once we are able to obtain the underlying explanations for what we observe, it generally becomes easy to isolate individual attributes from the


##tivating semi - supervised learning via unsupervised representation learning is that for many ai tasks, these two properties coincide : once we are able to obtain the underlying explanations for what we observe, it generally becomes easy to isolate individual attributes from the others. speciﬁcally, if a representation h represents many of the underlying causes of the observed x, and the outputs y are among the most salient causes, then it is easy to predict from. y h first, let us see how semi - supervised learning can fail because unsupervised learning of p ( x ) is of no help to learn p ( y x | ). consider for example the case where p ( x ) is uniformly distributed and we want to learn f ( x ) = e [ y | x ]. clearly, observing a training set of values alone gives us no information about. x p ( ) y x | 541


chapter 15. representation learning x p x ( ) y = 1 y = 2 y = 3 figure 15. 4 : example of a density over x that is a mixture over three components. the component identity is an underlying explanatory factor, y. because the mixture components ( e. g., natural object classes in image data ) are statistically salient, just modeling p ( x ) in an unsupervised way with no labeled example already reveals the factor y. next, let us see a simple example of how semi - supervised learning can succeed. consider the situation where x arises from a mixture, with one mixture component per value of y, as illustrated in ﬁgure. if the mixture components are well - 15. 4 separated, then modeling p ( x ) reveals precisely where each component is, and a single labeled example of each class will then be enough to perfectly learn p ( y x | ). but more generally, what could make and be tied together? p ( ) y x | p ( ) x if y is closely associated with one of the causal factors of x, then p ( x ) and p ( y x | ) will be strongly tied, and unsupervised representation learning that tries to disentangle


| p ( ) x if y is closely associated with one of the causal factors of x, then p ( x ) and p ( y x | ) will be strongly tied, and unsupervised representation learning that tries to disentangle the underlying factors of variation is likely to be useful as a semi - supervised learning strategy. consider the assumption that y is one of the causal factors of x, and let h represent all those factors. the true generative process can be conceived as structured according to this directed graphical model, with as the parent of : h x p, p p. ( h x ) = ( ) x h | ( ) h ( 15. 1 ) as a consequence, the data has marginal probability p ( ) = x ehp. ( ) x h | ( 15. 2 ) from this straightforward observation, we conclude that the best possible model of x ( from a generalization point of view ) is the one that uncovers the above “ true ” 542


chapter 15. representation learning structure, with h as a latent variable that explains the observed variations in x. the “ ideal ” representation learning discussed above should thus recover these latent factors. if y is one of these ( or closely related to one of them ), then it will be very easy to learn to predict y from such a representation. we also see that the conditional distribution of y given x is tied by bayes ’ rule to the components in the above equation : p ( ) = y x | p p ( ) x y | ( ) y p ( ) x. ( 15. 3 ) thus the marginal p ( x ) is intimately tied to the conditional p ( y x | ) and knowledge of the structure of the former should be helpful to learn the latter. therefore, in situations respecting these assumptions, semi - supervised learning should improve performance. an important research problem regards the fact that most observations are formed by an extremely large number of underlying causes. suppose y = hi, but the unsupervised learner does not know which hi. the brute force solution is for an unsupervised learner to learn a representation that captures the reasonably all salient generative factors hj and disentangles them from each other, thus


##ed learner does not know which hi. the brute force solution is for an unsupervised learner to learn a representation that captures the reasonably all salient generative factors hj and disentangles them from each other, thus making it easy to predict from, regardless of which h y h i is associated with. y in practice, the brute force solution is not feasible because it is not possible to capture all or most of the factors of variation that inﬂuence an observation. for example, in a visual scene, should the representation always encode all of the smallest objects in the background? it is a well - documented psychological phenomenon that human beings fail to perceive changes in their environment that are not immediately relevant to the task they are performing — see, e. g., simons and levin 1998 ( ). an important research frontier in semi - supervised learning is determining to encode in each situation. currently, two of the main strategies what for dealing with a large number of underlying causes are to use a supervised learning signal at the same time as the unsupervised learning signal so that the model will choose to capture the most relevant factors of variation, or to use much larger representations if using purely unsupervised learning.


to use a supervised learning signal at the same time as the unsupervised learning signal so that the model will choose to capture the most relevant factors of variation, or to use much larger representations if using purely unsupervised learning. an emerging strategy for unsupervised learning is to modify the deﬁnition of which underlying causes are most salient. historically, autoencoders and generative models have been trained to optimize a ﬁxed criterion, often similar to mean squared error. these ﬁxed criteria determine which causes are considered salient. for example, mean squared error applied to the pixels of an image implicitly speciﬁes that an underlying cause is only salient if it signiﬁcantly changes the brightness of a large number of pixels. this can be problematic if the task we wish to solve involves interacting with small objects. see ﬁgure for an example 15. 5 543


chapter 15. representation learning input reconstruction figure 15. 5 : an autoencoder trained with mean squared error for a robotics task has failed to reconstruct a ping pong ball. the existence of the ping pong ball and all of its spatial coordinates are important underlying causal factors that generate the image and are relevant to the robotics task. unfortunately, the autoencoder has limited capacity, and the training with mean squared error did not identify the ping pong ball as being salient enough to encode. images graciously provided by chelsea finn. of a robotics task in which an autoencoder has failed to learn to encode a small ping pong ball. this same robot is capable of successfully interacting with larger objects, such as baseballs, which are more salient according to mean squared error. other deﬁnitions of salience are possible. for example, if a group of pixels follow a highly recognizable pattern, even if that pattern does not involve extreme brightness or darkness, then that pattern could be considered extremely salient. one way to implement such a deﬁnition of salience is to use a recently developed approach called generative adversarial networks (, ). goodfellow et al. 2014c in this approach


could be considered extremely salient. one way to implement such a deﬁnition of salience is to use a recently developed approach called generative adversarial networks (, ). goodfellow et al. 2014c in this approach, a generative model is trained to fool a feedforward classiﬁer. the feedforward classiﬁer attempts to recognize all samples from the generative model as being fake, and all samples from the training set as being real. in this framework, any structured pattern that the feedforward network can recognize is highly salient. the generative adversarial network will be described in more detail in section. for the purposes of the present discussion, it is [UNK] to 20. 10. 4 understand that they learn how to determine what is salient. ( ) lotter et al. 2015 showed that models trained to generate images of human heads will often neglect to generate the ears when trained with mean squared error, but will successfully generate the ears when trained with the adversarial framework. because the ears are not extremely bright or dark compared to the surrounding skin, they are not especially salient according to mean squared error loss, but their highly 544


chapter 15. representation learning ground truth mse adversarial figure 15. 6 : predictive generative networks provide an example of the importance of learning which features are salient. in this example, the predictive generative network has been trained to predict the appearance of a 3 - d model of a human head at a speciﬁc viewing angle. ( left ) ground truth. this is the correct image, that the network should emit. image produced by a predictive generative network trained with mean ( center ) squared error alone. because the ears do not cause an extreme [UNK] in brightness compared to the neighboring skin, they were not [UNK] salient for the model to learn to represent them. ( right ) image produced by a model trained with a combination of mean squared error and adversarial loss. using this learned cost function, the ears are salient because they follow a predictable pattern. learning which underlying causes are important and relevant enough to model is an important active area of research. figures graciously provided by ( ). lotter et al. 2015 recognizable shape and consistent position means that a feedforward network can easily learn to detect them, making them highly salient under the generative adversarial framework. see ﬁgure


##ly provided by ( ). lotter et al. 2015 recognizable shape and consistent position means that a feedforward network can easily learn to detect them, making them highly salient under the generative adversarial framework. see ﬁgure for example images. generative adversarial 15. 6 networks are only one step toward determining which factors should be represented. we expect that future research will discover better ways of determining which factors to represent, and develop mechanisms for representing [UNK] factors depending on the task. a beneﬁt of learning the underlying causal factors, as pointed out by scholkopf et al. ( ), is that if the true generative process has 2012 x as an [UNK] and y as a cause, then modeling p ( x y | ) is robust to changes in p ( y ). if the cause - [UNK] relationship was reversed, this would not be true, since by bayes ’ rule, p ( x y | ) would be sensitive to changes in p ( y ). very often, when we consider changes in distribution due to [UNK] domains, temporal non - stationarity, or changes in the nature of the task, the causal mechanisms remain invariant ( the laws of the universe are constant ) while the marginal distribution over


. very often, when we consider changes in distribution due to [UNK] domains, temporal non - stationarity, or changes in the nature of the task, the causal mechanisms remain invariant ( the laws of the universe are constant ) while the marginal distribution over the underlying causes can change. hence, better generalization and robustness to all kinds of changes can 545


chapter 15. representation learning be expected via learning a generative model that attempts to recover the causal factors and. h p ( ) x h | 15. 4 distributed representation distributed representations of concepts — representations composed of many ele - ments that can be set separately from each other — are one of the most important tools for representation learning. distributed representations are powerful because they can use n features with k values to describe kn [UNK] concepts. as we have seen throughout this book, both neural networks with multiple hidden units and probabilistic models with multiple latent variables make use of the strategy of distributed representation. we now introduce an additional observation. many deep learning algorithms are motivated by the assumption that the hidden units can learn to represent the underlying causal factors that explain the data, as discussed in section. distributed representations are natural for this approach, 15. 3 because each direction in representation space can correspond to the value of a [UNK] underlying conﬁguration variable. an example of a distributed representation is a vector of n binary features, which can take 2n conﬁgurations, each potentially corresponding to a [UNK] region in input space, as illustrated in ﬁgure. this can be compared with 15. 7 a symbolic representation, where the input is associated with a single symbol or


which can take 2n conﬁgurations, each potentially corresponding to a [UNK] region in input space, as illustrated in ﬁgure. this can be compared with 15. 7 a symbolic representation, where the input is associated with a single symbol or category. if there are n symbols in the dictionary, one can imagine n feature detectors, each corresponding to the detection of the presence of the associated category. in that case only n [UNK] conﬁgurations of the representation space are possible, carving n [UNK] regions in input space, as illustrated in ﬁgure. 15. 8 such a symbolic representation is also called a one - hot representation, since it can be captured by a binary vector with n bits that are mutually exclusive ( only one of them can be active ). a symbolic representation is a speciﬁc example of the broader class of non - distributed representations, which are representations that may contain many entries but without signiﬁcant meaningful separate control over each entry. examples of learning algorithms based on non - distributed representations include : • clustering methods, including the k - means algorithm : each input point is assigned to exactly one cluster. • k - nearest neighbors algorithms : one or a few templates or prototype examples are associated with a given input. in the


representations include : • clustering methods, including the k - means algorithm : each input point is assigned to exactly one cluster. • k - nearest neighbors algorithms : one or a few templates or prototype examples are associated with a given input. in the case of k > 1, there are multiple 546


chapter 15. representation learning h1 h2 h3 h = [ 1,, 1 1 ] h = [ 0,, 1 1 ] h = [ 1,, 0 1 ] h = [ 1,, 1 0 ] h = [ 0,, 1 0 ] h = [ 0,, 0 1 ] h = [ 1,, 0 0 ] figure 15. 7 : illustration of how a learning algorithm based on a distributed representation breaks up the input space into regions. in this example, there are three binary features h1, h2, and h3. each feature is deﬁned by thresholding the output of a learned, linear transformation. each feature divides r2 into two half - planes. let h + i be the set of input points for which hi = 1 and h− i be the set of input points for which hi = 0. in this illustration, each line represents the decision boundary for onehi, with the corresponding arrow pointing to the h + i side of the boundary. the representation as a whole takes on a unique value at each possible intersection of these half - planes. for example, the representation value [ 1, 1, 1 ] corresponds to the region h + 1 ∩h + 2 ∩h + 3. compare


the representation as a whole takes on a unique value at each possible intersection of these half - planes. for example, the representation value [ 1, 1, 1 ] corresponds to the region h + 1 ∩h + 2 ∩h + 3. compare this to the non - distributed representations in ﬁgure. in the general case of 15. 8 d input dimensions, a distributed representation divides rd by intersecting half - spaces rather than half - planes. the distributed representation with n features assigns unique codes to o ( nd ) [UNK] regions, while the nearest neighbor algorithm withn examples assigns unique codes to only n regions. the distributed representation is thus able to distinguish exponentially many more regions than the non - distributed one. keep in mind that not allh values are feasible ( there is no h = 0 in this example ) and that a linear classiﬁer on top of the distributed representation is not able to assign [UNK] class identities to every neighboring region ; even a deep linear - threshold network has a vc dimension of onlyo ( w w log ) where w is the number of weights (, ). the combination of a powerful representation sontag 1998 layer and a weak classiﬁer layer can be a strong regularizer ; a classiﬁer trying to learn the


( w w log ) where w is the number of weights (, ). the combination of a powerful representation sontag 1998 layer and a weak classiﬁer layer can be a strong regularizer ; a classiﬁer trying to learn the concept of “ person ” versus “ not a person ” does not need to assign a [UNK] class to an input represented as “ woman with glasses ” than it assigns to an input represented as “ man without glasses. ” this capacity constraint encourages each classiﬁer to focus on few hi and encourages to learn to represent the classes in a linearly separable way. h 547


chapter 15. representation learning values describing each input, but they can not be controlled separately from each other, so this does not qualify as a true distributed representation. • decision trees : only one leaf ( and the nodes on the path from root to leaf ) is activated when an input is given. • gaussian mixtures and mixtures of experts : the templates ( cluster centers ) or experts are now associated with a degree of activation. as with the k - nearest neighbors algorithm, each input is represented with multiple values, but those values cannot readily be controlled separately from each other. • kernel machines with a gaussian kernel ( or other similarly local kernel ) : although the degree of activation of each “ support vector ” or template example is now continuous - valued, the same issue arises as with gaussian mixtures. • language or translation models based on n - grams. the set of contexts ( sequences of symbols ) is partitioned according to a tree structure of [UNK]. a leaf may correspond to the last two words being w1 and w2, for example. separate parameters are estimated for each leaf of the tree ( with some sharing being possible ). for some of these non - distributed algorithms, the output is not constant by parts but instead interpolates


words being w1 and w2, for example. separate parameters are estimated for each leaf of the tree ( with some sharing being possible ). for some of these non - distributed algorithms, the output is not constant by parts but instead interpolates between neighboring regions. the relationship between the number of parameters ( or examples ) and the number of regions they can deﬁne remains linear. an important related concept that distinguishes a distributed representation from a symbolic one is that generalization arises due to shared attributes between [UNK] concepts. as pure symbols, “ cat ” and “ dog ” are as far from each other as any other two symbols. however, if one associates them with a meaningful distributed representation, then many of the things that can be said about cats can generalize to dogs and vice - versa. for example, our distributed representation may contain entries such as “ has _ fur ” or “ number _ of _ legs ” that have the same value for the embedding of both “ cat ” and “ dog. ” neural language models that operate on distributed representations of words generalize much better than other models that operate directly on one - hot representations of words, as discussed in section. distributed representations induce a rich 12. 4 similarity space, in which semantically close concepts ( or


models that operate on distributed representations of words generalize much better than other models that operate directly on one - hot representations of words, as discussed in section. distributed representations induce a rich 12. 4 similarity space, in which semantically close concepts ( or inputs ) are close in distance, a property that is absent from purely symbolic representations. when and why can there be a statistical advantage from using a distributed representation as part of a learning algorithm? distributed representations can 548


chapter 15. representation learning figure 15. 8 : illustration of how the nearest neighbor algorithm breaks up the input space into [UNK] regions. the nearest neighbor algorithm provides an example of a learning algorithm based on a non - distributed representation. [UNK] non - distributed algorithms may have [UNK] geometry, but they typically break the input space into regions, with a separate set of parameters for each region. the advantage of a non - distributed approach is that, given enough parameters, it can ﬁt the training set without solving a [UNK] optimization algorithm, because it is straightforward to choose a [UNK] output independently for each region. the disadvantage is that such non - distributed models generalize only locally via the smoothness prior, making it [UNK] to learn a complicated function with more peaks and troughs than the available number of examples. contrast this with a distributed representation, ﬁgure. 15. 7 549


chapter 15. representation learning have a statistical advantage when an apparently complicated structure can be compactly represented using a small number of parameters. some traditional non - distributed learning algorithms generalize only due to the smoothness assumption, which states that if u v ≈, then the target function f to be learned has the property that f ( u ) ≈f ( v ), in general. there are many ways of formalizing such an assumption, but the end result is that if we have an example ( x, y ) for which we know that f ( x ) ≈y, then we choose an estimator [UNK] that approximately satisﬁes these constraints while changing as little as possible when we move to a nearby input x +. this assumption is clearly very useful, but it [UNK] from the curse of dimensionality : in order to learn a target function that increases and decreases many times in many [UNK] regions, 1 we may need a number of examples that is at least as large as the number of distinguishable regions. one can think of each of these regions as a category or symbol : by having a separate degree of freedom for each symbol ( or region ), we can learn an arbitrary decoder mapping from symbol to value. however, this does not allow us to generalize


think of each of these regions as a category or symbol : by having a separate degree of freedom for each symbol ( or region ), we can learn an arbitrary decoder mapping from symbol to value. however, this does not allow us to generalize to new symbols for new regions. if we are lucky, there may be some regularity in the target function, besides being smooth. for example, a convolutional network with max - pooling can recognize an object regardless of its location in the image, even though spatial translation of the object may not correspond to smooth transformations in the input space. let us examine a special case of a distributed representation learning algorithm, that extracts binary features by thresholding linear functions of the input. each binary feature in this representation divides rd into a pair of half - spaces, as illustrated in ﬁgure. the exponentially large number of intersections of 15. 7 n of the corresponding half - spaces determines how many regions this distributed representation learner can distinguish. how many regions are generated by an arrangement of n hyperplanes in rd? by applying a general result concerning the intersection of hyperplanes (, ), one can show ( zaslavsky 1975 pascanu 2014b et al., ) that the number of regions this


by an arrangement of n hyperplanes in rd? by applying a general result concerning the intersection of hyperplanes (, ), one can show ( zaslavsky 1975 pascanu 2014b et al., ) that the number of regions this binary feature representation can distinguish is d j = 0 n j = ( o nd ). ( 15. 4 ) therefore, we see a growth that is exponential in the input size and polynomial in the number of hidden units. 1potentially, we may want to learn a function whose behavior is distinct in exponentially many regions : in a d - dimensional space with at least 2 [UNK] values to distinguish per dimension, we might want to [UNK] in f 2d [UNK] regions, requiring o ( 2 d ) training examples. 550


chapter 15. representation learning this provides a geometric argument to explain the generalization power of distributed representation : with o ( nd ) parameters ( for n linear - threshold features in rd ) we can distinctly represent o ( nd ) regions in input space. if instead we made no assumption at all about the data, and used a representation with one unique symbol for each region, and separate parameters for each symbol to recognize its corresponding portion of rd, then specifying o ( nd ) regions would require o ( nd ) examples. more generally, the argument in favor of the distributed representation could be extended to the case where instead of using linear threshold units we use nonlinear, possibly continuous, feature extractors for each of the attributes in the distributed representation. the argument in this case is that if a parametric transformation with k parameters can learn about r regions in input space, with k r, and if obtaining such a representation was useful to the task of interest, then we could potentially generalize much better in this way than in a non - distributed setting where we would need o ( r ) examples to obtain the same features and associated partitioning of the input space into r regions. using fewer parameters to represent the model means that we have fewer parameters to ﬁt, and thus


a non - distributed setting where we would need o ( r ) examples to obtain the same features and associated partitioning of the input space into r regions. using fewer parameters to represent the model means that we have fewer parameters to ﬁt, and thus require far fewer training examples to generalize well. a further part of the argument for why models based on distributed represen - tations generalize well is that their capacity remains limited despite being able to distinctly encode so many [UNK] regions. for example, the vc dimension of a neural network of linear threshold units is only o ( w w log ), where w is the number of weights ( sontag 1998, ). this limitation arises because, while we can assign very many unique codes to representation space, we cannot use absolutely all of the code space, nor can we learn arbitrary functions mapping from the representation space h to the output y using a linear classiﬁer. the use of a distributed representation combined with a linear classiﬁer thus expresses a prior belief that the classes to be recognized are linearly separable as a function of the underlying causal factors captured by h. we will typically want to learn categories such as the set of all images of all green objects or the set of all images of cars,


that the classes to be recognized are linearly separable as a function of the underlying causal factors captured by h. we will typically want to learn categories such as the set of all images of all green objects or the set of all images of cars, but not categories that require nonlinear, xor logic. for example, we typically do not want to partition the data into the set of all red cars and green trucks as one class and the set of all green cars and red trucks as another class. the ideas discussed so far have been abstract, but they may be experimentally validated. ( ) ﬁnd that hidden units in a deep convolutional network zhou et al. 2015 trained on the imagenet and places benchmark datasets learn features that are very often interpretable, corresponding to a label that humans would naturally assign. in practice it is certainly not always the case that hidden units learn something that has a simple linguistic name, but it is interesting to see this emerge near the top levels of the best computer vision deep networks. what such features have in 551


chapter 15. representation learning - + = figure 15. 9 : a generative model has learned a distributed representation that disentangles the concept of gender from the concept of wearing glasses. if we begin with the repre - sentation of the concept of a man with glasses, then subtract the vector representing the concept of a man without glasses, and ﬁnally add the vector representing the concept of a woman without glasses, we obtain the vector representing the concept of a woman with glasses. the generative model correctly decodes all of these representation vectors to images that may be recognized as belonging to the correct class. images reproduced with permission from ( ). radford et al. 2015 common is that one could imagine learning about each of them without having to see all the conﬁgurations of all the others. ( ) demonstrated that radford et al. 2015 a generative model can learn a representation of images of faces, with separate directions in representation space capturing [UNK] underlying factors of variation. figure demonstrates that one direction in representation space corresponds 15. 9 to whether the person is male or female, while another corresponds to whether the person is wearing glasses. these features were discovered automatically, not ﬁxed a priori. there is no need to have labels for


one direction in representation space corresponds 15. 9 to whether the person is male or female, while another corresponds to whether the person is wearing glasses. these features were discovered automatically, not ﬁxed a priori. there is no need to have labels for the hidden unit classiﬁers : gradient descent on an objective function of interest naturally learns semantically interesting features, so long as the task requires such features. we can learn about the distinction between male and female, or about the presence or absence of glasses, without having to characterize all of the conﬁgurations of the n −1 other features by examples covering all of these combinations of values. this form of statistical separability is what allows one to generalize to new conﬁgurations of a person ’ s features that have never been seen during training. 552


chapter 15. representation learning 15. 5 exponential gains from depth we have seen in section that multilayer perceptrons are universal approxima - 6. 4. 1 tors, and that some functions can be represented by exponentially smaller deep networks compared to shallow networks. this decrease in model size leads to improved statistical [UNK]. in this section, we describe how similar results apply more generally to other kinds of models with distributed hidden representations. in section, we saw an example of a generative model that learned about 15. 4 the explanatory factors underlying images of faces, including the person ’ s gender and whether they are wearing glasses. the generative model that accomplished this task was based on a deep neural network. it would not be reasonable to expect a shallow network, such as a linear network, to learn the complicated relationship between these abstract explanatory factors and the pixels in the image. in this and other ai tasks, the factors that can be chosen almost independently from each other yet still correspond to meaningful inputs are more likely to be very high - level and related in highly nonlinear ways to the input. we argue that this demands deep distributed representations, where the higher level features ( seen as functions of the input ) or factors ( seen as generative causes


inputs are more likely to be very high - level and related in highly nonlinear ways to the input. we argue that this demands deep distributed representations, where the higher level features ( seen as functions of the input ) or factors ( seen as generative causes ) are obtained through the composition of many nonlinearities. it has been proven in many [UNK] settings that organizing computation through the composition of many nonlinearities and a hierarchy of reused features can give an exponential boost to statistical [UNK], on top of the exponential boost given by using a distributed representation. many kinds of networks ( e. g., with saturating nonlinearities, boolean gates, sum / products, or rbf units ) with a single hidden layer can be shown to be universal approximators. a model family that is a universal approximator can approximate a large class of functions ( including all continuous functions ) up to any non - zero tolerance level, given enough hidden units. however, the required number of hidden units may be very large. theoretical results concerning the expressive power of deep architectures state that there are families of functions that can be represented [UNK] by an architecture of depth k, but would require an exponential number of hidden units ( with respect to the input size ) with [UNK] depth ( depth 2 or depth


expressive power of deep architectures state that there are families of functions that can be represented [UNK] by an architecture of depth k, but would require an exponential number of hidden units ( with respect to the input size ) with [UNK] depth ( depth 2 or depth ). k −1 in section, we saw that deterministic feedforward networks are universal 6. 4. 1 approximators of functions. many structured probabilistic models with a single hidden layer of latent variables, including restricted boltzmann machines and deep belief networks, are universal approximators of probability distributions ( le roux and bengio 2008 2010 montufar and ay 2011 montufar 2014 krause,, ;, ;, ; et al., 2013 ). 553


chapter 15. representation learning in section, we saw that a [UNK] deep feedforward network can have 6. 4. 1 an exponential advantage over a network that is too shallow. such results can also be obtained for other models such as probabilistic models. one such probabilistic model is the sum - product network or spn ( poon and domingos 2011, ). these models use polynomial circuits to compute the probability distribution over a set of random variables. ( ) showed that there exist delalleau and bengio 2011 probability distributions for which a minimum depth of spn is required to avoid needing an exponentially large model. later, ( ) martens and medabalimi 2014 showed that there are signiﬁcant [UNK] between every two ﬁnite depths of spn, and that some of the constraints used to make spns tractable may limit their representational power. another interesting development is a set of theoretical results for the expressive power of families of deep circuits related to convolutional nets, highlighting an exponential advantage for the deep circuit even when the shallow circuit is allowed to only approximate the function computed by the deep circuit (, cohen et al. 2015 ). by comparison, previous theoretical work made claims regarding only the case


##tional nets, highlighting an exponential advantage for the deep circuit even when the shallow circuit is allowed to only approximate the function computed by the deep circuit (, cohen et al. 2015 ). by comparison, previous theoretical work made claims regarding only the case where the shallow circuit must exactly replicate particular functions. 15. 6 providing clues to discover underlying causes to close this chapter, we come back to one of our original questions : what makes one representation better than another? one answer, ﬁrst introduced in section, is 15. 3 that an ideal representation is one that disentangles the underlying causal factors of variation that generated the data, especially those factors that are relevant to our applications. most strategies for representation learning are based on introducing clues that help the learning to ﬁnd these underlying factors of variations. the clues can help the learner separate these observed factors from the others. supervised learning provides a very strong clue : a label y, presented with each x, that usually speciﬁes the value of at least one of the factors of variation directly. more generally, to make use of abundant unlabeled data, representation learning makes use of other, less direct, hints about the underlying factors. these hints take the form of implicit prior beliefs that we, the designers of


factors of variation directly. more generally, to make use of abundant unlabeled data, representation learning makes use of other, less direct, hints about the underlying factors. these hints take the form of implicit prior beliefs that we, the designers of the learning algorithm, impose in order to guide the learner. results such as the no free lunch theorem show that regularization strategies are necessary to obtain good generalization. while it is impossible to ﬁnd a universally superior regularization strategy, one goal of deep learning is to ﬁnd a set of fairly generic regularization strategies that are applicable to a wide variety of ai tasks, similar to the tasks that people and animals are able to solve. 554


chapter 15. representation learning we provide here a list of these generic regularization strategies. the list is clearly not exhaustive, but gives some concrete examples of ways that learning algorithms can be encouraged to discover features that correspond to underlying factors. this list was introduced in section 3. 1 of ( ) and has bengio et al. 2013d been partially expanded here. • smoothness : this is the assumption that f ( x + d ) ≈f ( x ) for unit d and small. this assumption allows the learner to generalize from training examples to nearby points in input space. many machine learning algorithms leverage this idea, but it is [UNK] to overcome the curse of dimensionality. • linearity : many learning algorithms assume that relationships between some variables are linear. this allows the algorithm to make predictions even very far from the observed data, but can sometimes lead to overly extreme predictions. most simple machine learning algorithms that do not make the smoothness assumption instead make the linearity assumption. these are in fact [UNK] assumptions — linear functions with large weights applied to high - dimensional spaces may not be very smooth. see goodfellow et al. ( ) for a further discussion of the limitations of the linearity assumption. 2014b • multiple explanatory factors : many representation


functions with large weights applied to high - dimensional spaces may not be very smooth. see goodfellow et al. ( ) for a further discussion of the limitations of the linearity assumption. 2014b • multiple explanatory factors : many representation learning algorithms are motivated by the assumption that the data is generated by multiple underlying explanatory factors, and that most tasks can be solved easily given the state of each of these factors. section describes how this view motivates semi - 15. 3 supervised learning via representation learning. learning the structure of p ( x ) requires learning some of the same features that are useful for modeling p ( y | x ) because both refer to the same underlying explanatory factors. section 15. 4 describes how this view motivates the use of distributed representations, with separate directions in representation space corresponding to separate factors of variation. • causal factors : the model is constructed in such a way that it treats the factors of variation described by the learned representation h as the causes of the observed data x, and not vice - versa. as discussed in section, this 15. 3 is advantageous for semi - supervised learning and makes the learned model more robust when the distribution over the underlying causes changes or when we use the model for a


observed data x, and not vice - versa. as discussed in section, this 15. 3 is advantageous for semi - supervised learning and makes the learned model more robust when the distribution over the underlying causes changes or when we use the model for a new task. • depth a hierarchical organization of explanatory factors, or : high - level, abstract concepts can be deﬁned in terms of simple concepts, forming a hierarchy. from another point of view, the use of a deep architecture 555


chapter 15. representation learning expresses our belief that the task should be accomplished via a multi - step program, with each step referring back to the output of the processing accomplished via previous steps. • shared factors across tasks : in the context where we have many tasks, corresponding to [UNK] yi variables sharing the same input x or where each task is associated with a subset or a function f ( ) i ( x ) of a global input x, the assumption is that each yi is associated with a [UNK] subset from a common pool of relevant factors h. because these subsets overlap, learning all the p ( yi | x ) via a shared intermediate representation p ( h x | ) allows sharing of statistical strength between the tasks. • manifolds : probability mass concentrates, and the regions in which it con - centrates are locally connected and occupy a tiny volume. in the continuous case, these regions can be approximated by low - dimensional manifolds with a much smaller dimensionality than the original space where the data lives. many machine learning algorithms behave sensibly only on this manifold (, ). some machine learning algorithms, especially goodfellow et al. 2014b autoencoders, attempt to explicitly learn the structure of the manifold. • natural clustering : many machine learning


algorithms behave sensibly only on this manifold (, ). some machine learning algorithms, especially goodfellow et al. 2014b autoencoders, attempt to explicitly learn the structure of the manifold. • natural clustering : many machine learning algorithms assume that each connected manifold in the input space may be assigned to a single class. the data may lie on many disconnected manifolds, but the class remains constant within each one of these. this assumption motivates a variety of learning algorithms, including tangent propagation, double backprop, the manifold tangent classiﬁer and adversarial training. • temporal and spatial coherence : slow feature analysis and related algorithms make the assumption that the most important explanatory factors change slowly over time, or at least that it is easier to predict the true underlying explanatory factors than to predict raw observations such as pixel values. see section for further description of this approach. 13. 3 • sparsity : most features should presumably not be relevant to describing most inputs — there is no need to use a feature that detects elephant trunks when representing an image of a cat. it is therefore reasonable to impose a prior that any feature that can be interpreted as “ present ” or “ absent ” should be


describing most inputs — there is no need to use a feature that detects elephant trunks when representing an image of a cat. it is therefore reasonable to impose a prior that any feature that can be interpreted as “ present ” or “ absent ” should be absent most of the time. • simplicity of factor dependencies : in good high - level representations, the factors are related to each other through simple dependencies. the simplest 556


chapter 15. representation learning possible is marginal independence, p ( h ) = ip ( hi ), but linear dependencies or those captured by a shallow autoencoder are also reasonable assumptions. this can be seen in many laws of physics, and is assumed when plugging a linear predictor or a factorized prior on top of a learned representation. the concept of representation learning ties together all of the many forms of deep learning. feedforward and recurrent networks, autoencoders and deep probabilistic models all learn and exploit representations. learning the best possible representation remains an exciting avenue of research. 557


chapter 16 structured probabilistic models for deep learning deep learning draws upon many modeling formalisms that researchers can use to guide their design [UNK] and describe their algorithms. one of these formalisms is the idea of structured probabilistic models. we have already discussed structured probabilistic models brieﬂy in section. that brief presentation was 3. 14 [UNK] to understand how to use structured probabilistic models as a language to describe some of the algorithms in part. now, in part, structured probabilistic ii iii models are a key ingredient of many of the most important research topics in deep learning. in order to prepare to discuss these research ideas, this chapter describes structured probabilistic models in much greater detail. this chapter is intended to be self - contained ; the reader does not need to review the earlier introduction before continuing with this chapter. a structured probabilistic model is a way of describing a probability distribution, using a graph to describe which random variables in the probability distribution interact with each other directly. here we use “ graph ” in the graph theory sense — a set of vertices connected to one another by a set of edges. because the structure of the model is deﬁned by a graph, these models are often also referred


each other directly. here we use “ graph ” in the graph theory sense — a set of vertices connected to one another by a set of edges. because the structure of the model is deﬁned by a graph, these models are often also referred to as graphical models. the graphical models research community is large and has developed many [UNK] models, training algorithms, and inference algorithms. in this chapter, we provide basic background on some of the most central ideas of graphical models, with an emphasis on the concepts that have proven most useful to the deep learning research community. if you already have a strong background in graphical models, you may wish to skip most of this chapter. however, even a graphical model expert 558


chapter 16. structured probabilistic models for deep learning may beneﬁt from reading the ﬁnal section of this chapter, section, in which we 16. 7 highlight some of the unique ways that graphical models are used for deep learning algorithms. deep learning practitioners tend to use very [UNK] model structures, learning algorithms and inference procedures than are commonly used by the rest of the graphical models research community. in this chapter, we identify these [UNK] in preferences and explain the reasons for them. in this chapter we ﬁrst describe the challenges of building large - scale proba - bilistic models. next, we describe how to use a graph to describe the structure of a probability distribution. while this approach allows us to overcome many challenges, it is not without its own complications. one of the major [UNK] in graphical modeling is understanding which variables need to be able to interact directly, i. e., which graph structures are most suitable for a given problem. we outline two approaches to resolving this [UNK] by learning about the dependen - cies in section. finally, we close with a discussion of the unique emphasis that 16. 5 deep learning practitioners place on speciﬁc approaches to graphical modeling in section. 16. 7 16. 1 the challenge of unstructured modeling the


cies in section. finally, we close with a discussion of the unique emphasis that 16. 5 deep learning practitioners place on speciﬁc approaches to graphical modeling in section. 16. 7 16. 1 the challenge of unstructured modeling the goal of deep learning is to scale machine learning to the kinds of challenges needed to solve artiﬁcial intelligence. this means being able to understand high - dimensional data with rich structure. for example, we would like ai algorithms to be able to understand natural images, 1 audio waveforms representing speech, and documents containing multiple words and punctuation characters. classiﬁcation algorithms can take an input from such a rich high - dimensional distribution and summarize it with a categorical label — what object is in a photo, what word is spoken in a recording, what topic a document is about. the process of classiﬁcation discards most of the information in the input and produces a single output ( or a probability distribution over values of that single output ). the classiﬁer is also often able to ignore many parts of the input. for example, when recognizing an object in a photo, it is usually possible to ignore the background of the photo. it is possible to ask probabilistic models to


classiﬁer is also often able to ignore many parts of the input. for example, when recognizing an object in a photo, it is usually possible to ignore the background of the photo. it is possible to ask probabilistic models to do many other tasks. these tasks are often more expensive than classiﬁcation. some of them require producing multiple output values. most require a complete understanding of the entire structure of 1 a natural image is an image that might be captured by a camera in a reasonably ordinary environment, as opposed to a synthetically rendered image, a screenshot of a web page, etc. 559