( ) and ( ) for examples of such an 2014b lin et al. 2015 garcia - duran et al. 2015 application. evaluating the performance of a model on a link prediction task is [UNK] because we have only a dataset of positive examples ( facts that are known to be true ). if the model proposes a fact that is not in the dataset, we are unsure whether the model has made a mistake or discovered a new, previously unknown fact. the metrics are thus somewhat imprecise and are based on testing how the model ranks a held - out of set of known true positive facts compared to other facts that are less likely to be true. a common way to construct interesting examples that are probably negative ( facts that are probably false ) is to begin with a true fact and create corrupted versions of that fact, for example by replacing one entity in the relation with a [UNK] entity selected at random. the popular precision at 10 % metric counts how many times the model ranks a “ correct ” fact among the top 10 % of all corrupted versions of that fact. 484


chapter 12. applications another application of knowledge bases and distributed representations for them is word - sense disambiguation ( navigli and velardi 2005 bordes, ; et al., 2012 ), which is the task of deciding which of the senses of a word is the appropriate one, in some context. eventually, knowledge of relations combined with a reasoning process and understanding of natural language could allow us to build a general question answering system. a general question answering system must be able to process input information and remember important facts, organized in a way that enables it to retrieve and reason about them later. this remains a [UNK] open problem which can only be solved in restricted “ toy ” environments. currently, the best approach to remembering and retrieving speciﬁc declarative facts is to use an explicit memory mechanism, as described in section. memory networks were 10. 12 ﬁrst proposed to solve a toy question answering task ( weston 2014 kumar et al., ). et al. ( ) have proposed an extension that uses gru recurrent nets to read 2015 the input into the memory and to produce the answer given the contents of the memory. deep learning has been applied to many other applications besides the ones described here, and will surely be applied to

/home/ricoiban/GEMMA/mnlp_chatsplaining/RAG/Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville (z-lib.org).pdf


an extension that uses gru recurrent nets to read 2015 the input into the memory and to produce the answer given the contents of the memory. deep learning has been applied to many other applications besides the ones described here, and will surely be applied to even more after this writing. it would be impossible to describe anything remotely resembling a comprehensive coverage of such a topic. this survey provides a representative sample of what is possible as of this writing. this concludes part, which has described modern practices involving deep ii networks, comprising all of the most successful methods. generally speaking, these methods involve using the gradient of a cost function to ﬁnd the parameters of a model that approximates some desired function. with enough training data, this approach is extremely powerful. we now turn to part, in which we step into the iii territory of research — methods that are designed to work with less training data or to perform a greater variety of tasks, where the challenges are more [UNK] and not as close to being solved as the situations we have described so far. 485


part iii deep learning research 486


this part of the book describes the more ambitious and advanced approaches to deep learning, currently pursued by the research community. in the previous parts of the book, we have shown how to solve supervised learning problems — how to learn to map one vector to another, given enough examples of the mapping. not all problems we might want to solve fall into this category. we may wish to generate new examples, or determine how likely some point is, or handle missing values and take advantage of a large set of unlabeled examples or examples from related tasks. a shortcoming of the current state of the art for industrial applications is that our learning algorithms require large amounts of supervised data to achieve good accuracy. in this part of the book, we discuss some of the speculative approaches to reducing the amount of labeled data necessary for existing models to work well and be applicable across a broader range of tasks. accomplishing these goals usually requires some form of unsupervised or semi - supervised learning. many deep learning algorithms have been designed to tackle unsupervised learning problems, but none have truly solved the problem in the same way that deep learning has largely solved the supervised learning problem for a wide variety of tasks. in this part of the book, we describe the existing approaches to un


unsupervised learning problems, but none have truly solved the problem in the same way that deep learning has largely solved the supervised learning problem for a wide variety of tasks. in this part of the book, we describe the existing approaches to unsupervised learning and some of the popular thought about how we can make progress in this ﬁeld. a central cause of the [UNK] with unsupervised learning is the high di - mensionality of the random variables being modeled. this brings two distinct challenges : a statistical challenge and a computational challenge. the statistical challenge regards generalization : the number of conﬁgurations we may want to distinguish can grow exponentially with the number of dimensions of interest, and this quickly becomes much larger than the number of examples one can possibly have ( or use with bounded computational resources ). the computational challenge associated with high - dimensional distributions arises because many algorithms for learning or using a trained model ( especially those based on estimating an explicit probability function ) involve intractable computations that grow exponentially with the number of dimensions. with probabilistic models, this computational challenge arises from the need to perform intractable inference or simply from the need to normalize the distribution. • intractable inference : inference is discussed mostly in


that grow exponentially with the number of dimensions. with probabilistic models, this computational challenge arises from the need to perform intractable inference or simply from the need to normalize the distribution. • intractable inference : inference is discussed mostly in chapter. it regards 19 the question of guessing the probable values of some variables a, given other variables b, with respect to a model that captures the joint distribution over 487


a, b and c. in order to even compute such conditional probabilities one needs to sum over the values of the variables c, as well as compute a normalization constant which sums over the values of a and c. • intractable normalization constants ( the partition function ) : the partition function is discussed mostly in chapter. normalizing constants of proba - 18 bility functions come up in inference ( above ) as well as in learning. many probabilistic models involve such a normalizing constant. unfortunately, learning such a model often requires computing the gradient of the loga - rithm of the partition function with respect to the model parameters. that computation is generally as intractable as computing the partition function itself. monte carlo markov chain ( mcmc ) methods ( chapter ) are of - 17 ten used to deal with the partition function ( computing it or its gradient ). unfortunately, mcmc methods [UNK] when the modes of the model distribu - tion are numerous and well - separated, especially in high - dimensional spaces ( section ). 17. 5 one way to confront these intractable computations is to approximate them, and many approaches have been proposed as discussed in this third part of the book. another interesting way,


- separated, especially in high - dimensional spaces ( section ). 17. 5 one way to confront these intractable computations is to approximate them, and many approaches have been proposed as discussed in this third part of the book. another interesting way, also discussed here, would be to avoid these intractable computations altogether by design, and methods that do not require such computations are thus very appealing. several generative models have been proposed in recent years, with that motivation. a wide variety of contemporary approaches to generative modeling are discussed in chapter. 20 part is the most important for a researcher — someone who wants to un - iii derstand the breadth of perspectives that have been brought to the ﬁeld of deep learning, and push the ﬁeld forward towards true artiﬁcial intelligence. 488


chapter 13 linear factor models many of the research frontiers in deep learning involve building a probabilistic model of the input, pmodel ( x ). such a model can, in principle, use probabilistic inference to predict any of the variables in its environment given any of the other variables. many of these models also have latent variables h, with pmodel ( ) = x ehpmodel ( ) x h |. these latent variables provide another means of representing the data. distributed representations based on latent variables can obtain all of the advantages of representation learning that we have seen with deep feedforward and recurrent networks. in this chapter, we describe some of the simplest probabilistic models with latent variables : linear factor models. these models are sometimes used as building blocks of mixture models ( hinton 1995a ghahramani and hinton 1996 et al., ;, ; roweis 2002 tang 2012 et al., ) or larger, deep probabilistic models ( et al., ). they also show many of the basic approaches necessary to build generative models that the more advanced deep models will extend further. a linear factor model is deﬁned by the use of a stochastic, linear


models ( et al., ). they also show many of the basic approaches necessary to build generative models that the more advanced deep models will extend further. a linear factor model is deﬁned by the use of a stochastic, linear decoder function that generates by adding noise to a linear transformation of. x h these models are interesting because they allow us to discover explanatory factors that have a simple joint distribution. the simplicity of using a linear decoder made these models some of the ﬁrst latent variable models to be extensively studied. a linear factor model describes the data generation process as follows. first, we sample the explanatory factors from a distribution h h [UNK], ( ) h ( 13. 1 ) where p ( h ) is a factorial distribution, with p ( h ) = i p ( hi ), so that it is easy to 489


chapter 13. linear factor models sample from. next we sample the real - valued observable variables given the factors : x w h b = + + noise ( 13. 2 ) where the noise is typically gaussian and diagonal ( independent across dimensions ). this is illustrated in ﬁgure. 13. 1 h1 h1 h2 h2 h3 h3 x1 x1 x2 x2 x3 x3 x h n ois e x h n ois e = w + + b = w + + b figure 13. 1 : the directed graphical model describing the linear factor model family, in which we assume that an observed data vector x is obtained by a linear combination of independent latent factors h, plus some noise. [UNK] models, such as probabilistic pca, factor analysis or ica, make [UNK] choices about the form of the noise and of the prior. p ( ) h 13. 1 probabilistic pca and factor analysis probabilistic pca ( principal components analysis ), factor analysis and other linear factor models are special cases of the above equations ( and ) and only 13. 1 13. 2 [UNK] in the choices made for the noise distribution and the model ’ s prior over latent


##a ( principal components analysis ), factor analysis and other linear factor models are special cases of the above equations ( and ) and only 13. 1 13. 2 [UNK] in the choices made for the noise distribution and the model ’ s prior over latent variables before observing. h x in factor analysis (, ;, ), the latent variable bartholomew 1987 basilevsky 1994 prior is just the unit variance gaussian h 0 [UNK] ( ; h, i ) ( 13. 3 ) while the observed variables xi are assumed to be conditionally independent, given h. speciﬁcally, the noise is assumed to be drawn from a diagonal co - variance gaussian distribution, with covariance matrix ψ = diag ( σ2 ), with σ2 = [ σ2 1, σ2 2,..., σ2 n ] a vector of per - variable variances. the role of the latent variables is thus to capture the dependencies between the [UNK] observed variables xi. indeed, it can easily be shown that x is just a multivariate normal random variable, with x [UNK] ( ; x b ww, + ) ψ. ( 13. 4 ) 490


chapter 13. linear factor models in order to cast pca in a probabilistic framework, we can make a slight modiﬁcation to the factor analysis model, making the conditional variances σ2 i equal to each other. in that case the covariance of x is just ww + σ2i, where σ2 is now a scalar. this yields the conditional distribution x [UNK] ( ; x b ww, + σ2i ) ( 13. 5 ) or equivalently x h z = w + + b σ ( 13. 6 ) where z [UNK] ( z ; 0, i ) is gaussian noise. ( ) then show an tipping and bishop 1999 iterative em algorithm for estimating the parameters and w σ2. this probabilistic pca model takes advantage of the observation that most variations in the data can be captured by the latent variables h, up to some small residual reconstruction error σ2. as shown by ( ), tipping and bishop 1999 probabilistic pca becomes pca as σ →0. in that case, the conditional expected value of h given x becomes an orthogonal projection of x b − onto the space spanned by the columns of, like in pca. d w as σ


##bilistic pca becomes pca as σ →0. in that case, the conditional expected value of h given x becomes an orthogonal projection of x b − onto the space spanned by the columns of, like in pca. d w as σ →0, the density model deﬁned by probabilistic pca becomes very sharp around these d dimensions spanned by the columns of w. this can make the model assign very low likelihood to the data if the data does not actually cluster near a hyperplane. 13. 2 independent component analysis ( ica ) independent component analysis ( ica ) is among the oldest representation learning algorithms (, ;, ;, ; herault and ans 1984 jutten and herault 1991 comon 1994 hyvarinen 1999 hyvarinen 2001a hinton 2001 teh 2003, ; et al., ; et al., ; et al., ). it is an approach to modeling linear factors that seeks to separate an observed signal into many underlying signals that are scaled and added together to form the observed data. these signals are intended to be fully independent, rather than merely decorrelated from each other. 1 many [UNK] speciﬁc methodologies are referred to as ica. the variant that is most


scaled and added together to form the observed data. these signals are intended to be fully independent, rather than merely decorrelated from each other. 1 many [UNK] speciﬁc methodologies are referred to as ica. the variant that is most similar to the other generative models we have described here is a variant (, ) that trains a fully parametric generative model. the pham et al. 1992 prior distribution over the underlying factors, p ( h ), must be ﬁxed ahead of time by the user. the model then deterministically generates x = wh. we can perform a 1see section for a discussion of the [UNK] between uncorrelated variables and indepen - 3. 8 dent variables. 491


chapter 13. linear factor models nonlinear change of variables ( using equation ) to determine 3. 47 p ( x ). learning the model then proceeds as usual, using maximum likelihood. the motivation for this approach is that by choosing p ( h ) to be independent, we can recover underlying factors that are as close as possible to independent. this is commonly used, not to capture high - level abstract causal factors, but to recover low - level signals that have been mixed together. in this setting, each training example is one moment in time, each xi is one sensor ’ s observation of the mixed signals, and each hi is one estimate of one of the original signals. for example, we might have n people speaking simultaneously. if we have n [UNK] microphones placed in [UNK] locations, ica can detect the changes in the volume between each speaker as heard by each microphone, and separate the signals so that each h i contains only one person speaking clearly. this is commonly used in neuroscience for electroencephalography, a technology for recording electrical signals originating in the brain. many electrode sensors placed on the subject ’ s head are used to measure many electrical signals coming from the body. the experimenter is typically only interested in signals from the brain, but signals from the subject ’


for recording electrical signals originating in the brain. many electrode sensors placed on the subject ’ s head are used to measure many electrical signals coming from the body. the experimenter is typically only interested in signals from the brain, but signals from the subject ’ s heart and eyes are strong enough to confound measurements taken at the subject ’ s scalp. the signals arrive at the electrodes mixed together, so ica is necessary to separate the electrical signature of the heart from the signals originating in the brain, and to separate signals in [UNK] brain regions from each other. as mentioned before, many variants of ica are possible. some add some noise in the generation of x rather than using a deterministic decoder. most do not use the maximum likelihood criterion, but instead aim to make the elements of h = w −1x independent from each other. many criteria that accomplish this goal are possible. equation requires taking the determinant of 3. 47 w, which can be an expensive and numerically unstable operation. some variants of ica avoid this problematic operation by constraining to be orthogonal. w all variants of ica require that p ( h ) be non - gaussian. this is because if p ( h ) is an independent prior with gauss


variants of ica avoid this problematic operation by constraining to be orthogonal. w all variants of ica require that p ( h ) be non - gaussian. this is because if p ( h ) is an independent prior with gaussian components, then w is not identiﬁable. we can obtain the same distribution over p ( x ) for many values of w. this is very [UNK] from other linear factor models like probabilistic pca and factor analysis, that often require p ( h ) to be gaussian in order to make many operations on the model have closed form solutions. in the maximum likelihood approach where the user explicitly speciﬁes the distribution, a typical choice is to use p ( hi ) = d dhiσ ( hi ). typical choices of these non - gaussian distributions have larger peaks near 0 than does the gaussian distribution, so we can also see most implementations of ica as learning sparse features. 492


chapter 13. linear factor models many variants of ica are not generative models in the sense that we use the phrase. in this book, a generative model either represents p ( x ) or can draw samples from it. many variants of ica only know how to transform between x and h, but do not have any way of representing p ( h ), and thus do not impose a distribution over p ( x ). for example, many ica variants aim to increase the sample kurtosis of h = w −1x, because high kurtosis indicates that p ( h ) is non - gaussian, but this is accomplished without explicitly representing p ( h ). this is because ica is more often used as an analysis tool for separating signals, rather than for generating data or estimating its density. just as pca can be generalized to the nonlinear autoencoders described in chapter, ica can be generalized to a nonlinear generative model, in which 14 we use a nonlinear function f to generate the observed data. see hyvarinen and pajunen 1999 ( ) for the initial work on nonlinear ica and its successful use with ensemble learning by ( ) and ( ). roberts and everson 2001 lappalainen et al.


generate the observed data. see hyvarinen and pajunen 1999 ( ) for the initial work on nonlinear ica and its successful use with ensemble learning by ( ) and ( ). roberts and everson 2001 lappalainen et al. 2000 another nonlinear extension of ica is the approach of nonlinear independent components estimation, or nice (, ), which stacks a series dinh et al. 2014 of invertible transformations ( encoder stages ) that have the property that the determinant of the jacobian of each transformation can be computed [UNK]. this makes it possible to compute the likelihood exactly and, like ica, attempts to transform the data into a space where it has a factorized marginal distribution, but is more likely to succeed thanks to the nonlinear encoder. because the encoder is associated with a decoder that is its perfect inverse, it is straightforward to generate samples from the model ( by ﬁrst sampling from p ( h ) and then applying the decoder ). another generalization of ica is to learn groups of features, with statistical dependence allowed within a group but discouraged between groups ( hyvarinen and hoyer 1999 hyvarinen 2001b, ; et al., ). when the groups of related


##ization of ica is to learn groups of features, with statistical dependence allowed within a group but discouraged between groups ( hyvarinen and hoyer 1999 hyvarinen 2001b, ; et al., ). when the groups of related units are chosen to be non - overlapping, this is called independent subspace analysis. it is also possible to assign spatial coordinates to each hidden unit and form overlapping groups of spatially neighboring units. this encourages nearby units to learn similar features. when applied to natural images, this topographic ica approach learns gabor ﬁlters, such that neighboring features have similar orientation, location or frequency. many [UNK] phase [UNK] of similar gabor functions occur within each region, so that pooling over small regions yields translation invariance. 13. 3 slow feature analysis slow feature analysis ( sfa ) is a linear factor model that uses information from 493


chapter 13. linear factor models time signals to learn invariant features (, ). wiskott and sejnowski 2002 slow feature analysis is motivated by a general principle called the slowness principle. the idea is that the important characteristics of scenes change very slowly compared to the individual measurements that make up a description of a scene. for example, in computer vision, individual pixel values can change very rapidly. if a zebra moves from left to right across the image, an individual pixel will rapidly change from black to white and back again as the zebra ’ s stripes pass over the pixel. by comparison, the feature indicating whether a zebra is in the image will not change at all, and the feature describing the zebra ’ s position will change slowly. we therefore may wish to regularize our model to learn features that change slowly over time. the slowness principle predates slow feature analysis and has been applied to a wide variety of models (, ;, ;, ; hinton 1989 foldiak 1989 mobahi et al. 2009 bergstra and bengio 2009, ). in general, we can apply the slowness principle to any [UNK] model trained with gradient descent. the slowness principle may be introduced by adding a term to the cost function of the form λ t l f


bergstra and bengio 2009, ). in general, we can apply the slowness principle to any [UNK] model trained with gradient descent. the slowness principle may be introduced by adding a term to the cost function of the form λ t l f ( ( x ( + 1 ) t ) (, f x ( ) t ) ) ( 13. 7 ) where λ is a hyperparameter determining the strength of the slowness regularization term, t is the index into a time sequence of examples, f is the feature extractor to be regularized, and l is a loss function measuring the distance between f ( x ( ) t ) and f ( x ( + 1 ) t ). a common choice for is the mean squared [UNK]. l slow feature analysis is a particularly [UNK] application of the slowness principle. it is [UNK] because it is applied to a linear feature extractor, and can thus be trained in closed form. like some variants of ica, sfa is not quite a generative model per se, in the sense that it deﬁnes a linear map between input space and feature space but does not deﬁne a prior over feature space and thus does not impose a distribution on input space. p ( ) x the sfa algorithm (


se, in the sense that it deﬁnes a linear map between input space and feature space but does not deﬁne a prior over feature space and thus does not impose a distribution on input space. p ( ) x the sfa algorithm ( wiskott and sejnowski 2002, ) consists of deﬁning f ( x ; θ ) to be a linear transformation, and solving the optimization problem min θ et ( ( f x ( + 1 ) t ) i −f ( x ( ) t ) i ) 2 ( 13. 8 ) subject to the constraints etf ( x ( ) t ) i = 0 ( 13. 9 ) and et [ ( f x ( ) t ) 2 i ] = 1. ( 13. 10 ) 494


chapter 13. linear factor models the constraint that the learned feature have zero mean is necessary to make the problem have a unique solution ; otherwise we could add a constant to all feature values and obtain a [UNK] solution with equal value of the slowness objective. the constraint that the features have unit variance is necessary to prevent the pathological solution where all features collapse to. like pca, the sfa features 0 are ordered, with the ﬁrst feature being the slowest. to learn multiple features, we must also add the constraint [UNK] < j, et [ ( f x ( ) t ) if ( x ( ) t ) j ] = 0. ( 13. 11 ) this speciﬁes that the learned features must be linearly decorrelated from each other. without this constraint, all of the learned features would simply capture the one slowest signal. one could imagine using other mechanisms, such as minimizing reconstruction error, to force the features to diversify, but this decorrelation mechanism admits a simple solution due to the linearity of sfa features. the sfa problem may be solved in closed form by a linear algebra package. sfa is typically used to learn nonlinear features by applying a nonlinear basis expansion to x before running sfa. for example


due to the linearity of sfa features. the sfa problem may be solved in closed form by a linear algebra package. sfa is typically used to learn nonlinear features by applying a nonlinear basis expansion to x before running sfa. for example, it is common to replace x by the quadratic basis expansion, a vector containing elements xixj for all i and j. linear sfa modules may then be composed to learn deep nonlinear slow feature extractors by repeatedly learning a linear sfa feature extractor, applying a nonlinear basis expansion to its output, and then learning another linear sfa feature extractor on top of that expansion. when trained on small spatial patches of videos of natural scenes, sfa with quadratic basis expansions learns features that share many characteristics with those of complex cells in v1 cortex ( berkes and wiskott 2005, ). when trained on videos of random motion within 3 - d computer rendered environments, deep sfa learns features that share many characteristics with the features represented by neurons in rat brains that are used for navigation ( franzius 2007 et al., ). sfa thus seems to be a reasonably biologically plausible model. a major advantage of sfa is that it is possibly to theoretically predict which features sfa will


in rat brains that are used for navigation ( franzius 2007 et al., ). sfa thus seems to be a reasonably biologically plausible model. a major advantage of sfa is that it is possibly to theoretically predict which features sfa will learn, even in the deep, nonlinear setting. to make such theoretical predictions, one must know about the dynamics of the environment in terms of conﬁguration space ( e. g., in the case of random motion in the 3 - d rendered environment, the theoretical analysis proceeds from knowledge of the probability distribution over position and velocity of the camera ). given the knowledge of how the underlying factors actually change, it is possible to analytically solve for the optimal functions expressing these factors. in practice, experiments with deep sfa applied to simulated data seem to recover the theoretically predicted functions. 495


chapter 13. linear factor models this is in comparison to other learning algorithms where the cost function depends highly on speciﬁc pixel values, making it much more [UNK] to determine what features the model will learn. deep sfa has also been used to learn features for object recognition and pose estimation ( franzius 2008 et al., ). so far, the slowness principle has not become the basis for any state of the art applications. it is unclear what factor has limited its performance. we speculate that perhaps the slowness prior is too strong, and that, rather than imposing a prior that features should be approximately constant, it would be better to impose a prior that features should be easy to predict from one time step to the next. the position of an object is a useful feature regardless of whether the object ’ s velocity is high or low, but the slowness principle encourages the model to ignore the position of objects that have high velocity. 13. 4 sparse coding sparse coding (, ) is a linear factor model that has olshausen and field 1996 been heavily studied as an unsupervised feature learning and feature extraction mechanism. strictly speaking, the term “ sparse coding ” refers to the process of inferring the value of h in this model,


has olshausen and field 1996 been heavily studied as an unsupervised feature learning and feature extraction mechanism. strictly speaking, the term “ sparse coding ” refers to the process of inferring the value of h in this model, while “ sparse modeling ” refers to the process of designing and learning the model, but the term “ sparse coding ” is often used to refer to both. like most other linear factor models, it uses a linear decoder plus noise to obtain reconstructions of x, as speciﬁed in equation. more speciﬁcally, sparse 13. 2 coding models typically assume that the linear factors have gaussian noise with isotropic precision : β p, ( ) = ( ; + x h | n x w h b 1 βi ). ( 13. 12 ) the distribution p ( h ) is chosen to be one with sharp peaks near 0 ( olshausen and field 1996, ). common choices include factorized laplace, cauchy or factorized student - t distributions. for example, the laplace prior parametrized in terms of the sparsity penalty [UNK] is given by λ p h ( i ) = laplace ( hi ; 0, 2 λ ) = λ 4


or factorized student - t distributions. for example, the laplace prior parametrized in terms of the sparsity penalty [UNK] is given by λ p h ( i ) = laplace ( hi ; 0, 2 λ ) = λ 4e−1 2 λ h | i | ( 13. 13 ) and the student - prior by t p h ( i ) [UNK] 1 ( 1 + h2 i ν ) ν + 1 2. ( 13. 14 ) 496


chapter 13. linear factor models training sparse coding with maximum likelihood is intractable. instead, the training alternates between encoding the data and training the decoder to better reconstruct the data given the encoding. this approach will be justiﬁed further as a principled approximation to maximum likelihood later, in section. 19. 3 for models such as pca, we have seen the use of a parametric encoder function that predicts h and consists only of multiplication by a weight matrix. the encoder that we use with sparse coding is not a parametric encoder. instead, the encoder is an optimization algorithm, that solves an optimization problem in which we seek the single most likely code value : h∗ = ( ) = arg max f x h p. ( ) h x | ( 13. 15 ) when combined with equation and equation, this yields the following 13. 13 13. 12 optimization problem : arg max h p ( ) h x | ( 13. 16 ) = arg max h log ( ) p h x | ( 13. 17 ) = arg min h λ | | | | h 1 + β | | − | | x wh 2 2, ( 13. 18 ) where we


) = arg max h log ( ) p h x | ( 13. 17 ) = arg min h λ | | | | h 1 + β | | − | | x wh 2 2, ( 13. 18 ) where we have dropped terms not depending on h and divided by positive scaling factors to simplify the equation. due to the imposition of an l1 norm on h, this procedure will yield a sparse h∗ ( see section ). 7. 1. 2 to train the model rather than just perform inference, we alternate between minimization with respect to h and minimization with respect to w. in this presentation, we treat β as a hyperparameter. typically it is set to 1 because its role in this optimization problem is shared with λ and there is no need for both hyperparameters. in principle, we could also treat β as a parameter of the model and learn it. our presentation here has discarded some terms that do not depend on h but do depend on β. to learn β, these terms must be included, or β will collapse to. 0 not all approaches to sparse coding explicitly build a p ( h ) and a p ( x h | ). often we are just interested in learning a dictionary


β. to learn β, these terms must be included, or β will collapse to. 0 not all approaches to sparse coding explicitly build a p ( h ) and a p ( x h | ). often we are just interested in learning a dictionary of features with activation values that will often be zero when extracted using this inference procedure. if we sample h from a laplace prior, it is in fact a zero probability event for an element of h to actually be zero. the generative model itself is not especially sparse, only the feature extractor is. ( ) describe approximate goodfellow et al. 2013d 497


chapter 13. linear factor models inference in a [UNK] model family, the spike and slab sparse coding model, for which samples from the prior usually contain true zeros. the sparse coding approach combined with the use of the non - parametric encoder can in principle minimize the combination of reconstruction error and log - prior better than any speciﬁc parametric encoder. another advantage is that there is no generalization error to the encoder. a parametric encoder must learn how to map x to h in a way that generalizes. for unusual x that do not resemble the training data, a learned, parametric encoder may fail to ﬁnd an h that results in accurate reconstruction or a sparse code. for the vast majority of formulations of sparse coding models, where the inference problem is convex, the optimization procedure will always ﬁnd the optimal code ( unless degenerate cases such as replicated weight vectors occur ). obviously, the sparsity and reconstruction costs can still rise on unfamiliar points, but this is due to generalization error in the decoder weights, rather than generalization error in the encoder. the lack of generalization error in sparse coding ’ s optimization - based encoding process may result in better generalization


unfamiliar points, but this is due to generalization error in the decoder weights, rather than generalization error in the encoder. the lack of generalization error in sparse coding ’ s optimization - based encoding process may result in better generalization when sparse coding is used as a feature extractor for a classiﬁer than when a parametric function is used to predict the code. coates and ng 2011 ( ) demonstrated that sparse coding features generalize better for object recognition tasks than the features of a related model based on a parametric encoder, the linear - sigmoid autoencoder. inspired by their work, goodfellow et al. ( ) showed that a variant of sparse coding generalizes better than other feature 2013d extractors in the regime where extremely few labels are available ( twenty or fewer labels per class ). the primary disadvantage of the non - parametric encoder is that it requires greater time to compute h given x because the non - parametric approach requires running an iterative algorithm. the parametric autoencoder approach, developed in chapter, uses only a ﬁxed number of layers, often only one. another 14 disadvantage is that it is not straight - forward to back - propagate through the non


iterative algorithm. the parametric autoencoder approach, developed in chapter, uses only a ﬁxed number of layers, often only one. another 14 disadvantage is that it is not straight - forward to back - propagate through the non - parametric encoder, which makes it [UNK] to pretrain a sparse coding model with an unsupervised criterion and then ﬁne - tune it using a supervised criterion. modiﬁed versions of sparse coding that permit approximate derivatives do exist but are not widely used (, ). bagnell and bradley 2009 sparse coding, like other linear factor models, often produces poor samples, as shown in ﬁgure. this happens even when the model is able to reconstruct 13. 2 the data well and provide useful features for a classiﬁer. the reason is that each individual feature may be learned well, but the factorial prior on the hidden code results in the model including random subsets of all of the features in each generated sample. this motivates the development of deeper models that can impose a non - 498


chapter 13. linear factor models figure 13. 2 : example samples and weights from a spike and slab sparse coding model trained on the mnist dataset. ( left ) the samples from the model do not resemble the training examples. at ﬁrst glance, one might assume the model is poorly ﬁt. the ( right ) weight vectors of the model have learned to represent penstrokes and sometimes complete digits. the model has thus learned useful features. the problem is that the factorial prior over features results in random subsets of features being combined. few such subsets are appropriate to form a recognizable mnist digit. this motivates the development of generative models that have more powerful distributions over their latent codes. figure reproduced with permission from goodfellow 2013d et al. ( ). factorial distribution on the deepest code layer, as well as the development of more sophisticated shallow models. 13. 5 manifold interpretation of pca linear factor models including pca and factor analysis can be interpreted as learning a manifold (, ). we can view probabilistic pca as hinton et al. 1997 deﬁning a thin pancake - shaped region of high probability — a gaussian distribution that is very narrow along some axes, just as a pan


, ). we can view probabilistic pca as hinton et al. 1997 deﬁning a thin pancake - shaped region of high probability — a gaussian distribution that is very narrow along some axes, just as a pancake is very ﬂat along its vertical axis, but is elongated along other axes, just as a pancake is wide along its horizontal axes. this is illustrated in ﬁgure. pca can be interpreted as aligning this 13. 3 pancake with a linear manifold in a higher - dimensional space. this interpretation applies not just to traditional pca but also to any linear autoencoder that learns matrices w and v with the goal of making the reconstruction of x lie as close to x as possible, let the encoder be h x w = ( f ) = ( ) x µ −. ( 13. 19 ) 499


chapter 13. linear factor models the encoder computes a low - dimensional representation of h. with the autoencoder view, we have a decoder computing the reconstruction [UNK] h b v h = ( g ) = +. ( 13. 20 ) figure 13. 3 : flat gaussian capturing probability concentration near a low - dimensional manifold. the ﬁgure shows the upper half of the “ pancake ” above the “ manifold plane ” which goes through its middle. the variance in the direction orthogonal to the manifold is very small ( arrow pointing out of plane ) and can be considered like “ noise, ” while the other variances are large ( arrows in the plane ) and correspond to “ signal, ” and a coordinate system for the reduced - dimension data. the choices of linear encoder and decoder that minimize reconstruction error e [ | | − x [UNK] | | 2 ] ( 13. 21 ) correspond to v = w, µ = b = e [ x ] and the columns of w form an orthonormal basis which spans the same subspace as the principal eigenvectors of the covariance matrix c x µ x µ = [ ( e − ) ( − ) ]. ( 13. 22 ) in the


form an orthonormal basis which spans the same subspace as the principal eigenvectors of the covariance matrix c x µ x µ = [ ( e − ) ( − ) ]. ( 13. 22 ) in the case of pca, the columns of w are these eigenvectors, ordered by the magnitude of the corresponding eigenvalues ( which are all real and non - negative ). one can also show that eigenvalue λi of c corresponds to the variance of x in the direction of eigenvector v ( ) i. if x ∈rd and h ∈rd with d < d, then the 500


chapter 13. linear factor models optimal reconstruction error ( choosing,, and as above ) is µ b v w min [ e | | − x [UNK] | | 2 ] = d i d = + 1 λi. ( 13. 23 ) hence, if the covariance has rank d, the eigenvalues λd + 1 to λd are 0 and recon - struction error is 0. furthermore, one can also show that the above solution can be obtained by maximizing the variances of the elements of h, under orthogonal w, instead of minimizing reconstruction error. linear factor models are some of the simplest generative models and some of the simplest models that learn a representation of data. much as linear classiﬁers and linear regression models may be extended to deep feedforward networks, these linear factor models may be extended to autoencoder networks and deep probabilistic models that perform the same tasks but with a much more powerful and ﬂexible model family. 501


chapter 14 autoencoders an autoencoder is a neural network that is trained to attempt to copy its input to its output. internally, it has a hidden layer h that describes a code used to represent the input. the network may be viewed as consisting of two parts : an encoder function h = f ( x ) and a decoder that produces a reconstruction r = g ( h ). this architecture is presented in ﬁgure. if an autoencoder succeeds in simply 14. 1 learning to set g ( f ( x ) ) = x everywhere, then it is not especially useful. instead, autoencoders are designed to be unable to learn to copy perfectly. usually they are restricted in ways that allow them to copy only approximately, and to copy only input that resembles the training data. because the model is forced to prioritize which aspects of the input should be copied, it often learns useful properties of the data. modern autoencoders have generalized the idea of an encoder and a de - coder beyond deterministic functions to stochastic mappings pencoder ( h x | ) and pdecoder ( ) x h |. the idea of autoencoders has been part of the historical


##coder and a de - coder beyond deterministic functions to stochastic mappings pencoder ( h x | ) and pdecoder ( ) x h |. the idea of autoencoders has been part of the historical landscape of neural networks for decades (, ;, ;, lecun 1987 bourlard and kamp 1988 hinton and zemel 1994 ). traditionally, autoencoders were used for dimensionality reduction or feature learning. recently, theoretical connections between autoencoders and latent variable models have brought autoencoders to the forefront of generative modeling, as we will see in chapter. autoencoders may be thought of as being 20 a special case of feedforward networks, and may be trained with all of the same techniques, typically minibatch gradient descent following gradients computed by back - propagation. unlike general feedforward networks, autoencoders may also be trained using recirculation ( hinton and mcclelland 1988, ), a learning algorithm based on comparing the activations of the network on the original input 502


chapter 14. autoencoders to the activations on the reconstructed input. recirculation is regarded as more biologically plausible than back - propagation, but is rarely used for machine learning applications. x r h f g figure 14. 1 : the general structure of an autoencoder, mapping an input to an output x ( called reconstruction ) r through an internal representation or code h. the autoencoder has two components : the encoder f ( mapping x to h ) and the decoder g ( mapping h to r ). 14. 1 undercomplete autoencoders copying the input to the output may sound useless, but we are typically not interested in the output of the decoder. instead, we hope that training the autoencoder to perform the input copying task will result in h taking on useful properties. one way to obtain useful features from the autoencoder is to constrain h to have smaller dimension than x. an autoencoder whose code dimension is less than the input dimension is called undercomplete. learning an undercomplete representation forces the autoencoder to capture the most salient features of the training data. the learning process is described simply as minimizing a


dimension is less than the input dimension is called undercomplete. learning an undercomplete representation forces the autoencoder to capture the most salient features of the training data. the learning process is described simply as minimizing a loss function l, g f ( x ( ( ) ) ) x ( 14. 1 ) where l is a loss function penalizing g ( f ( x ) ) for being dissimilar from x, such as the mean squared error. when the decoder is linear and l is the mean squared error, an undercomplete autoencoder learns to span the same subspace as pca. in this case, an autoencoder trained to perform the copying task has learned the principal subspace of the training data as a side - [UNK]. autoencoders with nonlinear encoder functions f and nonlinear decoder func - tions g can thus learn a more powerful nonlinear generalization of pca. unfortu - 503


chapter 14. autoencoders nately, if the encoder and decoder are allowed too much capacity, the autoencoder can learn to perform the copying task without extracting useful information about the distribution of the data. theoretically, one could imagine that an autoencoder with a one - dimensional code but a very powerful nonlinear encoder could learn to represent each training example x ( ) i with the code i. the decoder could learn to map these integer indices back to the values of speciﬁc training examples. this speciﬁc scenario does not occur in practice, but it illustrates clearly that an autoen - coder trained to perform the copying task can fail to learn anything useful about the dataset if the capacity of the autoencoder is allowed to become too great. 14. 2 regularized autoencoders undercomplete autoencoders, with code dimension less than the input dimension, can learn the most salient features of the data distribution. we have seen that these autoencoders fail to learn anything useful if the encoder and decoder are given too much capacity. a similar problem occurs if the hidden code is allowed to have dimension equal to the input, and in the overcom


have seen that these autoencoders fail to learn anything useful if the encoder and decoder are given too much capacity. a similar problem occurs if the hidden code is allowed to have dimension equal to the input, and in the overcomplete case in which the hidden code has dimension greater than the input. in these cases, even a linear encoder and linear decoder can learn to copy the input to the output without learning anything useful about the data distribution. ideally, one could train any architecture of autoencoder successfully, choosing the code dimension and the capacity of the encoder and decoder based on the complexity of distribution to be modeled. regularized autoencoders provide the ability to do so. rather than limiting the model capacity by keeping the encoder and decoder shallow and the code size small, regularized autoencoders use a loss function that encourages the model to have other properties besides the ability to copy its input to its output. these other properties include sparsity of the representation, smallness of the derivative of the representation, and robustness to noise or to missing inputs. a regularized autoencoder can be nonlinear and overcomplete but still learn something useful about the data distribution even if


##ity of the representation, smallness of the derivative of the representation, and robustness to noise or to missing inputs. a regularized autoencoder can be nonlinear and overcomplete but still learn something useful about the data distribution even if the model capacity is great enough to learn a trivial identity function. in addition to the methods described here which are most naturally interpreted as regularized autoencoders, nearly any generative model with latent variables and equipped with an inference procedure ( for computing latent representations given input ) may be viewed as a particular form of autoencoder. two generative modeling approaches that emphasize this connection with autoencoders are the descendants of the helmholtz machine (, ), such as the variational hinton et al. 1995b 504


chapter 14. autoencoders autoencoder ( section ) and the generative stochastic networks ( section ). 20. 10. 3 20. 12 these models naturally learn high - capacity, overcomplete encodings of the input and do not require regularization for these encodings to be useful. their encodings are naturally useful because the models were trained to approximately maximize the probability of the training data rather than to copy the input to the output. 14. 2. 1 sparse autoencoders a sparse autoencoder is simply an autoencoder whose training criterion involves a sparsity penalty ω ( h ) on the code layer h, in addition to the reconstruction error : l, g f ( x ( ( ) ) ) + ω ( ) x h ( 14. 2 ) where g ( h ) is the decoder output and typically we have h = f ( x ), the encoder output. sparse autoencoders are typically used to learn features for another task such as classiﬁcation. an autoencoder that has been regularized to be sparse must respond to unique statistical features of the dataset it has been trained on, rather than simply acting as an identity function. in this way,


task such as classiﬁcation. an autoencoder that has been regularized to be sparse must respond to unique statistical features of the dataset it has been trained on, rather than simply acting as an identity function. in this way, training to perform the copying task with a sparsity penalty can yield a model that has learned useful features as a byproduct. we can think of the penalty ω ( h ) simply as a regularizer term added to a feedforward network whose primary task is to copy the input to the output ( unsupervised learning objective ) and possibly also perform some supervised task ( with a supervised learning objective ) that depends on these sparse features. unlike other regularizers such as weight decay, there is not a straightforward bayesian interpretation to this regularizer. as described in section, training 5. 6. 1 with weight decay and other regularization penalties can be interpreted as a map approximation to bayesian inference, with the added regularizing penalty corresponding to a prior probability distribution over the model parameters. in this view, regularized maximum likelihood corresponds to maximizing p ( θ x | ), which is equivalent to maximizing log p ( x θ | ) + log p ( θ ). the log p (


probability distribution over the model parameters. in this view, regularized maximum likelihood corresponds to maximizing p ( θ x | ), which is equivalent to maximizing log p ( x θ | ) + log p ( θ ). the log p ( x θ | ) term is the usual data log - likelihood term and the log p ( θ ) term, the log - prior over parameters, incorporates the preference over particular values of θ. this view was described in section. regularized autoencoders defy such an interpretation 5. 6 because the regularizer depends on the data and is therefore by deﬁnition not a prior in the formal sense of the word. we can still think of these regularization terms as implicitly expressing a preference over functions. rather than thinking of the sparsity penalty as a regularizer for the copying task, we can think of the entire sparse autoencoder framework as approximating 505


chapter 14. autoencoders maximum likelihood training of a generative model that has latent variables. suppose we have a model with visible variables x and latent variables h, with an explicit joint distribution pmodel ( x h, ) = p model ( h ) pmodel ( x h | ). we refer to pmodel ( h ) as the model ’ s prior distribution over the latent variables, representing the model ’ s beliefs prior to seeing x. this is [UNK] from the way we have previously used the word “ prior, ” to refer to the distribution p ( θ ) encoding our beliefs about the model ’ s parameters before we have seen the training data. the log - likelihood can be decomposed as log pmodel ( ) = log x h pmodel ( ) h x,. ( 14. 3 ) we can think of the autoencoder as approximating this sum with a point estimate for just one highly likely value for h. this is similar to the sparse coding generative model ( section ), but with 13. 4 h being the output of the parametric encoder rather than the result of an optimization that infers the most likely h. from this point of view, with this chosen, we


coding generative model ( section ), but with 13. 4 h being the output of the parametric encoder rather than the result of an optimization that infers the most likely h. from this point of view, with this chosen, we are maximizing h log pmodel ( ) = log h x, pmodel ( ) + log h pmodel ( ) x h |. ( 14. 4 ) the log pmodel ( ) h term can be sparsity - inducing. for example, the laplace prior, pmodel ( hi ) = λ 2 e− | λ hi |, ( 14. 5 ) corresponds to an absolute value sparsity penalty. expressing the log - prior as an absolute value penalty, we obtain ω ( ) = h λ i | hi | ( 14. 6 ) −log pmodel ( ) = h i λ h | i | −log λ 2 = ω ( ) + const h ( 14. 7 ) where the constant term depends only on λ and not h. we typically treat λ as a hyperparameter and discard the constant term since it does not [UNK] the parameter learning. other priors such as the student - t prior can also induce sparsity.


term depends only on λ and not h. we typically treat λ as a hyperparameter and discard the constant term since it does not [UNK] the parameter learning. other priors such as the student - t prior can also induce sparsity. from this point of view of sparsity as resulting from the [UNK] of pmodel ( h ) on approximate maximum likelihood learning, the sparsity penalty is not a regularization term at all. it is just a consequence of the model ’ s distribution over its latent variables. this view provides a [UNK] motivation for training an autoencoder : it is a way of approximately training a generative model. it also provides a [UNK] reason for 506


chapter 14. autoencoders why the features learned by the autoencoder are useful : they describe the latent variables that explain the input. early work on sparse autoencoders (,, ) explored ranzato et al. 2007a 2008 various forms of sparsity and proposed a connection between the sparsity penalty and the log z term that arises when applying maximum likelihood to an undirected probabilistic model p ( x ) = 1 z [UNK] ( x ). the idea is that minimizing log z prevents a probabilistic model from having high probability everywhere, and imposing sparsity on an autoencoder prevents the autoencoder from having low reconstruction error everywhere. in this case, the connection is on the level of an intuitive understanding of a general mechanism rather than a mathematical correspondence. the interpretation of the sparsity penalty as corresponding to log pmodel ( h ) in a directed model pmodel ( ) h pmodel ( ) x h | is more mathematically straightforward. one way to achieve actual zeros in h for sparse ( and denoising ) autoencoders was introduced in ( ). the idea is to use rectiﬁed linear units to glorot


h | is more mathematically straightforward. one way to achieve actual zeros in h for sparse ( and denoising ) autoencoders was introduced in ( ). the idea is to use rectiﬁed linear units to glorot et al. 2011b produce the code layer. with a prior that actually pushes the representations to zero ( like the absolute value penalty ), one can thus indirectly control the average number of zeros in the representation. 14. 2. 2 denoising autoencoders rather than adding a penalty to the cost function, we can obtain an autoencoder ω that learns something useful by changing the reconstruction error term of the cost function. traditionally, autoencoders minimize some function l, g f ( x ( ( ) ) ) x ( 14. 8 ) where l is a loss function penalizing g ( f ( x ) ) for being dissimilar from x, such as the l2 norm of their [UNK]. this encourages g f [UNK] to learn to be merely an identity function if they have the capacity to do so. a or dae instead minimizes denoising autoencoder l, g f ( x ( ( [UNK] ) ) ), ( 14. 9 ) where [UNK] is


to be merely an identity function if they have the capacity to do so. a or dae instead minimizes denoising autoencoder l, g f ( x ( ( [UNK] ) ) ), ( 14. 9 ) where [UNK] is a copy of x that has been corrupted by some form of noise. denoising autoencoders must therefore undo this corruption rather than simply copying their input. denoising training forces f and g to implicitly learn the structure of pdata ( x ), as shown by ( ) and ( ). denoising alain and bengio 2013 bengio et al. 2013c 507


chapter 14. autoencoders autoencoders thus provide yet another example of how useful properties can emerge as a byproduct of minimizing reconstruction error. they are also an example of how overcomplete, high - capacity models may be used as autoencoders so long as care is taken to prevent them from learning the identity function. denoising autoencoders are presented in more detail in section. 14. 5 14. 2. 3 regularizing by penalizing derivatives another strategy for regularizing an autoencoder is to use a penalty as in sparse ω autoencoders, l, g f,, ( x ( ( ) ) ) + ω ( x h x ) ( 14. 10 ) but with a [UNK] form of : ω ω ( ) = h x, λ i | | ∇xh i | | 2. ( 14. 11 ) this forces the model to learn a function that does not change much when x changes slightly. because this penalty is applied only at training examples, it forces the autoencoder to learn features that capture information about the training distribution. an autoencoder regularized in this way is called a contractive autoencoder or cae. this approach has


applied only at training examples, it forces the autoencoder to learn features that capture information about the training distribution. an autoencoder regularized in this way is called a contractive autoencoder or cae. this approach has theoretical connections to denoising autoencoders, manifold learning and probabilistic modeling. the cae is described in more detail in section. 14. 7 14. 3 representational power, layer size and depth autoencoders are often trained with only a single layer encoder and a single layer decoder. however, this is not a requirement. in fact, using deep encoders and decoders [UNK] many advantages. recall from section that there are many advantages to depth in a feedfor - 6. 4. 1 ward network. because autoencoders are feedforward networks, these advantages also apply to autoencoders. moreover, the encoder is itself a feedforward network as is the decoder, so each of these components of the autoencoder can individually beneﬁt from depth. one major advantage of non - trivial depth is that the universal approximator theorem guarantees that a feedforward neural network with at least one hidden layer can represent an


these components of the autoencoder can individually beneﬁt from depth. one major advantage of non - trivial depth is that the universal approximator theorem guarantees that a feedforward neural network with at least one hidden layer can represent an approximation of any function ( within a broad class ) to an 508


chapter 14. autoencoders arbitrary degree of accuracy, provided that it has enough hidden units. this means that an autoencoder with a single hidden layer is able to represent the identity function along the domain of the data arbitrarily well. however, the mapping from input to code is shallow. this means that we are not able to enforce arbitrary constraints, such as that the code should be sparse. a deep autoencoder, with at least one additional hidden layer inside the encoder itself, can approximate any mapping from input to code arbitrarily well, given enough hidden units. depth can exponentially reduce the computational cost of representing some functions. depth can also exponentially decrease the amount of training data needed to learn some functions. see section for a review of the advantages of 6. 4. 1 depth in feedforward networks. experimentally, deep autoencoders yield much better compression than corre - sponding shallow or linear autoencoders ( hinton and salakhutdinov 2006, ). a common strategy for training a deep autoencoder is to greedily pretrain the deep architecture by training a stack of shallow autoencoders, so we often encounter shallow autoencoders, even


##akhutdinov 2006, ). a common strategy for training a deep autoencoder is to greedily pretrain the deep architecture by training a stack of shallow autoencoders, so we often encounter shallow autoencoders, even when the ultimate goal is to train a deep autoencoder. 14. 4 stochastic encoders and decoders autoencoders are just feedforward networks. the same loss functions and output unit types that can be used for traditional feedforward networks are also used for autoencoders. as described in section, a general strategy for designing the output units 6. 2. 2. 4 and the loss function of a feedforward network is to deﬁne an output distribution p ( y x | ) and minimize the negative log - likelihood −log p ( y x | ). in that setting, y was a vector of targets, such as class labels. in the case of an autoencoder, x is now the target as well as the input. however, we can still apply the same machinery as before. given a hidden code h, we may think of the decoder as providing a conditional distribution p decoder ( x h | ). we may then train the auto


well as the input. however, we can still apply the same machinery as before. given a hidden code h, we may think of the decoder as providing a conditional distribution p decoder ( x h | ). we may then train the autoencoder by minimizing −log pdecoder ( ) x h |. the exact form of this loss function will change depending on the form of pdecoder. as with traditional feedforward networks, we usually use linear output units to parametrize the mean of a gaussian distribution if x is real - valued. in that case, the negative log - likelihood yields a mean squared error criterion. similarly, binary x values correspond to a bernoulli distribution whose parameters are given by a sigmoid output unit, discrete x values correspond to a softmax distribution, and so on. 509


chapter 14. autoencoders typically, the output variables are treated as being conditionally independent given h so that this probability distribution is inexpensive to evaluate, but some techniques such as mixture density outputs allow tractable modeling of outputs with correlations. x r h pencoder ( ) h x | pdecoder ( ) x h | figure 14. 2 : the structure of a stochastic autoencoder, in which both the encoder and the decoder are not simple functions but instead involve some noise injection, meaning that their output can be seen as sampled from a distribution, pencoder ( h x | ) for the encoder and pdecoder ( ) x h | for the decoder. to make a more radical departure from the feedforward networks we have seen previously, we can also generalize the notion of an encoding function f ( x ) to an encoding distribution pencoder ( ) h x |, as illustrated in ﬁgure. 14. 2 any latent variable model pmodel ( ) h x, deﬁnes a stochastic encoder pencoder ( ) = h x | pmodel ( ) h x | ( 14. 12 ) and a stochastic decoder pdec


model pmodel ( ) h x, deﬁnes a stochastic encoder pencoder ( ) = h x | pmodel ( ) h x | ( 14. 12 ) and a stochastic decoder pdecoder ( ) = x h | pmodel ( ) x h |. ( 14. 13 ) in general, the encoder and decoder distributions are not necessarily conditional distributions compatible with a unique joint distribution pmodel ( x h, ). alain et al. ( ) showed that training the encoder and decoder as a denoising autoencoder 2015 will tend to make them compatible asymptotically ( with enough capacity and examples ). 14. 5 denoising autoencoders the denoising autoencoder ( dae ) is an autoencoder that receives a corrupted data point as input and is trained to predict the original, uncorrupted data point as its output. the dae training procedure is illustrated in ﬁgure. we introduce a 14. 3 corruption process c ( [UNK] x | ) which represents a conditional distribution over 510


chapter 14. autoencoders [UNK] l h f g x c ( [UNK] x | ) figure 14. 3 : the computational graph of the cost function for a denoising autoencoder, which is trained to reconstruct the clean data point x from its corrupted version [UNK]. this is accomplished by minimizing the loss l = −log pdecoder ( x h | = f ( [UNK] ) ), where [UNK] is a corrupted version of the data example x, obtained through a given corruption process c ( [UNK] x | ). typically the distribution pdecoder is a factorial distribution whose mean parameters are emitted by a feedforward network. g corrupted samples [UNK], given a data sample x. the autoencoder then learns a reconstruction distribution preconstruct ( x | [UNK] ) estimated from training pairs ( x, [UNK] ), as follows : 1. sample a training example from the training data. x 2. sample a corrupted version [UNK] from c ( [UNK] x | = ) x. 3. use ( x, [UNK] ) as a training example for estimating the autoencoder reconstruction distribution preconstruct ( x | [UNK] ) = pdecoder ( x h | ) with h the output of encoder


. 3. use ( x, [UNK] ) as a training example for estimating the autoencoder reconstruction distribution preconstruct ( x | [UNK] ) = pdecoder ( x h | ) with h the output of encoder f ( [UNK] ) and pdecoder typically deﬁned by a decoder. g ( ) h typically we can simply perform gradient - based approximate minimization ( such as minibatch gradient descent ) on the negative log - likelihood −log pdecoder ( x h | ). so long as the encoder is deterministic, the denoising autoencoder is a feedforward network and may be trained with exactly the same techniques as any other feedforward network. we can therefore view the dae as performing stochastic gradient descent on the following expectation : [UNK] ( ) x e [UNK] [UNK] ( [UNK] | x ) log pdecoder ( = ( x h | f [UNK] ) ) ( 14. 14 ) where [UNK] ( ) x is the training distribution. 511


chapter 14. autoencoders x [UNK] g f [UNK] [UNK] c ( [UNK] x | ) x figure 14. 4 : a denoising autoencoder is trained to map a corrupted data [UNK] back to the original data point x. we illustrate training examples x as red crosses lying near a low - dimensional manifold illustrated with the bold black line. we illustrate the corruption processc ( [UNK] x | ) with a gray circle of equiprobable corruptions. a gray arrow demonstrates how one training example is transformed into one sample from this corruption process. when the denoising autoencoder is trained to minimize the average of squared errors | | g ( f ( [UNK] ) ) − | | x 2, the reconstruction g ( f ( [UNK] ) ) estimates ex, [UNK] ( ) ( x c [UNK] x | ) [ x | [UNK] ]. the vector g ( f ( [UNK] ) ) [UNK] points approximately towards the nearest point on the manifold, sinceg ( f ( [UNK] ) ) estimates the center of mass of the clean pointsx which could have given rise to [UNK]. the autoencoder thus learns a vector ﬁeld g ( f ( x ) ) −x indicated by the green arrows. this vector ﬁeld estimates the score ∇xlog


mass of the clean pointsx which could have given rise to [UNK]. the autoencoder thus learns a vector ﬁeld g ( f ( x ) ) −x indicated by the green arrows. this vector ﬁeld estimates the score ∇xlog pdata ( x ) up to a multiplicative factor that is the average root mean square reconstruction error. 512


chapter 14. autoencoders 14. 5. 1 estimating the score score matching (, ) is an alternative to maximum likelihood. it hyvarinen 2005 provides a consistent estimator of probability distributions based on encouraging the model to have the same score as the data distribution at every training point x. in this context, the score is a particular gradient ﬁeld : ∇x log ( ) p x. ( 14. 15 ) score matching is discussed further in section. for the present discussion 18. 4 regarding autoencoders, it is [UNK] to understand that learning the gradient ﬁeld of log pdata is one way to learn the structure of pdata itself. a very important property of daes is that their training criterion ( with conditionally gaussian p ( x h | ) ) makes the autoencoder learn a vector ﬁeld ( g ( f ( x ) ) −x ) that estimates the score of the data distribution. this is illustrated in ﬁgure. 14. 4 denoising training of a speciﬁc kind of autoencoder ( sigmoidal hidden units, linear reconstruction units ) using gaussian noise and mean squared error as the reconstruction cost is equivalent (, ) to training a speci


##oising training of a speciﬁc kind of autoencoder ( sigmoidal hidden units, linear reconstruction units ) using gaussian noise and mean squared error as the reconstruction cost is equivalent (, ) to training a speciﬁc kind vincent 2011 of undirected probabilistic model called an rbm with gaussian visible units. this kind of model will be described in detail in section ; for the present 20. 5. 1 discussion it [UNK] to know that it is a model that provides an explicit pmodel ( x ; θ ). when the rbm is trained using denoising score matching (, kingma and lecun 2010 ), its learning algorithm is equivalent to denoising training in the corresponding autoencoder. with a ﬁxed noise level, regularized score matching is not a consistent estimator ; it instead recovers a blurred version of the distribution. however, if the noise level is chosen to approach 0 when the number of examples approaches inﬁnity, then consistency is recovered. denoising score matching is discussed in more detail in section. 18. 5 other connections between autoencoders and rbms exist. score matching applied to rbms yields a cost function that is identical


inﬁnity, then consistency is recovered. denoising score matching is discussed in more detail in section. 18. 5 other connections between autoencoders and rbms exist. score matching applied to rbms yields a cost function that is identical to reconstruction error combined with a regularization term similar to the contractive penalty of the cae ( swersky 2011 bengio and delalleau 2009 et al., ). ( ) showed that an autoen - coder gradient provides an approximation to contrastive divergence training of rbms. for continuous - valued x, the denoising criterion with gaussian corruption and reconstruction distribution yields an estimator of the score that is applicable to general encoder and decoder parametrizations (, ). this alain and bengio 2013 means a generic encoder - decoder architecture may be made to estimate the score 513


chapter 14. autoencoders by training with the squared error criterion | | g f ( ( [UNK] x ) ) − | | 2 ( 14. 16 ) and corruption c ( [UNK] = [UNK] x | ) = ( n [UNK] x ; = µ, σ σ = 2 i ) ( 14. 17 ) with noise variance σ2. see ﬁgure for an illustration of how this works. 14. 5 figure 14. 5 : vector ﬁeld learned by a denoising autoencoder around a 1 - d curved manifold near which the data concentrates in a 2 - d space. each arrow is proportional to the reconstruction minus input vector of the autoencoder and points towards higher probability according to the implicitly estimated probability distribution. the vector ﬁeld has zeros at both maxima of the estimated density function ( on the data manifolds ) and at minima of that density function. for example, the spiral arm forms a one - dimensional manifold of local maxima that are connected to each other. local minima appear near the middle of the gap between two arms. when the norm of reconstruction error ( shown by the length of the arrows ) is large, it means that probability can be signiﬁcantly increased by moving in the direction


. local minima appear near the middle of the gap between two arms. when the norm of reconstruction error ( shown by the length of the arrows ) is large, it means that probability can be signiﬁcantly increased by moving in the direction of the arrow, and that is mostly the case in places of low probability. the autoencoder maps these low probability points to higher probability reconstructions. where probability is maximal, the arrows shrink because the reconstruction becomes more accurate. figure reproduced with permission from ( ). alain and bengio 2013 in general, there is no guarantee that the reconstruction g ( f ( x ) ) minus the input x corresponds to the gradient of any function, let alone to the score. that is 514


chapter 14. autoencoders why the early results (, ) are specialized to particular parametrizations vincent 2011 where g ( f ( x ) ) −x may be obtained by taking the derivative of another function. kamyshanska and memisevic 2015 vincent 2011 ( ) generalized the results of ( ) by identifying a family of shallow autoencoders such that g ( f ( x ) ) −x corresponds to a score for all members of the family. so far we have described only how the denoising autoencoder learns to represent a probability distribution. more generally, one may want to use the autoencoder as a generative model and draw samples from this distribution. this will be described later, in section. 20. 11 14. 5. 1. 1 historical perspective the idea of using mlps for denoising dates back to the work of ( ) lecun 1987 and ( ). ( ) also used recurrent networks to denoise gallinari et al. 1987 behnke 2001 images. denoising autoencoders are, in some sense, just mlps trained to denoise. however, the name “ denoising autoencoder ” refers to a model that is intended not merely


1987 behnke 2001 images. denoising autoencoders are, in some sense, just mlps trained to denoise. however, the name “ denoising autoencoder ” refers to a model that is intended not merely to learn to denoise its input but to learn a good internal representation as a side [UNK] of learning to denoise. this idea came much later ( vincent et al.,, ). the learned representation may then be used to pretrain a 2008 2010 deeper unsupervised network or a supervised network. like sparse autoencoders, sparse coding, contractive autoencoders and other regularized autoencoders, the motivation for daes was to allow the learning of a very high - capacity encoder while preventing the encoder and decoder from learning a useless identity function. prior to the introduction of the modern dae, inayoshi and kurita 2005 ( ) explored some of the same goals with some of the same methods. their approach minimizes reconstruction error in addition to a supervised objective while injecting noise in the hidden layer of a supervised mlp, with the objective to improve generalization by introducing the reconstruction error and the injected noise. however, their method was based on a


minimizes reconstruction error in addition to a supervised objective while injecting noise in the hidden layer of a supervised mlp, with the objective to improve generalization by introducing the reconstruction error and the injected noise. however, their method was based on a linear encoder and could not learn function families as powerful as can the modern dae. 14. 6 learning manifolds with autoencoders like many other machine learning algorithms, autoencoders exploit the idea that data concentrates around a low - dimensional manifold or a small set of such manifolds, as described in section. some machine learning algorithms exploit 5. 11. 3 this idea only insofar as that they learn a function that behaves correctly on the manifold but may have unusual behavior if given an input that is [UNK] manifold. 515


chapter 14. autoencoders autoencoders take this idea further and aim to learn the structure of the manifold. to understand how autoencoders do this, we must present some important characteristics of manifolds. an important characterization of a manifold is the set of its tangent planes. at a point x on a d - dimensional manifold, the tangent plane is given by d basis vectors that span the local directions of variation allowed on the manifold. as illustrated in ﬁgure, these local directions specify how one can change 14. 6 x inﬁnitesimally while staying on the manifold. all autoencoder training procedures involve a compromise between two forces : 1. learning a representation h of a training example x such that x can be approximately recovered from h through a decoder. the fact that x is drawn from the training data is crucial, because it means the autoencoder need not successfully reconstruct inputs that are not probable under the data generating distribution. 2. satisfying the constraint or regularization penalty. this can be an architec - tural constraint that limits the capacity of the autoencoder, or it can be a regularization term added to the reconstruction cost. these techniques generally prefer solutions that are less sensitive to the


##ization penalty. this can be an architec - tural constraint that limits the capacity of the autoencoder, or it can be a regularization term added to the reconstruction cost. these techniques generally prefer solutions that are less sensitive to the input. clearly, neither force alone would be useful — copying the input to the output is not useful on its own, nor is ignoring the input. instead, the two forces together are useful because they force the hidden representation to capture information about the structure of the data generating distribution. the important principle is that the autoencoder can [UNK] to represent only the variations that are needed to reconstruct training examples. if the data generating distribution concentrates near a low - dimensional manifold, this yields representations that implicitly capture a local coordinate system for this manifold : only the variations tangent to the manifold around x need to correspond to changes in h = f ( x ). hence the encoder learns a mapping from the input space x to a representation space, a mapping that is only sensitive to changes along the manifold directions, but that is insensitive to changes orthogonal to the manifold. a one - dimensional example is illustrated in ﬁgure, showing that, by making 14. 7 the reconstruction function insensitive to per


only sensitive to changes along the manifold directions, but that is insensitive to changes orthogonal to the manifold. a one - dimensional example is illustrated in ﬁgure, showing that, by making 14. 7 the reconstruction function insensitive to perturbations of the input around the data points, we cause the autoencoder to recover the manifold structure. to understand why autoencoders are useful for manifold learning, it is in - structive to compare them to other approaches. what is most commonly learned to characterize a manifold is a representation of the data points on ( or near ) 516


chapter 14. autoencoders figure 14. 6 : an illustration of the concept of a tangent hyperplane. here we create a one - dimensional manifold in 784 - dimensional space. we take an mnist image with 784 pixels and transform it by translating it vertically. the amount of vertical translation deﬁnes a coordinate along a one - dimensional manifold that traces out a curved path through image space. this plot shows a few points along this manifold. for visualization, we have projected the manifold into two dimensional space using pca. an n - dimensional manifold has an n - dimensional tangent plane at every point. this tangent plane touches the manifold exactly at that point and is oriented parallel to the surface at that point. it deﬁnes the space of directions in which it is possible to move while remaining on the manifold. this one - dimensional manifold has a single tangent line. we indicate an example tangent line at one point, with an image showing how this tangent direction appears in image space. gray pixels indicate pixels that do not change as we move along the tangent line, white pixels indicate pixels that brighten, and black pixels indicate pixels that darken. 517


chapter 14. autoencoders x0 x 1 x2 x 0 0. 0 2. 0 4. 0 6. 0 8. 1 0. r x ( ) identity optimal reconstruction figure 14. 7 : if the autoencoder learns a reconstruction function that is invariant to small perturbations near the data points, it captures the manifold structure of the data. here the manifold structure is a collection of - dimensional manifolds. the dashed diagonal 0 line indicates the identity function target for reconstruction. the optimal reconstruction function crosses the identity function wherever there is a data point. the horizontal arrows at the bottom of the plot indicate the r ( x ) −x reconstruction direction vector at the base of the arrow, in input space, always pointing towards the nearest “ manifold ” ( a single datapoint, in the 1 - d case ). the denoising autoencoder explicitly tries to make the derivative of the reconstruction function r ( x ) small around the data points. the contractive autoencoder does the same for the encoder. although the derivative ofr ( x ) is asked to be small around the data points, it can be large between the data points. the space between the data points corresponds to the region between the manifolds


does the same for the encoder. although the derivative ofr ( x ) is asked to be small around the data points, it can be large between the data points. the space between the data points corresponds to the region between the manifolds, where the reconstruction function must have a large derivative in order to map corrupted points back onto the manifold. the manifold. such a representation for a particular example is also called its embedding. it is typically given by a low - dimensional vector, with less dimensions than the “ ambient ” space of which the manifold is a low - dimensional subset. some algorithms ( non - parametric manifold learning algorithms, discussed below ) directly learn an embedding for each training example, while others learn a more general mapping, sometimes called an encoder, or representation function, that maps any point in the ambient space ( the input space ) to its embedding. manifold learning has mostly focused on unsupervised learning procedures that attempt to capture these manifolds. most of the initial machine learning research on learning nonlinear manifolds has focused on non - parametric methods based on the nearest - neighbor graph. this graph has one node per training example and edges connecting near neighbors to each other. these methods ( scholkopf


the initial machine learning research on learning nonlinear manifolds has focused on non - parametric methods based on the nearest - neighbor graph. this graph has one node per training example and edges connecting near neighbors to each other. these methods ( scholkopf et al., ; 1998 roweis and saul 2000 tenenbaum 2000 brand 2003 belkin, ; et al., ;, ; 518


chapter 14. autoencoders figure 14. 8 : non - parametric manifold learning procedures build a nearest neighbor graph in which nodes represent training examples a directed edges indicate nearest neighbor relationships. various procedures can thus obtain the tangent plane associated with a neighborhood of the graph as well as a coordinate system that associates each training example with a real - valued vector position, or embedding. it is possible to generalize such a representation to new examples by a form of interpolation. so long as the number of examples is large enough to cover the curvature and twists of the manifold, these approaches work well. images from the qmul multiview face dataset (, gong et al. 2000 ). and niyogi 2003 donoho and grimes 2003 weinberger and saul 2004 hinton, ;, ;, ; and roweis 2003 van der maaten and hinton 2008, ;, ) associate each of nodes with a tangent plane that spans the directions of variations associated with the [UNK] vectors between the example and its neighbors, as illustrated in ﬁgure. 14. 8 a global coordinate system can then be obtained through an optimization or solving a linear system. figure illustrates how a manifold can be tiled by a 14. 9 large number of locally linear gauss


its neighbors, as illustrated in ﬁgure. 14. 8 a global coordinate system can then be obtained through an optimization or solving a linear system. figure illustrates how a manifold can be tiled by a 14. 9 large number of locally linear gaussian - like patches ( or “ pancakes, ” because the gaussians are ﬂat in the tangent directions ). however, there is a fundamental [UNK] with such local non - parametric approaches to manifold learning, raised in ( ) : if the bengio and monperrus 2005 manifolds are not very smooth ( they have many peaks and troughs and twists ), one may need a very large number of training examples to cover each one of 519


chapter 14. autoencoders figure 14. 9 : if the tangent planes ( see ﬁgure ) at each location are known, then they 14. 6 can be tiled to form a global coordinate system or a density function. each local patch can be thought of as a local euclidean coordinate system or as a locally ﬂat gaussian, or “ pancake, ” with a very small variance in the directions orthogonal to the pancake and a very large variance in the directions deﬁning the coordinate system on the pancake. a mixture of these gaussians provides an estimated density function, as in the manifold parzen window algorithm (, ) or its non - local neural - net based vincent and bengio 2003 variant (, ). bengio et al. 2006c these variations, with no chance to generalize to unseen variations. indeed, these methods can only generalize the shape of the manifold by interpolating between neighboring examples. unfortunately, the manifolds involved in ai problems can have very complicated structure that can be [UNK] to capture from only local interpolation. consider for example the manifold resulting from translation shown in ﬁgure. if we watch just one coordinate within the input vector, 14. 6 xi, as the image is translated, we


that can be [UNK] to capture from only local interpolation. consider for example the manifold resulting from translation shown in ﬁgure. if we watch just one coordinate within the input vector, 14. 6 xi, as the image is translated, we will observe that one coordinate encounters a peak or a trough in its value once for every peak or trough in brightness in the image. in other words, the complexity of the patterns of brightness in an underlying image template drives the complexity of the manifolds that are generated by performing simple image transformations. this motivates the use of distributed representations and deep learning for capturing manifold structure. 520


chapter 14. autoencoders 14. 7 contractive autoencoders the contractive autoencoder (,, ) introduces an explicit regularizer rifai et al. 2011a b on the code h = f ( x ), encouraging the derivatives of f to be as small as possible : ω ( ) = h λ ∂f ( ) x ∂x 2 f. ( 14. 18 ) the penalty ω ( h ) is the squared frobenius norm ( sum of squared elements ) of the jacobian matrix of partial derivatives associated with the encoder function. there is a connection between the denoising autoencoder and the contractive autoencoder : ( ) showed that in the limit of small gaussian alain and bengio 2013 input noise, the denoising reconstruction error is equivalent to a contractive penalty on the reconstruction function that maps x to r = g ( f ( x ) ). in other words, denoising autoencoders make the reconstruction function resist small but ﬁnite - sized perturbations of the input, while contractive autoencoders make the feature extraction function resist inﬁnitesimal perturbations of the input. when using the jacobian -


function resist small but ﬁnite - sized perturbations of the input, while contractive autoencoders make the feature extraction function resist inﬁnitesimal perturbations of the input. when using the jacobian - based contractive penalty to pretrain features f ( x ) for use with a classiﬁer, the best classiﬁcation accuracy usually results from applying the contractive penalty to f ( x ) rather than to g ( f ( x ) ). a contractive penalty on f ( x ) also has close connections to score matching, as discussed in section. 14. 5. 1 the name contractive arises from the way that the cae warps space. speciﬁ - cally, because the cae is trained to resist perturbations of its input, it is encouraged to map a neighborhood of input points to a smaller neighborhood of output points. we can think of this as contracting the input neighborhood to a smaller output neighborhood. to clarify, the cae is contractive only locally — all perturbations of a training point x are mapped near to f ( x ). globally, two [UNK] points x and xmay be mapped to f ( x ) and f ( x ) points that


cae is contractive only locally — all perturbations of a training point x are mapped near to f ( x ). globally, two [UNK] points x and xmay be mapped to f ( x ) and f ( x ) points that are farther apart than the original points. it is plausible that f be expanding in - between or far from the data manifolds ( see for example what happens in the 1 - d toy example of ﬁgure ). when the 14. 7 ω ( h ) penalty is applied to sigmoidal units, one easy way to shrink the jacobian is to make the sigmoid units saturate to or. this encourages the cae to encode 0 1 input points with extreme values of the sigmoid that may be interpreted as a binary code. it also ensures that the cae will spread its code values throughout most of the hypercube that its sigmoidal hidden units can span. we can think of the jacobian matrix j at a point x as approximating the nonlinear encoder f ( x ) as being a linear operator. this allows us to use the word “ contractive ” more formally. in the theory of linear operators, a linear operator 521


chapter 14. autoencoders is said to be contractive if the norm of jx remains less than or equal to for 1 all unit - norm x. in other words, j is contractive if it shrinks the unit sphere. we can think of the cae as penalizing the frobenius norm of the local linear approximation of f ( x ) at every training point x in order to encourage each of these local linear operator to become a contraction. as described in section, regularized autoencoders learn manifolds by 14. 6 balancing two opposing forces. in the case of the cae, these two forces are reconstruction error and the contractive penalty ω ( h ). reconstruction error alone would encourage the cae to learn an identity function. the contractive penalty alone would encourage the cae to learn features that are constant with respect tox. the compromise between these two forces yields an autoencoder whose derivatives ∂f ( ) x ∂x are mostly tiny. only a small number of hidden units, corresponding to a small number of directions in the input, may have signiﬁcant derivatives. the goal of the cae is to learn the manifold structure of the data. directions x with large jx rapidly change h, so


hidden units, corresponding to a small number of directions in the input, may have signiﬁcant derivatives. the goal of the cae is to learn the manifold structure of the data. directions x with large jx rapidly change h, so these are likely to be directions which approximate the tangent planes of the manifold. experiments by ( ) rifai et al. 2011a and ( ) show that training the cae results in most singular values rifai et al. 2011b of j dropping below in magnitude and therefore becoming contractive. however, 1 some singular values remain above, because the reconstruction error penalty 1 encourages the cae to encode the directions with the most local variance. the directions corresponding to the largest singular values are interpreted as the tangent directions that the contractive autoencoder has learned. ideally, these tangent directions should correspond to real variations in the data. for example, a cae applied to images should learn tangent vectors that show how the image changes as objects in the image gradually change pose, as shown in ﬁgure. visualizations 14. 6 of the experimentally obtained singular vectors do seem to correspond to meaningful transformations of the input image, as shown in ﬁgure. 14. 10 one practical issue with the cae


pose, as shown in ﬁgure. visualizations 14. 6 of the experimentally obtained singular vectors do seem to correspond to meaningful transformations of the input image, as shown in ﬁgure. 14. 10 one practical issue with the cae regularization criterion is that although it is cheap to compute in the case of a single hidden layer autoencoder, it becomes much more expensive in the case of deeper autoencoders. the strategy followed by rifai 2011a et al. ( ) is to separately train a series of single - layer autoencoders, each trained to reconstruct the previous autoencoder ’ s hidden layer. the composition of these autoencoders then forms a deep autoencoder. because each layer was separately trained to be locally contractive, the deep autoencoder is contractive as well. the result is not the same as what would be obtained by jointly training the entire architecture with a penalty on the jacobian of the deep model, but it captures many of the desirable qualitative characteristics. another practical issue is that the contraction penalty can obtain useless results 522


chapter 14. autoencoders input point tangent vectors local pca ( no sharing across regions ) contractive autoencoder figure 14. 10 : illustration of tangent vectors of the manifold estimated by local pca and by a contractive autoencoder. the location on the manifold is deﬁned by the input image of a dog drawn from the cifar - 10 dataset. the tangent vectors are estimated by the leading singular vectors of the jacobian matrix ∂h ∂x of the input - to - code mapping. although both local pca and the cae can capture local tangents, the cae is able to form more accurate estimates from limited training data because it exploits parameter sharing across [UNK] locations that share a subset of active hidden units. the cae tangent directions typically correspond to moving or changing parts of the object ( such as the head or legs ). images reproduced with permission from ( ). rifai et al. 2011c if we do not impose some sort of scale on the decoder. for example, the encoder could consist of multiplying the input by a small constant and the decoder could consist of dividing the code by. as approaches, the encoder drives the 0 contractive penalty ω ( h ) to


. for example, the encoder could consist of multiplying the input by a small constant and the decoder could consist of dividing the code by. as approaches, the encoder drives the 0 contractive penalty ω ( h ) to approach without having learned anything about the 0 distribution. meanwhile, the decoder maintains perfect reconstruction. in rifai et al. ( ), this is prevented by tying the weights of 2011a f and g. both f and g are standard neural network layers consisting of an [UNK] transformation followed by an element - wise nonlinearity, so it is straightforward to set the weight matrix of g to be the transpose of the weight matrix of. f 14. 8 predictive sparse decomposition predictive sparse decomposition ( psd ) is a model that is a hybrid of sparse coding and parametric autoencoders ( kavukcuoglu 2008 et al., ). a parametric encoder is trained to predict the output of iterative inference. psd has been applied to unsupervised feature learning for object recognition in images and video ( kavukcuoglu 2009 2010 jarrett 2009 farabet 2011 et al.,, ; et al., ; et al., ), as well


applied to unsupervised feature learning for object recognition in images and video ( kavukcuoglu 2009 2010 jarrett 2009 farabet 2011 et al.,, ; et al., ; et al., ), as well as for audio (, ). the model consists of an encoder [UNK] al. 2011 f ( x ) and a decoder g ( h ) that are both parametric. during training, h is controlled by the 523


chapter 14. autoencoders optimization algorithm. training proceeds by minimizing | | − | | x g ( ) h 2 + λ | | h 1 + ( ) γ f | | − h x | | 2. ( 14. 19 ) like in sparse coding, the training algorithm alternates between minimization with respect to h and minimization with respect to the model parameters. minimization with respect to h is fast because f ( x ) provides a good initial value of h and the cost function constrains h to remain near f ( x ) anyway. simple gradient descent can obtain reasonable values of in as few as ten steps. h the training procedure used by psd is [UNK] from ﬁrst training a sparse coding model and then training f ( x ) to predict the values of the sparse coding features. the psd training procedure regularizes the decoder to use parameters for which can infer good code values. f ( ) x predictive sparse coding is an example of learned approximate inference. in section, this topic is developed further. the tools presented in chapter 19. 5 19 make it clear that psd can be interpreted as training a directed sparse coding probabilistic model by maximizing a lower bound on the log - likelihood


. in section, this topic is developed further. the tools presented in chapter 19. 5 19 make it clear that psd can be interpreted as training a directed sparse coding probabilistic model by maximizing a lower bound on the log - likelihood of the model. in practical applications of psd, the iterative optimization is only used during training. the parametric encoder f is used to compute the learned features when the model is deployed. evaluating f is computationally inexpensive compared to inferring h via gradient descent. because f is a [UNK] parametric function, psd models may be stacked and used to initialize a deep network to be trained with another criterion. 14. 9 applications of autoencoders autoencoders have been successfully applied to dimensionality reduction and infor - mation retrieval tasks. dimensionality reduction was one of the ﬁrst applications of representation learning and deep learning. it was one of the early motivations for studying autoencoders. for example, hinton and salakhutdinov 2006 ( ) trained a stack of rbms and then used their weights to initialize a deep autoencoder with gradually smaller hidden layers, culminating in a bottleneck of 30 units. the resulting code yielded less reconstruction error than


##akhutdinov 2006 ( ) trained a stack of rbms and then used their weights to initialize a deep autoencoder with gradually smaller hidden layers, culminating in a bottleneck of 30 units. the resulting code yielded less reconstruction error than pca into 30 dimensions and the learned representation was qualitatively easier to interpret and relate to the underlying categories, with these categories manifesting as well - separated clusters. lower - dimensional representations can improve performance on many tasks, such as classiﬁcation. models of smaller spaces consume less memory and runtime. 524