[UNK] sparsity patterns in the weight matrix u, we can implement structures of boltzmann machines, such as rbms, or dbms with [UNK] numbers of layers. this is accomplished by partitioning x into visible and hidden units and zeroing out elements of u for units that do not interact. the centered boltzmann machine introduces a vector that is subtracted from all of the states : µ e ( ; ) = ( ) x u b, −x µ − u x µ x µ ( − ) ( − − ) b. ( 20. 37 ) typically µ is a hyperparameter ﬁxed at the beginning of training. it is usu - ally chosen to make sure that x µ − ≈0 when the model is initialized. this reparametrization does not change the set of probability distributions that the model can represent, but it does change the dynamics of stochastic gradient descent applied to the likelihood. speciﬁcally, in many cases, this reparametrization results 673


chapter 20. deep generative models in a hessian matrix that is better conditioned. ( ) experimentally melchior et al. 2013 conﬁrmed that the conditioning of the hessian matrix improves, and observed that the centering trick is equivalent to another boltzmann machine learning technique, the enhanced gradient (, ). the improved conditioning of the cho et al. 2011 hessian matrix allows learning to succeed, even in [UNK] cases like training a deep boltzmann machine with multiple layers. the other approach to jointly training deep boltzmann machines is the multi - prediction deep boltzmann machine ( mp - dbm ) which works by viewing the mean ﬁeld equations as deﬁning a family of recurrent networks for approximately solving every possible inference problem (, ). rather than training goodfellow et al. 2013b the model to maximize the likelihood, the model is trained to make each recurrent network obtain an accurate answer to the corresponding inference problem. the training process is illustrated in ﬁgure. it consists of randomly sampling a 20. 5 training example, randomly sampling a subset of inputs to the inference network, and then training the inference network to predict the values of the remaining units. this general principle of back - propagating through the computational graph

/home/ricoiban/GEMMA/mnlp_chatsplaining/RAG/Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville (z-lib.org).pdf


consists of randomly sampling a 20. 5 training example, randomly sampling a subset of inputs to the inference network, and then training the inference network to predict the values of the remaining units. this general principle of back - propagating through the computational graph for approximate inference has been applied to other models ( stoyanov 2011 et al., ; brakel 2013 et al., ). in these models and in the mp - dbm, the ﬁnal loss is not the lower bound on the likelihood. instead, the ﬁnal loss is typically based on the approximate conditional distribution that the approximate inference network imposes over the missing values. this means that the training of these models is somewhat heuristically motivated. if we inspect the p ( v ) represented by the boltzmann machine learned by the mp - dbm, it tends to be somewhat defective, in the sense that gibbs sampling yields poor samples. back - propagation through the inference graph has two main advantages. first, it trains the model as it is really used — with approximate inference. this means that approximate inference, for example, to ﬁll in missing inputs, or to perform classiﬁcation despite the presence of missing inputs, is more accurate in the mp - dbm than in the original db


used — with approximate inference. this means that approximate inference, for example, to ﬁll in missing inputs, or to perform classiﬁcation despite the presence of missing inputs, is more accurate in the mp - dbm than in the original dbm. the original dbm does not make an accurate classiﬁer on its own ; the best classiﬁcation results with the original dbm were based on training a separate classiﬁer to use features extracted by the dbm, rather than by using inference in the dbm to compute the distribution over the class labels. mean ﬁeld inference in the mp - dbm performs well as a classiﬁer without special modiﬁcations. the other advantage of back - propagating through approximate inference is that back - propagation computes the exact gradient of the loss. this is better for optimization than the approximate gradients of sml training, which [UNK] from both bias and variance. this probably explains why mp - 674


chapter 20. deep generative models figure 20. 5 : an illustration of the multi - prediction training process for a deep boltzmann machine. each row indicates a [UNK] example within a minibatch for the same training step. each column represents a time step within the mean ﬁeld inference process. for each example, we sample a subset of the data variables to serve as inputs to the inference process. these variables are shaded black to indicate conditioning. we then run the mean ﬁeld inference process, with arrows indicating which variables inﬂuence which other variables in the process. in practical applications, we unroll mean ﬁeld for several steps. in this illustration, we unroll for only two steps. dashed arrows indicate how the process could be unrolled for more steps. the data variables that were not used as inputs to the inference process become targets, shaded in gray. we can view the inference process for each example as a recurrent network. we use gradient descent and back - propagation to train these recurrent networks to produce the correct targets given their inputs. this trains the mean ﬁeld process for the mp - dbm to produce accurate estimates. figure adapted from ( ). goodfellow et al. 2013b 675


chapter 20. deep generative models dbms may be trained jointly while dbms require a greedy layer - wise pretraining. the disadvantage of back - propagating through the approximate inference graph is that it does not provide a way to optimize the log - likelihood, but rather a heuristic approximation of the generalized pseudolikelihood. the mp - dbm inspired the nade - k ( raiko 2014 et al., ) extension to the nade framework, which is described in section. 20. 10. 10 the mp - dbm has some connections to dropout. dropout shares the same pa - rameters among many [UNK] computational graphs, with the [UNK] between each graph being whether it includes or excludes each unit. the mp - dbm also shares parameters across many computational graphs. in the case of the mp - dbm, the [UNK] between the graphs is whether each input unit is observed or not. when a unit is not observed, the mp - dbm does not delete it entirely as dropout does. instead, the mp - dbm treats it as a latent variable to be inferred. one could imagine applying dropout to the mp - dbm by additionally removing some units rather than making them late


##ete it entirely as dropout does. instead, the mp - dbm treats it as a latent variable to be inferred. one could imagine applying dropout to the mp - dbm by additionally removing some units rather than making them latent. 20. 5 boltzmann machines for real - valued data while boltzmann machines were originally developed for use with binary data, many applications such as image and audio modeling seem to require the ability to represent probability distributions over real values. in some cases, it is possible to treat real - valued data in the interval [ 0, 1 ] as representing the expectation of a binary variable. for example, ( ) treats grayscale images in the training hinton 2000 set as deﬁning [ 0, 1 ] probability values. each pixel deﬁnes the probability of a binary value being 1, and the binary pixels are all sampled independently from each other. this is a common procedure for evaluating binary models on grayscale image datasets. however, it is not a particularly theoretically satisfying approach, and binary images sampled independently in this way have a noisy appearance. in this section, we present boltzmann machines that deﬁne a probability density over real - valued data. 20. 5. 1 gauss


not a particularly theoretically satisfying approach, and binary images sampled independently in this way have a noisy appearance. in this section, we present boltzmann machines that deﬁne a probability density over real - valued data. 20. 5. 1 gaussian - bernoulli rbms restricted boltzmann machines may be developed for many exponential family conditional distributions ( welling 2005 et al., ). of these, the most common is the rbm with binary hidden units and real - valued visible units, with the conditional distribution over the visible units being a gaussian distribution whose mean is a function of the hidden units. 676


chapter 20. deep generative models there are many ways of parametrizing gaussian - bernoulli rbms. one choice is whether to use a covariance matrix or a precision matrix for the gaussian distribution. here we present the precision formulation. the modiﬁcation to obtain the covariance formulation is straightforward. we wish to have the conditional distribution p, ( ) = ( ; v h | n v w h β−1 ). ( 20. 38 ) we can ﬁnd the terms we need to add to the energy function by expanding the unnormalized log conditional distribution : log ( ; n v w h β, −1 ) = −1 2 ( ) v w h − β v w h β ( − ) + ( f ). ( 20. 39 ) here f encapsulates all the terms that are a function only of the parameters and not the random variables in the model. we can discard f because its only role is to normalize the distribution, and the partition function of whatever energy function we choose will carry out that role. if we include all of the terms ( with their sign ﬂipped ) involving v from equa - tion in our energy function and do not add any other


, and the partition function of whatever energy function we choose will carry out that role. if we include all of the terms ( with their sign ﬂipped ) involving v from equa - tion in our energy function and do not add any other terms involving 20. 39 v, then our energy function will represent the desired conditional. p ( ) v h | we have some freedom regarding the other conditional distribution, p ( h v | ). note that equation contains a term 20. 39 1 2hw βw h. ( 20. 40 ) this term cannot be included in its entirety because it includes hihj terms. these correspond to edges between the hidden units. if we included these terms, we would have a linear factor model instead of a restricted boltzmann machine. when designing our boltzmann machine, we simply omit thesehihj cross terms. omitting them does not change the conditional p ( v h | ) so equation is still respected. 20. 39 however, we still have a choice about whether to include the terms involving only a single hi. if we assume a diagonal precision matrix, we ﬁnd that for each hidden unit hi we have a term 1 2hi j βjw 2 j, i. ( 20


have a choice about whether to include the terms involving only a single hi. if we assume a diagonal precision matrix, we ﬁnd that for each hidden unit hi we have a term 1 2hi j βjw 2 j, i. ( 20. 41 ) in the above, we used the fact that h2 i = hi because hi ∈ { 0, 1 }. if we include this term ( with its sign ﬂipped ) in the energy function, then it will naturally bias hi to be turned [UNK] the weights for that unit are large and connected to visible units with high precision. the choice of whether or not to include this bias term does not [UNK] the family of distributions the model can represent ( assuming that 677


chapter 20. deep generative models we include bias parameters for the hidden units ) but it does [UNK] the learning dynamics of the model. including the term may help the hidden unit activations remain reasonable even when the weights rapidly increase in magnitude. one way to deﬁne the energy function on a gaussian - bernoulli rbm is thus e, ( v h ) = 1 2v ( ) ( ) β v −v β wh b − h ( 20. 42 ) but we may also add extra terms or parametrize the energy in terms of the variance rather than precision if we choose. in this derivation, we have not included a bias term on the visible units, but one could easily be added. one ﬁnal source of variability in the parametrization of a gaussian - bernoulli rbm is the choice of how to treat the precision matrix. it may either be ﬁxed to a constant ( perhaps estimated based on the marginal precision of the data ) or learned. it may also be a scalar times the identity matrix, or it may be a diagonal matrix. typically we do not allow the precision matrix to be non - diagonal in this context, because some operations on the gaussian distribution require inverting the matrix


also be a scalar times the identity matrix, or it may be a diagonal matrix. typically we do not allow the precision matrix to be non - diagonal in this context, because some operations on the gaussian distribution require inverting the matrix, and a diagonal matrix can be inverted trivially. in the sections ahead, we will see that other forms of boltzmann machines permit modeling the covariance structure, using various techniques to avoid inverting the precision matrix. 20. 5. 2 undirected models of conditional covariance while the gaussian rbm has been the canonical energy model for real - valued data, ( ) argue that the gaussian rbm inductive bias is not ranzato et al. 2010a well suited to the statistical variations present in some types of real - valued data, especially natural images. the problem is that much of the information content present in natural images is embedded in the covariance between pixels rather than in the raw pixel values. in other words, it is the relationships between pixels and not their absolute values where most of the useful information in images resides. since the gaussian rbm only models the conditional mean of the input given the hidden units, it cannot capture conditional covariance


words, it is the relationships between pixels and not their absolute values where most of the useful information in images resides. since the gaussian rbm only models the conditional mean of the input given the hidden units, it cannot capture conditional covariance information. in response to these criticisms, alternative models have been proposed that attempt to better account for the covariance of real - valued data. these models include the mean and covariance rbm ( mcrbm1 ), the mean - product of t - distribution ( mpot ) model and the spike and slab rbm ( ssrbm ). 1the term “ mcrbm ” is pronounced by saying the name of the letters m - c - r - b - m ; the “ mc ” is not pronounced like the “ mc ” in “ mcdonald ’ s. ” 678


chapter 20. deep generative models mean and covariance rbm the mcrbm uses its hidden units to indepen - dently encode the conditional mean and covariance of all observed units. the mcrbm hidden layer is divided into two groups of units : mean units and covariance units. the group that models the conditional mean is simply a gaussian rbm. the other half is a covariance rbm (, ), also called a crbm, ranzato et al. 2010a whose components model the conditional covariance structure, as described below. speciﬁcally, with binary mean units h ( ) m and binary covariance units h ( ) c, the mcrbm model is deﬁned as the combination of two energy functions : emc ( x h, ( ) m, h ( ) c ) = em ( x h, ( ) m ) + ec ( x h, ( ) c ), ( 20. 43 ) where em is the standard gaussian - bernoulli rbm energy function : 2 em ( x h, ( ) m ) = 1 2 xx − j xw :, jh ( ) m j − j b ( ) m j


where em is the standard gaussian - bernoulli rbm energy function : 2 em ( x h, ( ) m ) = 1 2 xx − j xw :, jh ( ) m j − j b ( ) m j h ( ) m j, ( 20. 44 ) and ec is the crbm energy function that models the conditional covariance information : ec ( x h, ( ) c ) = 1 2 j h ( ) c j xr ( ) j 2 − j b ( ) c j h ( ) c j. ( 20. 45 ) the parameter r ( ) j corresponds to the covariance weight vector associated with h ( ) c j and b ( ) c is a vector of covariance [UNK]. the combined energy function deﬁnes a joint distribution : pmc ( x h, ( ) m, h ( ) c ) = 1 z exp −emc ( x h, ( ) m, h ( ) c ), ( 20. 46 ) and a corresponding conditional distribution over the observations given h ( ) m and h ( ) c as a multivariate gaussian distribution : pmc ( x h | ( ) m, h ( ) c ) = n x c ;


46 ) and a corresponding conditional distribution over the observations given h ( ) m and h ( ) c as a multivariate gaussian distribution : pmc ( x h | ( ) m, h ( ) c ) = n x c ; mc x h | j w :, jh ( ) m j, c mc x h |. ( 20. 47 ) note that the covariance matrix cmc x h | = j h ( ) c j r ( ) j r ( ) j + i −1 is non - diagonal and that w is the weight matrix associated with the gaussian rbm modeling the 2this version of the gaussian - bernoulli rbm energy function assumes the image data has zero mean, per pixel. pixel [UNK] can easily be added to the model to account for nonzero pixel means. 679


chapter 20. deep generative models conditional means. it is [UNK] to train the mcrbm via contrastive divergence or persistent contrastive divergence because of its non - diagonal conditional covariance structure. cd and pcd require sampling from the joint distribution of x h, ( ) m, h ( ) c which, in a standard rbm, is accomplished by gibbs sampling over the conditionals. however, in the mcrbm, sampling from pmc ( x h | ( ) m, h ( ) c ) requires computing ( cmc ) −1 at every iteration of learning. this can be an impractical computational burden for larger observations. ( ) avoid direct sampling ranzato and hinton 2010 from the conditional pmc ( x h | ( ) m, h ( ) c ) by sampling directly from the marginal p ( x ) using hamiltonian ( hybrid ) monte carlo (, ) on the mcrbm free neal 1993 energy. mean - product of student ’ s - distributions t the mean - product of student ’ s t - distribution ( mpot ) model (, ) extends the pot model ( ranzato et al. 2010b welling et al., ) in a manner similar to how the mcrbm extends the


mean - product of student ’ s t - distribution ( mpot ) model (, ) extends the pot model ( ranzato et al. 2010b welling et al., ) in a manner similar to how the mcrbm extends the crbm. this 2003a is achieved by including nonzero gaussian means by the addition of gaussian rbm - like hidden units. like the mcrbm, the pot conditional distribution over the observation is a multivariate gaussian ( with non - diagonal covariance ) distribution ; however, unlike the mcrbm, the complementary conditional distribution over the hidden variables is given by conditionally independent gamma distributions. the gamma distribution g ( k, θ ) is a probability distribution over positive real numbers, with mean kθ. it is not necessary to have a more detailed understanding of the gamma distribution to understand the basic ideas underlying the mpot model. the mpot energy function is : empot ( x h, ( ) m, h ( ) c ) ( 20. 48 ) = em ( x h, ( ) m ) + j h ( ) c j 1 + 1 2 r ( ) j x 2 + ( 1 −γj ) log h ( ) c j ( 20.


c ) ( 20. 48 ) = em ( x h, ( ) m ) + j h ( ) c j 1 + 1 2 r ( ) j x 2 + ( 1 −γj ) log h ( ) c j ( 20. 49 ) where r ( ) j is the covariance weight vector associated with unith ( ) c j and em ( x h, ( ) m ) is as deﬁned in equation. 20. 44 just as with the mcrbm, the mpot model energy function speciﬁes a mul - tivariate gaussian, with a conditional distribution over x that has non - diagonal covariance. learning in the mpot model — again, like the mcrbm — is compli - cated by the inability to sample from the non - diagonal gaussian conditional pmpot ( x h | ( ) m, h ( ) c ), so ( ) also advocate direct sampling of ranzato et al. 2010b p ( ) x via hamiltonian ( hybrid ) monte carlo. 680


chapter 20. deep generative models spike and slab restricted boltzmann machines spike and slab restricted boltzmann machines (, ) or ssrbms provide another means courville et al. 2011 of modeling the covariance structure of real - valued data. compared to mcrbms, ssrbms have the advantage of requiring neither matrix inversion nor hamiltonian monte carlo methods. like the mcrbm and the mpot model, the ssrbm ’ s binary hidden units encode the conditional covariance across pixels through the use of auxiliary real - valued variables. the spike and slab rbm has two sets of hidden units : binary spike units h, and real - valued slab units s. the mean of the visible units conditioned on the hidden units is given by ( h s ) w. in other words, each column w :, i deﬁnes a component that can appear in the input when hi = 1. the corresponding spike variable hi determines whether that component is present at all. the corresponding slab variable si determines the intensity of that component, if it is present. when a spike variable is active, the corresponding slab variable adds variance to the input along the axis deﬁned by w :, i. this allows us to model the covariance of


determines the intensity of that component, if it is present. when a spike variable is active, the corresponding slab variable adds variance to the input along the axis deﬁned by w :, i. this allows us to model the covariance of the inputs. fortunately, contrastive divergence and persistent contrastive divergence with gibbs sampling are still applicable. there is no need to invert any matrix. formally, the ssrbm model is deﬁned via its energy function : ess ( ) = x s h,, − i xw :, isihi + 1 2x λ + i φihi x ( 20. 50 ) + 1 2 i αis2 i − i αiµisih i − i bih i + i αiµ2 ihi, ( 20. 51 ) where bi is the [UNK] of the spike hi and λ is a diagonal precision matrix on the observations x. the parameter αi > 0 is a scalar precision parameter for the real - valued slab variable si. the parameter φi is a non - negative diagonal matrix that deﬁnes an h - modulated quadratic penalty on x. each µi is a mean parameter for the slab variable si. with the joint distribution deﬁned via


variable si. the parameter φi is a non - negative diagonal matrix that deﬁnes an h - modulated quadratic penalty on x. each µi is a mean parameter for the slab variable si. with the joint distribution deﬁned via the energy function, it is relatively straightforward to derive the ssrbm conditional distributions. for example, by marginalizing out the slab variables s, the conditional distribution over the observations given the binary spike variables is given by : h pss ( ) = x h | 1 p ( ) h 1 z exp ( ) { −e x s h,, } ds ( 20. 52 ) = n x c ; ss x h | i w :, iµih i, css x h | ( 20. 53 ) 681


chapter 20. deep generative models where c ss x h | = λ + iφihi − i α−1 i hiw :, iw :, i −1. the last equality holds only if the covariance matrix css x h | is positive deﬁnite. gating by the spike variables means that the true marginal distribution over h s is sparse. this is [UNK] from sparse coding, where samples from the model “ almost never ” ( in the measure theoretic sense ) contain zeros in the code, and map inference is required to impose sparsity. comparing the ssrbm to the mcrbm and the mpot models, the ssrbm parametrizes the conditional covariance of the observation in a signiﬁcantly [UNK] way. the mcrbm and mpot both model the covariance structure of the observation as j h ( ) c j r ( ) j r ( ) j + i −1, using the activation of the hidden units hj > 0 to enforce constraints on the conditional covariance in the direction r ( ) j. in contrast, the ssrbm speciﬁes the conditional covariance of the observations using the hidden spike activations hi = 1 to pinch the


##j > 0 to enforce constraints on the conditional covariance in the direction r ( ) j. in contrast, the ssrbm speciﬁes the conditional covariance of the observations using the hidden spike activations hi = 1 to pinch the precision matrix along the direction speciﬁed by the corresponding weight vector. the ssrbm conditional covariance is very similar to that given by a [UNK] model : the product of probabilistic principal components analysis ( poppca ) ( williams and agakov 2002, ). in the overcomplete setting, sparse activations with the ssrbm parametrization permit signiﬁcant variance ( above the nominal variance given by λ−1 ) only in the selected directions of the sparsely activated hi. in the mcrbm or mpot models, an overcomplete representation would mean that to capture variation in a particular direction in the observation space requires removing potentially all constraints with positive projection in that direction. this would suggest that these models are less well suited to the overcomplete setting. the primary disadvantage of the spike and slab restricted boltzmann machine is that some settings of the parameters can correspond to a covariance matrix that is not positive deﬁnite. such a covariance


to the overcomplete setting. the primary disadvantage of the spike and slab restricted boltzmann machine is that some settings of the parameters can correspond to a covariance matrix that is not positive deﬁnite. such a covariance matrix places more unnormalized probability on values that are farther from the mean, causing the integral over all possible outcomes to diverge. generally this issue can be avoided with simple heuristic tricks. there is not yet any theoretically satisfying solution. using constrained optimization to explicitly avoid the regions where the probability is undeﬁned is [UNK] to do without being overly conservative and also preventing the model from accessing high - performing regions of parameter space. qualitatively, convolutional variants of the ssrbm produce excellent samples of natural images. some examples are shown in ﬁgure. 16. 1 the ssrbm allows for several extensions. including higher - order interactions and average - pooling of the slab variables (, ) enables the model courville et al. 2014 to learn excellent features for a classiﬁer when labeled data is scarce. adding a 682


chapter 20. deep generative models term to the energy function that prevents the partition function from becoming undeﬁned results in a sparse coding model, spike and slab sparse coding ( goodfellow et al., ), also known as s3c. 2013d 20. 6 convolutional boltzmann machines as seen in chapter, extremely high dimensional inputs such as images place 9 great strain on the computation, memory and statistical requirements of machine learning models. replacing matrix multiplication by discrete convolution with a small kernel is the standard way of solving these problems for inputs that have translation invariant spatial or temporal structure. ( ) desjardins and bengio 2008 showed that this approach works well when applied to rbms. deep convolutional networks usually require a pooling operation so that the spatial size of each successive layer decreases. feedforward convolutional networks often use a pooling function such as the maximum of the elements to be pooled. it is unclear how to generalize this to the setting of energy - based models. we could introduce a binary pooling unit p over n binary detector units d and enforce p = maxi di by setting the energy function to be ∞whenever that constraint is violated. this does


##ize this to the setting of energy - based models. we could introduce a binary pooling unit p over n binary detector units d and enforce p = maxi di by setting the energy function to be ∞whenever that constraint is violated. this does not scale well though, as it requires evaluating 2n [UNK] energy conﬁgurations to compute the normalization constant. for a small 3 × 3 pooling region this requires 29 = 512 energy function evaluations per pooling unit! lee 2009 et al. ( ) developed a solution to this problem called probabilistic max pooling ( not to be confused with “ stochastic pooling, ” which is a technique for implicitly constructing ensembles of convolutional feedforward networks ). the strategy behind probabilistic max pooling is to constrain the detector units so at most one may be active at a time. this means there are only n + 1 total states ( one state for each of the n detector units being on, and an additional state corresponding to all of the detector units being [UNK] ). the pooling unit is on if and only if one of the detector units is on. the state with all units [UNK] assigned energy zero. we can think of this as describing a


an additional state corresponding to all of the detector units being [UNK] ). the pooling unit is on if and only if one of the detector units is on. the state with all units [UNK] assigned energy zero. we can think of this as describing a model with a single variable that has n + 1 states, or equivalently as a model that has n + 1 variables that assigns energy to all but joint assignments of variables. ∞ n + 1 while [UNK], probabilistic max pooling does force the detector units to be mutually exclusive, which may be a useful regularizing constraint in some contexts or a harmful limit on model capacity in other contexts. it also does not support overlapping pooling regions. overlapping pooling regions are usually required to obtain the best performance from feedforward convolutional networks, so this constraint probably greatly reduces the performance of convolutional boltzmann 683


chapter 20. deep generative models machines. lee 2009 et al. ( ) demonstrated that probabilistic max pooling could be used to build convolutional deep boltzmann machines. 3 this model is able to perform operations such as ﬁlling in missing portions of its input. while intellectually appealing, this model is challenging to make work in practice, and usually does not perform as well as a classiﬁer as traditional convolutional networks trained with supervised learning. many convolutional models work equally well with inputs of many [UNK] spatial sizes. for boltzmann machines, it is [UNK] to change the input size for a variety of reasons. the partition function changes as the size of the input changes. moreover, many convolutional networks achieve size invariance by scaling up the size of their pooling regions proportional to the size of the input, but scaling boltzmann machine pooling regions is awkward. traditional convolutional neural networks can use a ﬁxed number of pooling units and dynamically increase the size of their pooling regions in order to obtain a ﬁxed - size representation of a variable - sized input. for boltzmann machines, large pooling regions become too expensive for the


ﬁxed number of pooling units and dynamically increase the size of their pooling regions in order to obtain a ﬁxed - size representation of a variable - sized input. for boltzmann machines, large pooling regions become too expensive for the naive approach. the approach of ( ) of making lee et al. 2009 each of the detector units in the same pooling region mutually exclusive solves the computational problems, but still does not allow variable - size pooling regions. for example, suppose we learn a model with 2× 2 probabilistic max pooling over detector units that learn edge detectors. this enforces the constraint that only one of these edges may appear in each 2 × 2 region. if we then increase the size of the input image by 50 % in each direction, we would expect the number of edges to increase correspondingly. instead, if we increase the size of the pooling regions by 50 % in each direction to 3× 3, then the mutual exclusivity constraint now speciﬁes that each of these edges may only appear once in a 3 × 3 region. as we grow a model ’ s input image in this way, the model generates edges with less density. of course, these issues only arise when the model must


##ﬁes that each of these edges may only appear once in a 3 × 3 region. as we grow a model ’ s input image in this way, the model generates edges with less density. of course, these issues only arise when the model must use variable amounts of pooling in order to emit a ﬁxed - size output vector. models that use probabilistic max pooling may still accept variable - sized input images so long as the output of the model is a feature map that can scale in size proportional to the input image. pixels at the boundary of the image also pose some [UNK], which is exac - erbated by the fact that connections in a boltzmann machine are symmetric. if we do not implicitly zero - pad the input, then there are fewer hidden units than visible units, and the visible units at the boundary of the image are not modeled 3the publication describes the model as a “ deep belief network ” but because it can be described as a purely undirected model with tractable layer - wise mean ﬁeld ﬁxed point updates, it best ﬁts the deﬁnition of a deep boltzmann machine. 684


chapter 20. deep generative models well because they lie in the receptive ﬁeld of fewer hidden units. however, if we do implicitly zero - pad the input, then the hidden units at the boundary are driven by fewer input pixels, and may fail to activate when needed. 20. 7 boltzmann machines for structured or sequential outputs in the structured output scenario, we wish to train a model that can map from some input x to some output y, and the [UNK] entries of y are related to each other and must obey some constraints. for example, in the speech synthesis task, y is a waveform, and the entire waveform must sound like a coherent utterance. a natural way to represent the relationships between the entries in y is to use a probability distribution p ( y | x ). boltzmann machines, extended to model conditional distributions, can supply this probabilistic model. the same tool of conditional modeling with a boltzmann machine can be used not just for structured output tasks, but also for sequence modeling. in the latter case, rather than mapping an input x to an output y, the model must estimate a probability distribution over a sequence of variables, p ( x ( 1 ),..., x ( ) τ )


also for sequence modeling. in the latter case, rather than mapping an input x to an output y, the model must estimate a probability distribution over a sequence of variables, p ( x ( 1 ),..., x ( ) τ ). conditional boltzmann machines can represent factors of the form p ( x ( ) t | x ( 1 ),..., x ( 1 ) t− ) in order to accomplish this task. an important sequence modeling task for the video game and ﬁlm industry is modeling sequences of joint angles of skeletons used to render 3 - d characters. these sequences are often collected using motion capture systems to record the movements of actors. a probabilistic model of a character ’ s movement allows the generation of new, previously unseen, but realistic animations. to solve this sequence modeling task, taylor 2007 et al. ( ) introduced a conditional rbm modeling p ( x ( ) t | x ( 1 ) t−,..., x ( ) t m − ) for small m. the model is an rbm over p ( x ( ) t ) whose bias parameters are a linear function of the preceding m values of x. when we condition on [UNK] values of x ( 1 ) t− and earlier


− ) for small m. the model is an rbm over p ( x ( ) t ) whose bias parameters are a linear function of the preceding m values of x. when we condition on [UNK] values of x ( 1 ) t− and earlier variables, we get a new rbm over x. the weights in the rbm over x never change, but by conditioning on [UNK] past values, we can change the probability of [UNK] hidden units in the rbm being active. by activating and deactivating [UNK] subsets of hidden units, we can make large changes to the probability distribution induced on x. other variants of conditional rbm (, ) and other variants of sequence mnih et al. 2011 modeling using conditional rbms are possible ( taylor and hinton 2009 sutskever, ; et al., ; 2009 boulanger - lewandowski 2012 et al., ). another sequence modeling task is to model the distribution over sequences 685


chapter 20. deep generative models of musical notes used to compose songs. boulanger - lewandowski 2012 et al. ( ) introduced the rnn - rbm sequence model and applied it to this task. the rnn - rbm is a generative model of a sequence of frames x ( ) t consisting of an rnn that emits the rbm parameters for each time step. unlike previous approaches in which only the bias parameters of the rbm varied from one time step to the next, the rnn - rbm uses the rnn to emit all of the parameters of the rbm, including the weights. to train the model, we need to be able to back - propagate the gradient of the loss function through the rnn. the loss function is not applied directly to the rnn outputs. instead, it is applied to the rbm. this means that we must approximately [UNK] the loss with respect to the rbm parameters using contrastive divergence or a related algorithm. this approximate gradient may then be back - propagated through the rnn using the usual back - propagation through time algorithm. 20. 8 other boltzmann machines many other variants of boltzmann machines are possible. boltzmann machines may be extended


approximate gradient may then be back - propagated through the rnn using the usual back - propagation through time algorithm. 20. 8 other boltzmann machines many other variants of boltzmann machines are possible. boltzmann machines may be extended with [UNK] training criteria. we have focused on boltzmann machines trained to approximately maximize the generative criterion log p ( v ). it is also possible to train discriminative rbms that aim to maximize log p ( y | v ) instead (, ). this approach often larochelle and bengio 2008 performs the best when using a linear combination of both the generative and the discriminative criteria. unfortunately, rbms do not seem to be as powerful supervised learners as mlps, at least using existing methodology. most boltzmann machines used in practice have only second - order interactions in their energy functions, meaning that their energy functions are the sum of many terms and each individual term only includes the product between two random variables. an example of such a term is viwi, jhj. it is also possible to train higher - order boltzmann machines (, ) whose energy function terms sejnowski 1987 involve the products between many variables. three - way interactions between a hidden unit


such a term is viwi, jhj. it is also possible to train higher - order boltzmann machines (, ) whose energy function terms sejnowski 1987 involve the products between many variables. three - way interactions between a hidden unit and two [UNK] images can model spatial transformations from one frame of video to the next ( memisevic and hinton 2007 2010,, ). multiplication by a one - hot class variable can change the relationship between visible and hidden units depending on which class is present (, ). one recent example nair and hinton 2009 of the use of higher - order interactions is a boltzmann machine with two groups of hidden units, with one group of hidden units that interact with both the visible units v and the class label y, and another group of hidden units that interact only with the v input values (, ). this can be interpreted as encouraging luo et al. 2011 686


chapter 20. deep generative models some hidden units to learn to model the input using features that are relevant to the class but also to learn extra hidden units that explain nuisance details that are necessary for the samples of v to be realistic but do not determine the class of the example. another use of higher - order interactions is to gate some features. sohn 2013 et al. ( ) introduced a boltzmann machine with third - order interactions with binary mask variables associated with each visible unit. when these masking variables are set to zero, they remove the inﬂuence of a visible unit on the hidden units. this allows visible units that are not relevant to the classiﬁcation problem to be removed from the inference pathway that estimates the class. more generally, the boltzmann machine framework is a rich space of models permitting many more model structures than have been explored so far. developing a new form of boltzmann machine requires some more care and creativity than developing a new neural network layer, because it is often [UNK] to ﬁnd an energy function that maintains tractability of all of the [UNK] conditional distributions needed to use the boltzmann machine, but despite this required [UNK] the ﬁeld remains open to innovation. 20. 9 back - propagation through random operations traditional


[UNK] to ﬁnd an energy function that maintains tractability of all of the [UNK] conditional distributions needed to use the boltzmann machine, but despite this required [UNK] the ﬁeld remains open to innovation. 20. 9 back - propagation through random operations traditional neural networks implement a deterministic transformation of some input variables x. when developing generative models, we often wish to extend neural networks to implement stochastic transformations of x. one straightforward way to do this is to augment the neural network with extra inputs z that are sampled from some simple probability distribution, such as a uniform or gaussian distribution. the neural network can then continue to perform deterministic computation internally, but the function f ( x z, ) will appear stochastic to an observer who does not have access to z. provided that f is continuous and [UNK], we can then compute the gradients necessary for training using back - propagation as usual. as an example, let us consider the operation consisting of drawing samples y from a gaussian distribution with mean and variance µ σ2 : y [UNK] ( µ, σ2 ). ( 20. 54 ) because an individual sample of y is not produced by a function, but rather by a sampling process whose output changes every time we query it,


with mean and variance µ σ2 : y [UNK] ( µ, σ2 ). ( 20. 54 ) because an individual sample of y is not produced by a function, but rather by a sampling process whose output changes every time we query it, it may seem counterintuitive to take the derivatives of y with respect to the parameters of its distribution, µ and σ2. however, we can rewrite the sampling process as 687


chapter 20. deep generative models transforming an underlying random value z [UNK] ( z ; 0, 1 ) to obtain a sample from the desired distribution : y µ σz = + ( 20. 55 ) we are now able to back - propagate through the sampling operation, by regard - ing it as a deterministic operation with an extra input z. crucially, the extra input is a random variable whose distribution is not a function of any of the variables whose derivatives we want to calculate. the result tells us how an inﬁnitesimal change in µ or σ would change the output if we could repeat the sampling operation again with the same value of z. being able to back - propagate through this sampling operation allows us to incorporate it into a larger graph. we can build elements of the graph on top of the output of the sampling distribution. for example, we can compute the derivatives of some loss function j ( y ). we can also build elements of the graph whose outputs are the inputs or the parameters of the sampling operation. for example, we could build a larger graph with µ = f ( x ; θ ) and σ = g ( x ; θ ). in this augmented graph, we can use back - propagation through these functions


or the parameters of the sampling operation. for example, we could build a larger graph with µ = f ( x ; θ ) and σ = g ( x ; θ ). in this augmented graph, we can use back - propagation through these functions to derive ∇θj y ( ). the principle used in this gaussian sampling example is more generally appli - cable. we can express any probability distribution of the form p ( y ; θ ) or p ( y | x ; θ ) as p ( y | ω ), where ω is a variable containing both parameters θ, and if applicable, the inputs x. given a value y sampled from distribution p ( y | ω ), where ω may in turn be a function of other variables, we can rewrite y y [UNK] ( | ω ) ( 20. 56 ) as y z ω = ( f ; ), ( 20. 57 ) where z is a source of randomness. we may then compute the derivatives of y with respect to ω using traditional tools such as the back - propagation algorithm applied to f, so long as f is continuous and [UNK] almost everywhere. crucially, ω must not be a function of z, and z must not be a function of ω. this technique is often


traditional tools such as the back - propagation algorithm applied to f, so long as f is continuous and [UNK] almost everywhere. crucially, ω must not be a function of z, and z must not be a function of ω. this technique is often called the reparametrization trick, stochastic back - propagation or perturbation analysis. the requirement that f be continuous and [UNK] of course requires y to be continuous. if we wish to back - propagate through a sampling process that produces discrete - valued samples, it may still be possible to estimate a gradient on ω, using reinforcement learning algorithms such as variants of the reinforce algorithm (, ), discussed in section. williams 1992 20. 9. 1 688


chapter 20. deep generative models in neural network applications, we typically choose z to be drawn from some simple distribution, such as a unit uniform or unit gaussian distribution, and achieve more complex distributions by allowing the deterministic portion of the network to reshape its input. the idea of propagating gradients or optimizing through stochastic operations dates back to the mid - twentieth century (, ;, ) and was price 1958 bonnet 1964 ﬁrst used for machine learning in the context of reinforcement learning (, williams 1992 ). more recently, it has been applied to variational approximations ( opper and archambeau 2009, ) and stochastic or generative neural networks ( bengio et al., ;, ; 2013b kingma 2013 kingma and welling 2014b a rezende 2014,, ; et al., ; goodfellow 2014c et al., ). many networks, such as denoising autoencoders or networks regularized with dropout, are also naturally designed to take noise as an input without requiring any special reparametrization to make the noise independent from the model. 20. 9. 1 back - propagating through discrete stochastic operations when a model emi


dropout, are also naturally designed to take noise as an input without requiring any special reparametrization to make the noise independent from the model. 20. 9. 1 back - propagating through discrete stochastic operations when a model emits a discrete variable y, the reparametrization trick is not applicable. suppose that the model takes inputs x and parameters θ, both encapsulated in the vector ω, and combines them with random noise z to produce y : y z ω = ( f ; ). ( 20. 58 ) because y is discrete, f must be a step function. the derivatives of a step function are not useful at any point. right at each step boundary, the derivatives are undeﬁned, but that is a small problem. the large problem is that the derivatives are zero almost everywhere, on the regions between step boundaries. the derivatives of any cost function j ( y ) therefore do not give any information for how to update the model parameters. θ the reinforce algorithm ( reward increment = non - negative factor × [UNK] reinforcement × characteristic eligibility ) provides a framework deﬁning a family of simple but powerful solutions (, ). the core idea is that williams 1992 even though j ( f ( z ; ω )


reward increment = non - negative factor × [UNK] reinforcement × characteristic eligibility ) provides a framework deﬁning a family of simple but powerful solutions (, ). the core idea is that williams 1992 even though j ( f ( z ; ω ) ) is a step function with useless derivatives, the expected cost ez z [UNK] ( ) j f ( ( ; ) ) z ω is often a smooth function amenable to gradient descent. although that expectation is typically not tractable when y is high - dimensional ( or is the result of the composition of many discrete stochastic decisions ), it can be estimated without bias using a monte carlo average. the stochastic estimate of the gradient can be used with sgd or other stochastic gradient - based optimization techniques. 689


chapter 20. deep generative models the simplest version of reinforce can be derived by simply [UNK] the expected cost : ez [ ( ) ] = j y y j p ( ) y ( ) y ( 20. 59 ) ∂ j e [ ( ) ] y ∂ω = y j ( ) y ∂p ( ) y ∂ω ( 20. 60 ) = y j p ( ) y ( ) y ∂ p log ( ) y ∂ω ( 20. 61 ) ≈1 m m y ( ) i [UNK], i ( ) y = 1 j ( y ( ) i ) ∂ p log ( y ( ) i ) ∂ω. ( 20. 62 ) equation relies on the assumption that 20. 60 j does not reference ω directly. it is trivial to extend the approach to relax this assumption. equation exploits 20. 61 the derivative rule for the logarithm, ∂ p log ( ) y ∂ω = 1 p ( ) y ∂p ( ) y ∂ω. equation gives 20. 62 an unbiased monte carlo estimator of the gradient. anywhere we write p ( y ) in this section, one could equally write p ( y x | ). this is because p ( y ) is parametrized by ω, and


an unbiased monte carlo estimator of the gradient. anywhere we write p ( y ) in this section, one could equally write p ( y x | ). this is because p ( y ) is parametrized by ω, and ω contains both θ and x, if x is present. one issue with the above simple reinforce estimator is that it has a very high variance, so that many samples of y need to be drawn to obtain a good estimator of the gradient, or equivalently, if only one sample is drawn, sgd will converge very slowly and will require a smaller learning rate. it is possible to considerably reduce the variance of that estimator by using variance reduction methods (, ;, ). the idea is to modify the estimator so wilson 1984 l ’ ecuyer 1994 that its expected value remains unchanged but its variance get reduced. in the context of reinforce, the proposed variance reduction methods involve the computation of a baseline that is used to [UNK] j ( y ). note that any [UNK] b ( ω ) that does not depend on y would not change the expectation of the estimated gradient because ep ( ) y ∂ p log ( ) y ∂ω = y p ( ) y ∂ p log ( ) y


). note that any [UNK] b ( ω ) that does not depend on y would not change the expectation of the estimated gradient because ep ( ) y ∂ p log ( ) y ∂ω = y p ( ) y ∂ p log ( ) y ∂ω ( 20. 63 ) = y ∂p ( ) y ∂ω ( 20. 64 ) = ∂ ∂ω y p ( ) = y ∂ ∂ω1 = 0, ( 20. 65 ) 690


chapter 20. deep generative models which means that ep ( ) y ( ( ) ( ) ) j y −b ω ∂ p log ( ) y ∂ω = ep ( ) y j ( ) y ∂ p log ( ) y ∂ω −b e ( ) ω p ( ) y ∂ p log ( ) y ∂ω ( 20. 66 ) = ep ( ) y j ( ) y ∂ p log ( ) y ∂ω. ( 20. 67 ) furthermore, we can obtain the optimal b ( ω ) by computing the variance of ( j ( y ) − b ( ω ) ) ∂ p log ( ) y ∂ω under p ( y ) and minimizing with respect to b ( ω ). what we ﬁnd is that this optimal baseline b∗ ( ) ω i is [UNK] for each element ωi of the vector : ω b∗ ( ) ω i = ep ( ) y j ( ) y ∂ p log ( ) y ∂ωi 2 ep ( ) y ∂ p log ( ) y ∂ωi 2. ( 20. 68 ) the gradient estimator with respect to ωi then becomes ( ( ) ( ) j y −b ω i ) ∂ p log ( ) y ∂ωi ( 20.


log ( ) y ∂ωi 2. ( 20. 68 ) the gradient estimator with respect to ωi then becomes ( ( ) ( ) j y −b ω i ) ∂ p log ( ) y ∂ωi ( 20. 69 ) where b ( ω ) i estimates the above b∗ ( ω ) i. the estimate b is usually obtained by adding extra outputs to the neural network and training the new outputs to estimate ep ( ) y [ j ( y ) ∂ p log ( ) y ∂ω i 2 ] and ep ( ) y ∂ p log ( ) y ∂ωi 2 for each element of ω. these extra outputs can be trained with the mean squared error objective, using respectively j ( y ) ∂ p log ( ) y ∂ωi 2 and ∂ p log ( ) y ∂ωi 2 as targets when y is sampled from p ( y ), for a given ω. the estimate b may then be recovered by substituting these estimates into equation. ( ) preferred to use a single shared output 20. 68 mnih and gregor 2014 ( across all elements i of ω ) trained with the target j ( y ), using as baseline b ( ω ) ≈ ep ( ) y [ ( ) ] j y. variance


use a single shared output 20. 68 mnih and gregor 2014 ( across all elements i of ω ) trained with the target j ( y ), using as baseline b ( ω ) ≈ ep ( ) y [ ( ) ] j y. variance reduction methods have been introduced in the reinforcement learning context (, ; sutton et al. 2000 weaver and tao 2001, ), generalizing previous work on the case of binary reward by dayan 1990 bengio 2013b mnih ( ). see et al. ( ), and gregor 2014 ba 2014 mnih 2014 xu 2015 ( ), et al. ( ), et al. ( ), or et al. ( ) for examples of modern uses of the reinforce algorithm with reduced variance in the context of deep learning. in addition to the use of an input - dependent baseline b ( ω ) (, ( ) found that the scale of mnih and gregor 2014 j ( y ) −b ( ω ) ) could be adjusted during training by dividing it by its standard deviation estimated by a moving average during training, as a kind of adaptive learning rate, to counter the [UNK] of important variations that occur during the course of training in the 691


chapter 20. deep generative models magnitude of this quantity. ( ) called this heuristic mnih and gregor 2014 variance normalization. reinforce - based estimators can be understood as estimating the gradient by correlating choices of y with corresponding values of j ( y ). if a good value of y is unlikely under the current parametrization, it might take a long time to obtain it by chance, and get the required signal that this conﬁguration should be reinforced. 20. 10 directed generative nets as discussed in chapter, directed graphical models make up a prominent class 16 of graphical models. while directed graphical models have been very popular within the greater machine learning community, within the smaller deep learning community they have until roughly 2013 been overshadowed by undirected models such as the rbm. in this section we review some of the standard directed graphical models that have traditionally been associated with the deep learning community. we have already described deep belief networks, which are a partially directed model. we have also already described sparse coding models, which can be thought of as shallow directed generative models. they are often used as feature learners in the context of deep learning, though they tend to perform poorly at sample generation and density estimation. we now describe a variety of


described sparse coding models, which can be thought of as shallow directed generative models. they are often used as feature learners in the context of deep learning, though they tend to perform poorly at sample generation and density estimation. we now describe a variety of deep, fully directed models. 20. 10. 1 sigmoid belief nets sigmoid belief networks (, ) are a simple form of directed graphical model neal 1990 with a speciﬁc kind of conditional probability distribution. in general, we can think of a sigmoid belief network as having a vector of binary states s, with each element of the state inﬂuenced by its ancestors : p s ( i ) = σ j < i wj, isj + bi. ( 20. 70 ) the most common structure of sigmoid belief network is one that is divided into many layers, with ancestral sampling proceeding through a series of many hidden layers and then ultimately generating the visible layer. this structure is very similar to the deep belief network, except that the units at the beginning of 692


chapter 20. deep generative models the sampling process are independent from each other, rather than sampled from a restricted boltzmann machine. such a structure is interesting for a variety of reasons. one reason is that the structure is a universal approximator of probability distributions over the visible units, in the sense that it can approximate any probability distribution over binary variables arbitrarily well, given enough depth, even if the width of the individual layers is restricted to the dimensionality of the visible layer ( sutskever and hinton 2008, ). while generating a sample of the visible units is very [UNK] in a sigmoid belief network, most other operations are not. inference over the hidden units given the visible units is intractable. mean ﬁeld inference is also intractable because the variational lower bound involves taking expectations of cliques that encompass entire layers. this problem has remained [UNK] enough to restrict the popularity of directed discrete networks. one approach for performing inference in a sigmoid belief network is to construct a [UNK] lower bound that is specialized for sigmoid belief networks (, saul et al. 1996 ). this approach has only been applied to very small networks. another approach is to use learned inference mechanisms as described in section. the 19. 5 helmholtz


bound that is specialized for sigmoid belief networks (, saul et al. 1996 ). this approach has only been applied to very small networks. another approach is to use learned inference mechanisms as described in section. the 19. 5 helmholtz machine ( dayan 1995 dayan and hinton 1996 et al., ;, ) is a sigmoid belief network combined with an inference network that predicts the parameters of the mean ﬁeld distribution over the hidden units. modern approaches (, gregor et al. 2014 mnih and gregor 2014 ;, ) to sigmoid belief networks still use this inference network approach. these techniques remain [UNK] due to the discrete nature of the latent variables. one cannot simply back - propagate through the output of the inference network, but instead must use the relatively unreliable machinery for back - propagating through discrete sampling processes, described in section. recent 20. 9. 1 approaches based on importance sampling, reweighted wake - sleep ( bornschein and bengio 2015 bornschein 2015, ) and bidirectional helmholtz machines ( et al., ) make it possible to quickly train sigmoid belief networks and reach state - of - the - art performance on benchmark tasks. a special case


bornschein 2015, ) and bidirectional helmholtz machines ( et al., ) make it possible to quickly train sigmoid belief networks and reach state - of - the - art performance on benchmark tasks. a special case of sigmoid belief networks is the case where there are no latent variables. learning in this case is [UNK], because there is no need to marginalize latent variables out of the likelihood. a family of models called auto - regressive networks generalize this fully visible belief network to other kinds of variables besides binary variables and other structures of conditional distributions besides log - linear relationships. auto - regressive networks are described later, in section. 20. 10. 7 693


chapter 20. deep generative models 20. 10. 2 [UNK] generator nets many generative models are based on the idea of using a [UNK] generator network. the model transforms samples of latent variables z to samples x or to distributions over samples x using a [UNK] function g ( z ; θ ( ) g ) which is typically represented by a neural network. this model class includes variational autoencoders, which pair the generator net with an inference net, generative adversarial networks, which pair the generator network with a discriminator network, and techniques that train generator networks in isolation. generator networks are essentially just parametrized computational procedures for generating samples, where the architecture provides the family of possible distributions to sample from and the parameters select a distribution from within that family. as an example, the standard procedure for drawing samples from a normal distribution with mean µ and covariance σ is to feed samples z from a normal distribution with zero mean and identity covariance into a very simple generator network. this generator network contains just one [UNK] layer : x z lz = ( g ) = + µ ( 20. 71 ) where is given by the cholesky decomposition of. l σ pseudorandom number generators can also use nonlinear transformations of simple distributions.


network contains just one [UNK] layer : x z lz = ( g ) = + µ ( 20. 71 ) where is given by the cholesky decomposition of. l σ pseudorandom number generators can also use nonlinear transformations of simple distributions. for example, inverse transform sampling ( devroye 2013, ) draws a scalar z from u ( 0, 1 ) and applies a nonlinear transformation to a scalar x. in this case g ( z ) is given by the inverse of the cumulative distribution function f ( x ) = x −∞p ( v ) dv. if we are able to specify p ( x ), integrate over x, and invert the resulting function, we can sample from without using machine learning. p x ( ) to generate samples from more complicated distributions that are [UNK] to specify directly, [UNK] to integrate over, or whose resulting integrals are [UNK] to invert, we use a feedforward network to represent a parametric family of nonlinear functions g, and use training data to infer the parameters selecting the desired function. we can think of g as providing a nonlinear change of variables that transforms the distribution over into the desired distribution over. z x recall from equation that, for invertible, [UNK], continuous, 3.


infer the parameters selecting the desired function. we can think of g as providing a nonlinear change of variables that transforms the distribution over into the desired distribution over. z x recall from equation that, for invertible, [UNK], continuous, 3. 47 g pz ( ) = z px ( ( ) ) g z det ( ∂g ∂z ). ( 20. 72 ) 694


chapter 20. deep generative models this implicitly imposes a probability distribution over : x px ( ) = x pz ( g−1 ( ) ) x det ( ∂g ∂z ). ( 20. 73 ) of course, this formula may be [UNK] to evaluate, depending on the choice of g, so we often use indirect means of learning g, rather than trying to maximize log ( ) p x directly. in some cases, rather than using g to provide a sample of x directly, we use g to deﬁne a conditional distribution over x. for example, we could use a generator net whose ﬁnal layer consists of sigmoid outputs to provide the mean parameters of bernoulli distributions : p ( xi = 1 ) = ( ) | z g z i. ( 20. 74 ) in this case, when we use g to deﬁne p ( x z | ), we impose a distribution over x by marginalizing : z p ( ) = x ezp. ( ) x z | ( 20. 75 ) both approaches deﬁne a distribution pg ( x ) and allow us to train various criteria of pg using the reparametrization trick of section. 20. 9 the two [UNK] approaches to formulating generator nets


x z | ( 20. 75 ) both approaches deﬁne a distribution pg ( x ) and allow us to train various criteria of pg using the reparametrization trick of section. 20. 9 the two [UNK] approaches to formulating generator nets — emitting the parameters of a conditional distribution versus directly emitting samples — have complementary strengths and weaknesses. when the generator net deﬁnes a conditional distribution over x, it is capable of generating discrete data as well as continuous data. when the generator net provides samples directly, it is capable of generating only continuous data ( we could introduce discretization in the forward propagation, but doing so would mean the model could no longer be trained using back - propagation ). the advantage to direct sampling is that we are no longer forced to use conditional distributions whose form can be easily written down and algebraically manipulated by a human designer. approaches based on [UNK] generator networks are motivated by the success of gradient descent applied to [UNK] feedforward networks for classiﬁcation. in the context of supervised learning, deep feedforward networks trained with gradient - based learning seem practically guaranteed to succeed given enough hidden units and enough training data. can this same recipe for success transfer to generative modeling? generative modeling seems to be more [UNK] than class


supervised learning, deep feedforward networks trained with gradient - based learning seem practically guaranteed to succeed given enough hidden units and enough training data. can this same recipe for success transfer to generative modeling? generative modeling seems to be more [UNK] than classiﬁcation or regression because the learning process requires optimizing intractable criteria. in the context 695


chapter 20. deep generative models of [UNK] generator nets, the criteria are intractable because the data does not specify both the inputs z and the outputs x of the generator net. in the case of supervised learning, both the inputs x and the outputs y were given, and the optimization procedure needs only to learn how to produce the speciﬁed mapping. in the case of generative modeling, the learning procedure needs to determine how to arrange space in a useful way and additionally how to map from to. z z x dosovitskiy 2015 et al. ( ) studied a simpliﬁed problem, where the correspondence between z and x is given. speciﬁcally, the training data is computer - rendered imagery of chairs. the latent variables z are parameters given to the rendering engine describing the choice of which chair model to use, the position of the chair, and other conﬁguration details that [UNK] the rendering of the image. using this synthetically generated data, a convolutional network is able to learn to map z descriptions of the content of an image to x approximations of rendered images. this suggests that contemporary [UNK] generator networks have [UNK] model capacity to be good generative models, and that contemporary optimization algorithms have the


##tional network is able to learn to map z descriptions of the content of an image to x approximations of rendered images. this suggests that contemporary [UNK] generator networks have [UNK] model capacity to be good generative models, and that contemporary optimization algorithms have the ability to ﬁt them. the [UNK] lies in determining how to train generator networks when the value of z for each x is not ﬁxed and known ahead of each time. the following sections describe several approaches to training [UNK] generator nets given only training samples of. x 20. 10. 3 variational autoencoders the variational autoencoder or vae (, ;, ) is a kingma 2013 rezende et al. 2014 directed model that uses learned approximate inference and can be trained purely with gradient - based methods. to generate a sample from the model, the vae ﬁrst draws a sample z from the code distribution pmodel ( z ). the sample is then run through a [UNK] generator network g ( z ). finally, x is sampled from a distribution pmodel ( x ; g ( z ) ) = pmodel ( x z | ). however, during training, the approximate inference network ( or encoder ) q ( z x | ) is used to obtain


sampled from a distribution pmodel ( x ; g ( z ) ) = pmodel ( x z | ). however, during training, the approximate inference network ( or encoder ) q ( z x | ) is used to obtain z and pmodel ( x z | ) is then viewed as a decoder network. the key insight behind variational autoencoders is that they may be trained by maximizing the variational lower bound associated with data point : l ( ) q x l ( ) = q ez z x [UNK] ( | ) log pmodel ( ) + ( ( ) ) z x, h q z | x ( 20. 76 ) = ez z x [UNK] ( | ) log pmodel ( ) x z | −dkl ( ( ) q z | x | | pmodel ( ) ) z ( 20. 77 ) ≤log pmodel ( ) x. ( 20. 78 ) 696


chapter 20. deep generative models in equation, we recognize the ﬁrst term as the joint log - likelihood of the visible 20. 76 and hidden variables under the approximate posterior over the latent variables ( just like with em, except that we use an approximate rather than the exact posterior ). we recognize also a second term, the entropy of the approximate posterior. when q is chosen to be a gaussian distribution, with noise added to a predicted mean value, maximizing this entropy term encourages increasing the standard deviation of this noise. more generally, this entropy term encourages the variational posterior to place high probability mass on many z values that could have generated x, rather than collapsing to a single point estimate of the most likely value. in equation, we recognize the ﬁrst term as the reconstruction 20. 77 log - likelihood found in other autoencoders. the second term tries to make the approximate posterior distribution q ( z | x ) and the model prior p model ( z ) approach each other. traditional approaches to variational inference and learning infer q via an opti - mization algorithm, typically iterated ﬁxed point equations ( section ). these 19. 4 approaches are slow and often require the ability to compute [UNK] log p model ( z x,


variational inference and learning infer q via an opti - mization algorithm, typically iterated ﬁxed point equations ( section ). these 19. 4 approaches are slow and often require the ability to compute [UNK] log p model ( z x, ) in closed form. the main idea behind the variational autoencoder is to train a parametric encoder ( also sometimes called an inference network or recognition model ) that produces the parameters of q. so long as z is a continuous variable, we can then back - propagate through samples of z drawn from q ( z x | ) = q ( z ; f ( x ; θ ) ) in order to obtain a gradient with respect to θ. learning then consists solely of maximizing l with respect to the parameters of the encoder and decoder. all of the expectations in may be approximated by monte carlo sampling. l the variational autoencoder approach is elegant, theoretically pleasing, and simple to implement. it also obtains excellent results and is among the state of the art approaches to generative modeling. its main drawback is that samples from variational autoencoders trained on images tend to be somewhat blurry. the causes of this phenomenon are not yet known. one


results and is among the state of the art approaches to generative modeling. its main drawback is that samples from variational autoencoders trained on images tend to be somewhat blurry. the causes of this phenomenon are not yet known. one possibility is that the blurriness is an intrinsic [UNK] of maximum likelihood, which minimizes dkl ( pdatapmodel ). as illustrated in ﬁgure, this means that the model will assign high probability to 3. 6 points that occur in the training set, but may also assign high probability to other points. these other points may include blurry images. part of the reason that the model would choose to put probability mass on blurry images rather than some other part of the space is that the variational autoencoders used in practice usually have a gaussian distribution for pmodel ( x ; g ( z ) ). maximizing a lower bound on the likelihood of such a distribution is similar to training a traditional autoencoder with mean squared error, in the sense that it has a tendency to ignore features of the input that occupy few pixels or that cause only a small change in the brightness of the pixels that they occupy. this issue is not speciﬁc to vaes


mean squared error, in the sense that it has a tendency to ignore features of the input that occupy few pixels or that cause only a small change in the brightness of the pixels that they occupy. this issue is not speciﬁc to vaes and 697


chapter 20. deep generative models is shared with generative models that optimize a log - likelihood, or equivalently, dkl ( pdatapmodel ), as argued by ( ) and by ( ). another theis et al. 2015 huszar 2015 troubling issue with contemporary vae models is that they tend to use only a small subset of the dimensions of z, as if the encoder was not able to transform enough of the local directions in input space to a space where the marginal distribution matches the factorized prior. the vae framework is very straightforward to extend to a wide range of model architectures. this is a key advantage over boltzmann machines, which require extremely careful model design to maintain tractability. vaes work very well with a diverse family of [UNK] operators. one particularly sophisticated vae is the deep recurrent attention writer or draw model (, ). gregor et al. 2015 draw uses a recurrent encoder and recurrent decoder combined with an attention mechanism. the generation process for the draw model consists of sequentially visiting [UNK] small image patches and drawing the values of the pixels at those points. vaes can also be extended to generate sequences by deﬁning variational rnns (


with an attention mechanism. the generation process for the draw model consists of sequentially visiting [UNK] small image patches and drawing the values of the pixels at those points. vaes can also be extended to generate sequences by deﬁning variational rnns (, ) by using a recurrent encoder and decoder within chung et al. 2015b the vae framework. generating a sample from a traditional rnn involves only non - deterministic operations at the output space. variational rnns also have random variability at the potentially more abstract level captured by the vae latent variables. the vae framework has been extended to maximize not just the traditional variational lower bound, but instead the importance weighted autoencoder (, ) objective : burda et al. 2015 lk ( ) = x, q ez ( 1 ),..., z ( ) k [UNK] | q ( z x ) log 1 k k i = 1 pmodel ( x z, ( ) i ) q ( z ( ) i | x ). ( 20. 79 ) this new objective is equivalent to the traditional lower bound l when k = 1. however, it may also be interpreted as forming an estimate of the true log pmodel ( x )


z ( ) i | x ). ( 20. 79 ) this new objective is equivalent to the traditional lower bound l when k = 1. however, it may also be interpreted as forming an estimate of the true log pmodel ( x ) using importance sampling of z from proposal distribution q ( z x | ). the importance weighted autoencoder objective is also a lower bound on log pmodel ( x ) and becomes tighter as increases. k variational autoencoders have some interesting connections to the mp - dbm and other approaches that involve back - propagation through the approximate inference graph ( goodfellow 2013b stoyanov 2011 brakel 2013 et al., ; et al., ; et al., ). these previous approaches required an inference procedure such as mean ﬁeld ﬁxed point equations to provide the computational graph. the variational autoencoder is deﬁned for arbitrary computational graphs, which makes it applicable to a wider range of probabilistic model families because there is no need to restrict the choice 698


chapter 20. deep generative models of models to those with tractable mean ﬁeld ﬁxed point equations. the variational autoencoder also has the advantage that it increases a bound on the log - likelihood of the model, while the criteria for the mp - dbm and related models are more heuristic and have little probabilistic interpretation beyond making the results of approximate inference accurate. one disadvantage of the variational autoencoder is that it learns an inference network for only one problem, inferring z given x. the older methods are able to perform approximate inference over any subset of variables given any other subset of variables, because the mean ﬁeld ﬁxed point equations specify how to share parameters between the computational graphs for all of these [UNK] problems. one very nice property of the variational autoencoder is that simultaneously training a parametric encoder in combination with the generator network forces the model to learn a predictable coordinate system that the encoder can capture. this makes it an excellent manifold learning algorithm. see ﬁgure for examples of 20. 6 low - dimensional manifolds learned by the variational autoencoder. in one of the cases demonstrated in the ﬁgure, the algorithm discovered two independent factors of variation present


manifold learning algorithm. see ﬁgure for examples of 20. 6 low - dimensional manifolds learned by the variational autoencoder. in one of the cases demonstrated in the ﬁgure, the algorithm discovered two independent factors of variation present in images of faces : angle of rotation and emotional expression. 20. 10. 4 generative adversarial networks generative adversarial networks or gans (, ) are another goodfellow et al. 2014c generative modeling approach based on [UNK] generator networks. generative adversarial networks are based on a game theoretic scenario in which the generator network must compete against an adversary. the generator network directly produces samplesx = g ( z ; θ ( ) g ). its adversary, the discriminator network, attempts to distinguish between samples drawn from the training data and samples drawn from the generator. the discriminator emits a probability value given by d ( x ; θ ( ) d ), indicating the probability that x is a real training example rather than a fake sample drawn from the model. the simplest way to formulate learning in generative adversarial networks is as a zero - sum game, in which a function v ( θ ( ) g, θ ( )


real training example rather than a fake sample drawn from the model. the simplest way to formulate learning in generative adversarial networks is as a zero - sum game, in which a function v ( θ ( ) g, θ ( ) d ) determines the [UNK] the discriminator. the generator receives −v ( θ ( ) g, θ ( ) d ) as its own [UNK]. during learning, each player attempts to maximize its own [UNK], so that at convergence g∗ = arg min g max d v g, d. ( ) ( 20. 80 ) the default choice for is v v ( θ ( ) g, θ ( ) d ) = [UNK] ( ) + d x e [UNK] log ( 1 ( ) ) −d x. ( 20. 81 ) 699


chapter 20. deep generative models figure 20. 6 : examples of two - dimensional coordinate systems for high - dimensional mani - folds, learned by a variational autoencoder ( kingma and welling 2014a, ). two dimensions may be plotted directly on the page for visualization, so we can gain an understanding of how the model works by training a model with a 2 - d latent code, even if we believe the intrinsic dimensionality of the data manifold is much higher. the images shown are not examples from the training set but images x actually generated by the model p ( x z | ), simply by changing the 2 - d “ code ” z ( each image corresponds to a [UNK] choice of “ code ” z on a 2 - d uniform grid ). ( left ) the two - dimensional map of the frey faces manifold. one dimension that has been discovered ( horizontal ) mostly corresponds to a rotation of the face, while the other ( vertical ) corresponds to the emotional expression. the ( right ) two - dimensional map of the mnist manifold. this drives the discriminator to attempt to learn to correctly classify samples as real or fake. simultaneously, the generator attempts to fool the classiﬁer into believing its samples are real. at convergence


two - dimensional map of the mnist manifold. this drives the discriminator to attempt to learn to correctly classify samples as real or fake. simultaneously, the generator attempts to fool the classiﬁer into believing its samples are real. at convergence, the generator ’ s samples are indistinguishable from real data, and the discriminator outputs 1 2 everywhere. the discriminator may then be discarded. the main motivation for the design of gans is that the learning process requires neither approximate inference nor approximation of a partition function gradient. in the case where maxd v ( g, d ) is convex in θ ( ) g ( such as the case where optimization is performed directly in the space of probability density functions ) the procedure is guaranteed to converge and is asymptotically consistent. unfortunately, learning in gans can be [UNK] in practice when g and d are represented by neural networks and max d v ( g, d ) is not convex. goodfellow 700


chapter 20. deep generative models ( ) identiﬁed non - convergence as an issue that may cause gans to underﬁt. 2014 in general, simultaneous gradient descent on two players ’ costs is not guaranteed to reach an equilibrium. consider for example the value function v ( a, b ) = ab, where one player controls a and incurs cost ab, while the other player controls b and receives a cost −ab. if we model each player as making inﬁnitesimally small gradient steps, each player reducing their own cost at the expense of the other player, then a and b go into a stable, circular orbit, rather than arriving at the equilibrium point at the origin. note that the equilibria for a minimax game are not local minima of v. instead, they are points that are simultaneously minima for both players ’ costs. this means that they are saddle points of v that are local minima with respect to the ﬁrst player ’ s parameters and local maxima with respect to the second player ’ s parameters. it is possible for the two players to take turns increasing then decreasing v forever, rather than landing exactly on the saddle point where neither player is capable of reducing its cost. it is not known to


##a with respect to the second player ’ s parameters. it is possible for the two players to take turns increasing then decreasing v forever, rather than landing exactly on the saddle point where neither player is capable of reducing its cost. it is not known to what extent this non - convergence problem [UNK] gans. goodfellow 2014 ( ) identiﬁed an alternative formulation of the [UNK], in which the game is no longer zero - sum, that has the same expected gradient as maximum likelihood learning whenever the discriminator is optimal. because maximum likelihood training converges, this reformulation of the gan game should also converge, given enough samples. unfortunately, this alternative formulation does not seem to improve convergence in practice, possibly due to suboptimality of the discriminator, or possibly due to high variance around the expected gradient. in realistic experiments, the best - performing formulation of the gan game is a [UNK] formulation that is neither zero - sum nor equivalent to maximum likelihood, introduced by ( ) with a heuristic motivation. in goodfellow et al. 2014c this best - performing formulation, the generator aims to increase the log probability that the discriminator makes a mistake, rather than aiming to decrease the log probability that the discriminator makes the


##stic motivation. in goodfellow et al. 2014c this best - performing formulation, the generator aims to increase the log probability that the discriminator makes a mistake, rather than aiming to decrease the log probability that the discriminator makes the correct prediction. this reformulation is motivated solely by the observation that it causes the derivative of the generator ’ s cost function with respect to the discriminator ’ s logits to remain large even in the situation where the discriminator conﬁdently rejects all generator samples. stabilization of gan learning remains an open problem. fortunately, gan learning performs well when the model architecture and hyperparameters are care - fully selected. ( ) crafted a deep convolutional gan ( dcgan ) radford et al. 2015 that performs very well for image synthesis tasks, and showed that its latent repre - sentation space captures important factors of variation, as shown in ﬁgure. 15. 9 see ﬁgure for examples of images generated by a dcgan generator. 20. 7 the gan learning problem can also be simpliﬁed by breaking the generation 701


chapter 20. deep generative models figure 20. 7 : images generated by gans trained on the lsun dataset. ( left ) images of bedrooms generated by a dcgan model, reproduced with permission from radford et al. ( ). images of churches generated by a lapgan model, reproduced with 2015 ( right ) permission from ( ). denton et al. 2015 process into many levels of detail. it is possible to train conditional gans ( mirza and osindero 2014, ) that learn to sample from a distribution p ( x y | ) rather than simply sampling from a marginal distribution p ( x ). ( ) denton et al. 2015 showed that a series of conditional gans can be trained to ﬁrst generate a very low - resolution version of an image, then incrementally add details to the image. this technique is called the lapgan model, due to the use of a laplacian pyramid to generate the images containing varying levels of detail. lapgan generators are able to fool not only discriminator networks but also human observers, with experimental subjects identifying up to 40 % of the outputs of the network as being real data. see ﬁgure for examples of images generated by a lapgan 20. 7 generator. one unusual


not only discriminator networks but also human observers, with experimental subjects identifying up to 40 % of the outputs of the network as being real data. see ﬁgure for examples of images generated by a lapgan 20. 7 generator. one unusual capability of the gan training procedure is that it can ﬁt proba - bility distributions that assign zero probability to the training points. rather than maximizing the log probability of speciﬁc points, the generator net learns to trace out a manifold whose points resemble training points in some way. somewhat para - doxically, this means that the model may assign a log - likelihood of negative inﬁnity to the test set, while still representing a manifold that a human observer judges to capture the essence of the generation task. this is not clearly an advantage or a disadvantage, and one may also guarantee that the generator network assigns non - zero probability to all points simply by making the last layer of the generator network add gaussian noise to all of the generated values. generator networks that add gaussian noise in this manner sample from the same distribution that one obtains by using the generator network to parametrize the mean of a conditional 702


chapter 20. deep generative models gaussian distribution. dropout seems to be important in the discriminator network. in particular, units should be stochastically dropped while computing the gradient for the generator network to follow. following the gradient of the deterministic version of the discriminator with its weights divided by two does not seem to be as [UNK]. likewise, never using dropout seems to yield poor results. while the gan framework is designed for [UNK] generator networks, similar principles can be used to train other kinds of models. for example, self - supervised boosting can be used to train an rbm generator to fool a logistic regression discriminator ( welling 2002 et al., ). 20. 10. 5 generative moment matching networks generative moment matching networks (, ;, li et al. 2015 dziugaite et al. 2015 ) are another form of generative model based on [UNK] generator networks. unlike vaes and gans, they do not need to pair the generator network with any other network — neither an inference network as used with vaes nor a discriminator network as used with gans. these networks are trained with a technique called moment matching. the basic idea behind moment matching is to train the generator


generator network with any other network — neither an inference network as used with vaes nor a discriminator network as used with gans. these networks are trained with a technique called moment matching. the basic idea behind moment matching is to train the generator in such a way that many of the statistics of samples generated by the model are as similar as possible to those of the statistics of the examples in the training set. in this context, a moment is an expectation of [UNK] powers of a random variable. for example, the ﬁrst moment is the mean, the second moment is the mean of the squared values, and so on. in multiple dimensions, each element of the random vector may be raised to [UNK] powers, so that a moment may be any quantity of the form exπixni i ( 20. 82 ) where n = [ n1, n2,..., nd ] is a vector of non - negative integers. upon ﬁrst examination, this approach seems to be computationally infeasible. for example, if we want to match all the moments of the form xix j, then we need to minimize the [UNK] between a number of values that is quadratic in the dimension of x. moreover, even matching all of the ﬁrst


. for example, if we want to match all the moments of the form xix j, then we need to minimize the [UNK] between a number of values that is quadratic in the dimension of x. moreover, even matching all of the ﬁrst and second moments would only be [UNK] to ﬁt a multivariate gaussian distribution, which captures only linear relationships between values. our ambitions for neural networks are to capture complex nonlinear relationships, which would require far more moments. gans avoid this problem of exhaustively enumerating all moments by using a 703


chapter 20. deep generative models dynamically updated discriminator that automatically focuses its attention on whichever statistic the generator network is matching the least [UNK]. instead, generative moment matching networks can be trained by minimizing a cost function called maximum mean discrepancy ( scholkopf and smola, 2002 gretton 2012 ; et al., ) or mmd. this cost function measures the error in the ﬁrst moments in an inﬁnite - dimensional space, using an implicit mapping to feature space deﬁned by a kernel function in order to make computations on inﬁnite - dimensional vectors tractable. the mmd cost is zero if and only if the two distributions being compared are equal. visually, the samples from generative moment matching networks are somewhat disappointing. fortunately, they can be improved by combining the generator network with an autoencoder. first, an autoencoder is trained to reconstruct the training set. next, the encoder of the autoencoder is used to transform the entire training set into code space. the generator network is then trained to generate code samples, which may be mapped to visually pleasing samples via the decoder. unlike gans, the cost function is deﬁ


autoencoder is used to transform the entire training set into code space. the generator network is then trained to generate code samples, which may be mapped to visually pleasing samples via the decoder. unlike gans, the cost function is deﬁned only with respect to a batch of examples from both the training set and the generator network. it is not possible to make a training update as a function of only one training example or only one sample from the generator network. this is because the moments must be computed as an empirical average across many samples. when the batch size is too small, mmd can underestimate the true amount of variation in the distributions being sampled. no ﬁnite batch size is [UNK] large to eliminate this problem entirely, but larger batches reduce the amount of underestimation. when the batch size is too large, the training procedure becomes infeasibly slow, because many examples must be processed in order to compute a single small gradient step. as with gans, it is possible to train a generator net using mmd even if that generator net assigns zero probability to the training points. 20. 10. 6 convolutional generative networks when generating images, it is often useful to use a generator network that includes a


to train a generator net using mmd even if that generator net assigns zero probability to the training points. 20. 10. 6 convolutional generative networks when generating images, it is often useful to use a generator network that includes a convolutional structure ( see for example goodfellow 2014c dosovitskiy et al. ( ) or et al. ( ) ). to do so, we use the “ transpose ” of the convolution operator, 2015 described in section. this approach often yields more realistic images and does 9. 5 so using fewer parameters than using fully connected layers without parameter sharing. convolutional networks for recognition tasks have information ﬂow from the image to some summarization layer at the top of the network, often a class label. 704


chapter 20. deep generative models as this image ﬂows upward through the network, information is discarded as the representation of the image becomes more invariant to nuisance transformations. in a generator network, the opposite is true. rich details must be added as the representation of the image to be generated propagates through the network, culminating in the ﬁnal representation of the image, which is of course the image itself, in all of its detailed glory, with object positions and poses and textures and lighting. the primary mechanism for discarding information in a convolutional recognition network is the pooling layer. the generator network seems to need to add information. we cannot put the inverse of a pooling layer into the generator network because most pooling functions are not invertible. a simpler operation is to merely increase the spatial size of the representation. an approach that seems to perform acceptably is to use an “ un - pooling ” as introduced by dosovitskiy et al. ( ). this layer corresponds to the inverse of the max - pooling operation under 2015 certain simplifying conditions. first, the stride of the max - pooling operation is constrained to be equal to the width of the pooling region. second, the maximum input within each


to the inverse of the max - pooling operation under 2015 certain simplifying conditions. first, the stride of the max - pooling operation is constrained to be equal to the width of the pooling region. second, the maximum input within each pooling region is assumed to be the input in the upper - left corner. finally, all non - maximal inputs within each pooling region are assumed to be zero. these are very strong and unrealistic assumptions, but they do allow the max - pooling operator to be inverted. the inverse un - pooling operation allocates a tensor of zeros, then copies each value from spatial coordinate i of the input to spatial coordinate i k × of the output. the integer value k deﬁnes the size of the pooling region. even though the assumptions motivating the deﬁnition of the un - pooling operator are unrealistic, the subsequent layers are able to learn to compensate for its unusual output, so the samples generated by the model as a whole are visually pleasing. 20. 10. 7 auto - regressive networks auto - regressive networks are directed probabilistic models with no latent random variables. the conditional probability distributions in these models are represented by neural networks (


a whole are visually pleasing. 20. 10. 7 auto - regressive networks auto - regressive networks are directed probabilistic models with no latent random variables. the conditional probability distributions in these models are represented by neural networks ( sometimes extremely simple neural networks such as logistic regression ). the graph structure of these models is the complete graph. they decompose a joint probability over the observed variables using the chain rule of probability to obtain a product of conditionals of the form p ( xd | xd−1,..., x1 ). such models have been called fully - visible bayes networks ( fvbns ) and used successfully in many forms, ﬁrst with logistic regression for each conditional distribution ( frey 1998, ) and then with neural networks with hidden units ( bengio and bengio 2000b larochelle and murray 2011, ;, ). in some forms of auto - regressive networks, such as nade (, ), described larochelle and murray 2011 705


chapter 20. deep generative models in section below, we can introduce a form of parameter sharing that 20. 10. 10 brings both a statistical advantage ( fewer unique parameters ) and a computational advantage ( less computation ). this is one more instance of the recurring deep learning motif of reuse of features. x1 x1 x2 x2 x3 x3 x4 x4 p x ( 4 | x1, x2, x3 ) p x ( 4 | x1, x2, x3 ) p x ( 3 | x1, x2 ) p x ( 3 | x1, x2 ) p x ( 2 | x1 ) p x ( 2 | x1 ) p x ( 1 ) p x ( 1 ) x1 x1 x2 x2 x3 x3 x4 x4 figure 20. 8 : a fully visible belief network predicts the i - th variable from the i −1 previous ones. ( top ) ( bottom ) the directed graphical model for an fvbn. corresponding computational graph, in the case of the logistic fvbn, where each prediction is made by a linear predictor. 20. 10. 8 linear auto - regressive networks the simplest form of auto - regress


fvbn. corresponding computational graph, in the case of the logistic fvbn, where each prediction is made by a linear predictor. 20. 10. 8 linear auto - regressive networks the simplest form of auto - regressive network has no hidden units and no sharing of parameters or features. each p ( xi | xi−1,..., x1 ) is parametrized as a linear model ( linear regression for real - valued data, logistic regression for binary data, softmax regression for discrete data ). this model was introduced by frey 1998 ( ) and has o ( d2 ) parameters when there are d variables to model. it is illustrated in ﬁgure. 20. 8 if the variables are continuous, a linear auto - regressive model is merely another way to formulate a multivariate gaussian distribution, capturing linear pairwise interactions between the observed variables. linear auto - regressive networks are essentially the generalization of linear classiﬁcation methods to generative modeling. they therefore have the same 706


chapter 20. deep generative models advantages and disadvantages as linear classiﬁers. like linear classiﬁers, they may be trained with convex loss functions, and sometimes admit closed form solutions ( as in the gaussian case ). like linear classiﬁers, the model itself does not [UNK] a way of increasing its capacity, so capacity must be raised using techniques like basis expansions of the input or the kernel trick. x1 x1 x2 x2 x3 x3 x4 x4 h1 h1 h2 h2 h3 h3 p x ( 4 | x1, x2, x3 ) p x ( 4 | x1, x2, x3 ) p x ( 3 | x1, x2 ) p x ( 3 | x1, x2 ) p x ( 2 | x1 ) p x ( 2 | x1 ) p x ( 1 ) p x ( 1 ) figure 20. 9 : a neural auto - regressive network predicts thei - th variable xi from the i −1 previous ones, but is parametrized so that features ( groups of hidden units denotedhi ) that are functions of x1,..., xi can be reused in predicting all


predicts thei - th variable xi from the i −1 previous ones, but is parametrized so that features ( groups of hidden units denotedhi ) that are functions of x1,..., xi can be reused in predicting all of the subsequent variables xi + 1, xi + 2,..., xd. 20. 10. 9 neural auto - regressive networks neural auto - regressive networks (,, ) have the same bengio and bengio 2000a b left - to - right graphical model as logistic auto - regressive networks ( ﬁgure ) but 20. 8 employ a [UNK] parametrization of the conditional distributions within that graphical model structure. the new parametrization is more powerful in the sense that its capacity can be increased as much as needed, allowing approximation of any joint distribution. the new parametrization can also improve generalization by introducing a parameter sharing and feature sharing principle common to deep learning in general. the models were motivated by the objective of avoiding the curse of dimensionality arising out of traditional tabular graphical models, sharing the same structure as ﬁgure. in tabular discrete probabilistic models, each 20. 8 conditional distribution is represented by a table of probabilities


of avoiding the curse of dimensionality arising out of traditional tabular graphical models, sharing the same structure as ﬁgure. in tabular discrete probabilistic models, each 20. 8 conditional distribution is represented by a table of probabilities, with one entry and one parameter for each possible conﬁguration of the variables involved. by using a neural network instead, two advantages are obtained : 707


chapter 20. deep generative models 1. the parametrization of each p ( xi | xi−1,..., x1 ) by a neural network with ( i −1 ) × k inputs and k outputs ( if the variables are discrete and take k values, encoded one - hot ) allows one to estimate the conditional probability without requiring an exponential number of parameters ( and examples ), yet still is able to capture high - order dependencies between the random variables. 2. instead of having a [UNK] neural network for the prediction of each xi, a connectivity illustrated in ﬁgure allows one to merge all left - to - right 20. 9 the neural networks into one. equivalently, it means that the hidden layer features computed for predicting xi can be reused for predicting xi k + ( k > 0 ). the hidden units are thus organized in groups that have the particularity that all the units in the i - th group only depend on the input values x1,..., xi. the parameters used to compute these hidden units are jointly optimized to improve the prediction of all the variables in the sequence. this is an instance of the reuse principle that recurs throughout deep learning in scenarios ranging from recurrent and convolutional