the value θ ( 0 ) used in the e - step. fortunately, the e - step reduces the gap to zero again as we enter the loop for the next time. the em algorithm contains a few [UNK] insights. first, there is the basic structure of the learning process, in which we update the model parameters to improve the likelihood of a completed dataset, where all missing variables have their values provided by an estimate of the posterior distribution. this particular insight is not unique to the em algorithm. for example, using gradient descent to maximize the log - likelihood also has this same property ; the log - likelihood gradient computations require taking expectations with respect to the posterior distribution over the hidden units. another key insight in the em algorithm is that we can continue to use one value of q even after we have moved to a [UNK] value of θ. this particular insight is used throughout classical machine learning to derive large m - step updates. in the context of deep learning, most models are too complex to admit a tractable solution for an optimal large m - step update, so this second insight which is more unique to the em algorithm is rarely used. 19. 3 map inference and sparse coding we usually use the term inference to refer to computing the probability distribution over one set of variables given


optimal large m - step update, so this second insight which is more unique to the em algorithm is rarely used. 19. 3 map inference and sparse coding we usually use the term inference to refer to computing the probability distribution over one set of variables given another. when training probabilistic models with latent variables, we are usually interested in computing p ( h v | ). an alternative form of inference is to compute the single most likely value of the missing variables, rather than to infer the entire distribution over their possible values. in the context 635

/home/ricoiban/GEMMA/mnlp_chatsplaining/RAG/Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville (z-lib.org).pdf


chapter 19. approximate inference of latent variable models, this means computing h∗ = arg max h p. ( ) h v | ( 19. 9 ) this is known as maximum a posteriori inference, abbreviated map inference. map inference is usually not thought of as approximate inference — it does compute the exact most likely value of h∗. however, if we wish to develop a learning process based on maximizing l ( v h,, q ), then it is helpful to think of map inference as a procedure that provides a value of q. in this sense, we can think of map inference as approximate inference, because it does not provide the optimal q. recall from section that exact inference consists of maximizing 19. 1 l ( ) = v θ,, q [UNK] [ log ( ) ] + ( ) p h v, h q ( 19. 10 ) with respect to q over an unrestricted family of probability distributions, using an exact optimization algorithm. we can derive map inference as a form of approximate inference by restricting the family of distributions q may be drawn from. speciﬁcally, we require to take on a dirac distribution : q q δ. ( ) = h v | ( ) h µ − ( 19. 11 ) this


inference by restricting the family of distributions q may be drawn from. speciﬁcally, we require to take on a dirac distribution : q q δ. ( ) = h v | ( ) h µ − ( 19. 11 ) this means that we can now control q entirely via µ. dropping terms of l that do not vary with, we are left with the optimization problem µ µ∗ = arg max µ log ( = ) p h µ v,, ( 19. 12 ) which is equivalent to the map inference problem h∗ = arg max h p. ( ) h v | ( 19. 13 ) we can thus justify a learning procedure similar to em, in which we alternate between performing map inference to infer h∗and then update θ to increase log p ( h∗, v ). as with em, this is a form of coordinate ascent on l, where we alternate between using inference to optimize l with respect to q and using parameter updates to optimize l with respect to θ. the procedure as a whole can be justiﬁed by the fact that l is a lower bound on log p ( v ). in the case of map inference, this justiﬁcation is rather vacuous, because


respect to θ. the procedure as a whole can be justiﬁed by the fact that l is a lower bound on log p ( v ). in the case of map inference, this justiﬁcation is rather vacuous, because the bound is inﬁnitely loose, due to the dirac distribution ’ s [UNK] entropy of negative inﬁnity. however, adding noise to would make the bound meaningful again. µ 636


chapter 19. approximate inference map inference is commonly used in deep learning as both a feature extractor and a learning mechanism. it is primarily used for sparse coding models. recall from section that sparse coding is a linear factor model that imposes 13. 4 a sparsity - inducing prior on its hidden units. a common choice is a factorial laplace prior, with p h ( i ) = λ 2 e− | λ hi |. ( 19. 14 ) the visible units are then generated by performing a linear transformation and adding noise : p, β ( ) = ( ; + x h | n v wh b −1i ). ( 19. 15 ) computing or even representing p ( h v | ) is [UNK]. every pair of variables hi and hj are both parents of v. this means that when v is observed, the graphical model contains an active path connecting hi and hj. all of the hidden units thus participate in one massive clique in p ( h v | ). if the model were gaussian then these interactions could be modeled [UNK] via the covariance matrix, but the sparse prior makes these interactions non - gaussian. because p ( h v | ) is intractable, so is the computation of the log


model were gaussian then these interactions could be modeled [UNK] via the covariance matrix, but the sparse prior makes these interactions non - gaussian. because p ( h v | ) is intractable, so is the computation of the log - likelihood and its gradient. we thus cannot use exact maximum likelihood learning. instead, we use map inference and learn the parameters by maximizing the elbo deﬁned by the dirac distribution around the map estimate of. h if we concatenate all of the h vectors in the training set into a matrix h, and concatenate all of the vectors into a matrix, then the sparse coding learning v v process consists of minimizing j, ( h w ) = i, j | hi, j | + i, j v hw − 2 i, j. ( 19. 16 ) most applications of sparse coding also involve weight decay or a constraint on the norms of the columns of w, in order to prevent the pathological solution with extremely small and large. h w we can minimize j by alternating between minimization with respect to h and minimization with respect to w. both sub - problems are convex. in fact, the minimization with respect to w is just a linear regression


large. h w we can minimize j by alternating between minimization with respect to h and minimization with respect to w. both sub - problems are convex. in fact, the minimization with respect to w is just a linear regression problem. however, minimization of j with respect to both arguments is usually not a convex problem. minimization with respect to h requires specialized algorithms such as the feature - sign search algorithm (, ). lee et al. 2007 637


chapter 19. approximate inference 19. 4 variational inference and learning we have seen how the evidence lower bound l ( v θ,, q ) is a lower bound on log p ( v ; θ ), how inference can be viewed as maximizing l with respect to q, and how learning can be viewed as maximizing l with respect to θ. we have seen that the em algorithm allows us to make large learning steps with a ﬁxed q and that learning algorithms based on map inference allow us to learn using a point estimate of p ( h v | ) rather than inferring the entire distribution. now we develop the more general approach to variational learning. the core idea behind variational learning is that we can maximize l over a restricted family of distributions q. this family should be chosen so that it is easy to compute eq log p ( h v, ). a typical way to do this is to introduce assumptions about how factorizes. q a common approach to variational learning is to impose the restriction that q is a factorial distribution : q ( ) = h v | i q h ( i | v ). ( 19. 17 ) this is called the mean ﬁeld approach. more generally, we can impose any graphi - cal model structure


q is a factorial distribution : q ( ) = h v | i q h ( i | v ). ( 19. 17 ) this is called the mean ﬁeld approach. more generally, we can impose any graphi - cal model structure we choose on q, to ﬂexibly determine how many interactions we want our approximation to capture. this fully general graphical model approach is called structured variational inference (, ). saul and jordan 1996 the beauty of the variational approach is that we do not need to specify a speciﬁc parametric form for q. we specify how it should factorize, but then the optimization problem determines the optimal probability distribution within those factorization constraints. for discrete latent variables, this just means that we use traditional optimization techniques to optimize a ﬁnite number of variables describing the q distribution. for continuous latent variables, this means that we use a branch of mathematics called calculus of variations to perform optimization over a space of functions, and actually determine which function should be used to represent q. calculus of variations is the origin of the names “ variational learning ” and “ variational inference, ” though these names apply even when the latent variables are discrete and calculus of variations is not needed. in the case of


used to represent q. calculus of variations is the origin of the names “ variational learning ” and “ variational inference, ” though these names apply even when the latent variables are discrete and calculus of variations is not needed. in the case of continuous latent variables, calculus of variations is a powerful technique that removes much of the responsibility from the human designer of the model, who now must specify only how q factorizes, rather than needing to guess how to design a speciﬁc that can accurately approximate the posterior. q because l ( v θ,, q ) is deﬁned to be log p ( v ; θ ) −dkl ( q ( h v | ) p ( h v | ; θ ) ), we can think of maximizing l with respect to q as minimizing dkl ( q ( h v | ) p ( h v | ) ). 638


chapter 19. approximate inference in this sense, we are ﬁtting q to p. however, we are doing so with the opposite direction of the kl divergence than we are used to using for ﬁtting an approximation. when we use maximum likelihood learning to ﬁt a model to data, we minimize dkl ( pdatapmodel ). as illustrated in ﬁgure, this means that maximum likelihood 3. 6 encourages the model to have high probability everywhere that the data has high probability, while our optimization - based inference procedure encourages q to have low probability everywhere the true posterior has low probability. both directions of the kl divergence can have desirable and undesirable properties. the choice of which to use depends on which properties are the highest priority for each application. in the case of the inference optimization problem, we choose to use dkl ( q ( h v | ) p ( h v | ) ) for computational reasons. speciﬁcally, computing dkl ( q ( h v | ) p ( h v | ) ) involves evaluating expectations with respect to q, so by designing q to be simple, we can simplify the required expectations. the opposite direction of the kl divergence would require computing expectations with respect


h v | ) p ( h v | ) ) involves evaluating expectations with respect to q, so by designing q to be simple, we can simplify the required expectations. the opposite direction of the kl divergence would require computing expectations with respect to the true posterior. because the form of the true posterior is determined by the choice of model, we cannot design a reduced - cost approach to computing dkl ( ( ) ( ) ) p h v | q h v | exactly. 19. 4. 1 discrete latent variables variational inference with discrete latent variables is relatively straightforward. we deﬁne a distribution q, typically one where each factor of q is just deﬁned by a lookup table over discrete states. in the simplest case, h is binary and we make the mean ﬁeld assumption that factorizes over each individual q hi. in this case we can parametrize q with a vector [UNK] whose entries are probabilities. then q h ( i = 1 ) = | v [UNK]. after determining how to represent q, we simply optimize its parameters. in the case of discrete latent variables, this is just a standard optimization problem. in principle the selection of q could be done with any optimization algorithm, such as gradient


. after determining how to represent q, we simply optimize its parameters. in the case of discrete latent variables, this is just a standard optimization problem. in principle the selection of q could be done with any optimization algorithm, such as gradient descent. because this optimization must occur in the inner loop of a learning algorithm, it must be very fast. to achieve this speed, we typically use special optimization algorithms that are designed to solve comparatively small and simple problems in very few iterations. a popular choice is to iterate ﬁxed point equations, in other words, to solve ∂ [UNK] l = 0 ( 19. 18 ) for [UNK]. we repeatedly update [UNK] elements of [UNK] until we satisfy a convergence 639


chapter 19. approximate inference criterion. to make this more concrete, we show how to apply variational inference to the binary sparse coding model ( we present here the model developed by henniges et al. ( ) but demonstrate traditional, generic mean ﬁeld applied to the model, 2010 while they introduce a specialized algorithm ). this derivation goes into considerable mathematical detail and is intended for the reader who wishes to fully resolve any ambiguity in the high - level conceptual description of variational inference and learning we have presented so far. readers who do not plan to derive or implement variational learning algorithms may safely skip to the next section without missing any new high - level concepts. readers who proceed with the binary sparse coding example are encouraged to review the list of useful properties of functions that commonly arise in probabilistic models in section. we use these properties 3. 10 liberally throughout the following derivations without highlighting exactly where we use each one. in the binary sparse coding model, the input v ∈rn is generated from the model by adding gaussian noise to the sum of m [UNK] components which can each be present or absent. each component is switched on or [UNK] the corresponding hidden unit in h ∈ { } 0 1, m : p h ( i = 1 ) = (


adding gaussian noise to the sum of m [UNK] components which can each be present or absent. each component is switched on or [UNK] the corresponding hidden unit in h ∈ { } 0 1, m : p h ( i = 1 ) = ( σ bi ) ( 19. 19 ) p, ( ) = ( ; v h | n v w h β−1 ) ( 19. 20 ) where b is a learnable set of biases, w is a learnable weight matrix, and β is a learnable, diagonal precision matrix. training this model with maximum likelihood requires taking the derivative with respect to the parameters. consider the derivative with respect to one of the biases : ∂ ∂bi log ( ) p v ( 19. 21 ) = ∂ ∂b ip ( ) v p ( ) v ( 19. 22 ) = ∂ ∂b i h p, ( h v ) p ( ) v ( 19. 23 ) = ∂ ∂b i h p p ( ) h ( ) v h | p ( ) v ( 19. 24 ) 640


chapter 19. approximate inference h1 h1 h2 h2 h3 h3 v1 v1 v2 v2 v3 v3 h4 h4 h1 h1 h2 h2 h3 h3 h4 h4 figure 19. 2 : the graph structure of a binary sparse coding model with four hidden units. ( left ) the graph structure of p ( h v, ). note that the edges are directed, and that every two hidden units are co - parents of every visible unit. the graph structure of ( right ) p ( h v | ). in order to account for the active paths between co - parents, the posterior distribution needs an edge between all of the hidden units. = h p ( ) v h | ∂ ∂b ip ( ) h p ( ) v ( 19. 25 ) = h p ( ) h v | ∂ ∂bi p ( ) h p ( ) h ( 19. 26 ) = [UNK] | p ( h v ) ∂ ∂bi log ( ) p h. ( 19. 27 ) this requires computing expectations with respect to p ( h v | ). unfortunately, p ( h v | ) is a complicated distribution. see ﬁgure for the graph structure of 19. 2 p ( h


p h. ( 19. 27 ) this requires computing expectations with respect to p ( h v | ). unfortunately, p ( h v | ) is a complicated distribution. see ﬁgure for the graph structure of 19. 2 p ( h v, ) and p ( h v | ). the posterior distribution corresponds to the complete graph over the hidden units, so variable elimination algorithms do not help us to compute the required expectations any faster than brute force. we can resolve this [UNK] by using variational inference and variational learning instead. we can make a mean ﬁeld approximation : q ( ) = h v | i q h ( i | v ). ( 19. 28 ) the latent variables of the binary sparse coding model are binary, so to represent a factorial q we simply need to model m bernoulli distributionsq ( hi | v ). a natural way to represent the means of the bernoulli distributions is with a vector [UNK] of probabilities, with q ( hi = 1 | v ) = [UNK]. we impose a restriction that [UNK] is never equal to 0 or to 1, in order to avoid errors when computing, for example, log [UNK]. we will see that the variational inference equations never assign or to 0 1 [UNK]


v ) = [UNK]. we impose a restriction that [UNK] is never equal to 0 or to 1, in order to avoid errors when computing, for example, log [UNK]. we will see that the variational inference equations never assign or to 0 1 [UNK] 641


chapter 19. approximate inference analytically. however, in a software implementation, machine rounding error could result in or values. in software, we may wish to implement binary sparse 0 1 coding using an unrestricted vector of variational parameters z and obtain [UNK] via the relation [UNK] = σ ( z ). we can thus safely compute log [UNK] on a computer by using the identity log ( σ zi ) = ( −ζ −zi ) relating the sigmoid and the softplus. to begin our derivation of variational learning in the binary sparse coding model, we show that the use of this mean ﬁeld approximation makes learning tractable. the evidence lower bound is given by l ( ) v θ,, q ( 19. 29 ) = [UNK] [ log ( ) ] + ( ) p h v, h q ( 19. 30 ) = [UNK] [ log ( ) + log ( ) log ( ) ] p h p v h | − q h v | ( 19. 31 ) = [UNK] m i = 1 log ( p hi ) + n i = 1 log ( p vi | − h ) m i = 1 log ( q hi | v ) ( 19. 32 ) = m i = 1 [UNK] ( log ( σ bi ) log − [UNK] )


log ( p hi ) + n i = 1 log ( p vi | − h ) m i = 1 log ( q hi | v ) ( 19. 32 ) = m i = 1 [UNK] ( log ( σ bi ) log − [UNK] ) + ( 1 [UNK] ) ( log ( σ −bi ) log ( 1 − [UNK] ) ) ( 19. 33 ) + [UNK] n i = 1 log βi 2π exp −βi 2 ( vi −wi, : h ) 2 ( 19. 34 ) = m i = 1 [UNK] ( log ( σ bi ) log − [UNK] ) + ( 1 [UNK] ) ( log ( σ −bi ) log ( 1 − [UNK] ) ) ( 19. 35 ) + 1 2 n i = 1 log βi 2π −βi v2 i −2viwi, : [UNK] + j w 2 i, j [UNK] + k j = wi, jwi, [UNK]. ( 19. 36 ) while these equations are somewhat unappealing aesthetically, they show that l can be expressed in a small number of simple arithmetic operations. the evidence lower bound l is therefore tractable. we can use l as a replacement for the intractable log - likelihood. in principle, we could simply run gradient ascent on


show that l can be expressed in a small number of simple arithmetic operations. the evidence lower bound l is therefore tractable. we can use l as a replacement for the intractable log - likelihood. in principle, we could simply run gradient ascent on both v and h and this would make a perfectly acceptable combined inference and training algorithm. usually, however, we do not do this, for two reasons. first, this would require storing [UNK] for each v. we typically prefer algorithms that do not require per - example memory. it is [UNK] to scale learning algorithms to billions of examples if we must remember a dynamically updated vector associated with each example. 642


chapter 19. approximate inference second, we would like to be able to extract the features [UNK] very quickly, in order to recognize the content of v. in a realistic deployed setting, we would need to be able to compute [UNK] in real time. for both these reasons, we typically do not use gradient descent to compute the mean ﬁeld parameters [UNK]. instead, we rapidly estimate them with ﬁxed point equations. the idea behind ﬁxed point equations is that we are seeking a local maximum with respect to [UNK], where ∇hl ( v θ,, [UNK] ) = 0. we cannot [UNK] solve this equation with respect to all of [UNK] simultaneously. however, we can solve for a single variable : ∂ [UNK] l ( v θ,, [UNK] ) = 0. ( 19. 37 ) we can then iteratively apply the solution to the equation for i = 1,..., m, and repeat the cycle until we satisfy a converge criterion. common convergence criteria include stopping when a full cycle of updates does not improve l by more than some tolerance amount, or when the cycle does not change [UNK] by more than some amount. iterating mean ﬁeld ﬁxed point equations is a general technique that can provide fast variational inference in a broad variety of models. to


by more than some tolerance amount, or when the cycle does not change [UNK] by more than some amount. iterating mean ﬁeld ﬁxed point equations is a general technique that can provide fast variational inference in a broad variety of models. to make this more concrete, we show how to derive the updates for the binary sparse coding model in particular. first, we must write an expression for the derivatives with respect to [UNK]. to do so, we substitute equation into the left side of equation : 19. 36 19. 37 ∂ [UNK] l ( v θ,, [UNK] ) ( 19. 38 ) = ∂ [UNK] m j = 1 [UNK] ( log ( σ bj ) log − [UNK] j ) + ( 1 [UNK] ) ( log ( σ −bj ) log ( 1 − [UNK] ) ) ( 19. 39 ) + 1 2 n j = 1 log βj 2π −βj v2 j −2vjwj, : [UNK] + k w2 j, [UNK] + l k = wj, kwj, [UNK] ( 19. 40 ) = log ( σ bi ) log − [UNK] − − 1 + log ( 1 [UNK] i ) + 1 log ( − σ −bi ) ( 19. 41 ) + n j = 1 βj


, kwj, [UNK] ( 19. 40 ) = log ( σ bi ) log − [UNK] − − 1 + log ( 1 [UNK] i ) + 1 log ( − σ −bi ) ( 19. 41 ) + n j = 1 βj vjwj, i −1 2w 2 j, i − k i = wj, kwj, i [UNK] ( 19. 42 ) 643


chapter 19. approximate inference = bi −log [UNK] + log ( 1 [UNK] ) + v βw :, i −1 2w :, iβw :, i − j i = w :, j βw :, [UNK]. ( 19. 43 ) to apply the ﬁxed point update inference rule, we solve for the [UNK] that sets equation to 0 : 19. 43 [UNK] = σ bi + vβw :, i −1 2w :, iβw :, i − j i = w :, jβw :, [UNK]. ( 19. 44 ) at this point, we can see that there is a close connection between recurrent neural networks and inference in graphical models. speciﬁcally, the mean ﬁeld ﬁxed point equations deﬁned a recurrent neural network. the task of this network is to perform inference. we have described how to derive this network from a model description, but it is also possible to train the inference network directly. several ideas based on this theme are described in chapter. 20 in the case of binary sparse coding, we can see that the recurrent network connection speciﬁed by equation consists of repeatedly updating the hidden 19. 44 units based on the changing values of the neighboring


this theme are described in chapter. 20 in the case of binary sparse coding, we can see that the recurrent network connection speciﬁed by equation consists of repeatedly updating the hidden 19. 44 units based on the changing values of the neighboring hidden units. the input always sends a ﬁxed message of vβw to the hidden units, but the hidden units constantly update the message they send to each other. speciﬁcally, two units [UNK] and [UNK] inhibit each other when their weight vectors are aligned. this is a form of competition — between two hidden units that both explain the input, only the one that explains the input best will be allowed to remain active. this competition is the mean ﬁeld approximation ’ s attempt to capture the explaining away interactions in the binary sparse coding posterior. the explaining away [UNK] actually should cause a multi - modal posterior, so that if we draw samples from the posterior, some samples will have one unit active, other samples will have the other unit active, but very few samples have both active. unfortunately, explaining away interactions cannot be modeled by the factorial q used for mean ﬁeld, so the mean ﬁeld approximation is forced to choose one mode to model. this is an instance of the behavior illustrated in ﬁgure


have both active. unfortunately, explaining away interactions cannot be modeled by the factorial q used for mean ﬁeld, so the mean ﬁeld approximation is forced to choose one mode to model. this is an instance of the behavior illustrated in ﬁgure. 3. 6 we can rewrite equation into an equivalent form that reveals some further 19. 44 insights : [UNK] = σ bi + v − j i = w :, [UNK] βw :, i−1 2 w :, iβw :, i. ( 19. 45 ) in this reformulation, we see the input at each step as consisting of v− j i = w :, [UNK] rather than v. we can thus think of unit i as attempting to encode the residual 644


chapter 19. approximate inference error in v given the code of the other units. we can thus think of sparse coding as an iterative autoencoder, that repeatedly encodes and decodes its input, attempting to ﬁx mistakes in the reconstruction after each iteration. in this example, we have derived an update rule that updates a single unit at a time. it would be advantageous to be able to update more units simultaneously. some graphical models, such as deep boltzmann machines, are structured in such a way that we can solve for many entries of [UNK] simultaneously. unfortunately, binary sparse coding does not admit such block updates. instead, we can use a heuristic technique called damping to perform block updates. in the damping approach, we solve for the individually optimal values of every element of [UNK], then move all of the values in a small step in that direction. this approach is no longer guaranteed to increase l at each step, but works well in practice for many models. see koller and friedman 2009 ( ) for more information about choosing the degree of synchrony and damping strategies in message passing algorithms. 19. 4. 2 calculus of variations before continuing with our presentation of variational learning, we must brieﬂy introduce an important


2009 ( ) for more information about choosing the degree of synchrony and damping strategies in message passing algorithms. 19. 4. 2 calculus of variations before continuing with our presentation of variational learning, we must brieﬂy introduce an important set of mathematical tools used in variational learning : calculus of variations. many machine learning techniques are based on minimizing a function j ( θ ) by ﬁnding the input vector θ ∈rn for which it takes on its minimal value. this can be accomplished with multivariate calculus and linear algebra, by solving for the critical points where ∇θj ( θ ) = 0. in some cases, we actually want to solve for a function f ( x ), such as when we want to ﬁnd the probability density function over some random variable. this is what calculus of variations enables us to do. a function of a function f is known as a functional j [ f ]. much as we can take partial derivatives of a function with respect to elements of its vector - valued argument, we can take functional derivatives, also known as variational derivatives, of a functional j [ f ] with respect to individual values of the function f ( x ) at any speciﬁc value of x. the functional derivative of the


valued argument, we can take functional derivatives, also known as variational derivatives, of a functional j [ f ] with respect to individual values of the function f ( x ) at any speciﬁc value of x. the functional derivative of the functional j with respect to the value of the function at point is denoted f x δ δf x ( ) j. a complete formal development of functional derivatives is beyond the scope of this book. for our purposes, it is [UNK] to state that for [UNK] functions f g y, ( ) x and [UNK] functions ( x ) with continuous derivatives, that δ δf ( ) x g f, d ( ( ) x x ) x = ∂ ∂y g f,. ( ( ) x x ) ( 19. 46 ) 645


chapter 19. approximate inference to gain some intuition for this identity, one can think of f ( x ) as being a vector with uncountably many elements, indexed by a real vector x. in this ( somewhat incomplete view ), the identity providing the functional derivatives is the same as we would obtain for a vector θ ∈rn indexed by positive integers : ∂ ∂θ i j g θ ( j, j ) = ∂ ∂θi g θ ( i, i. ) ( 19. 47 ) many results in other machine learning publications are presented using the more general euler - lagrange equation which allows g to depend on the derivatives of f as well as the value of f, but we do not need this fully general form for the results presented in this book. to optimize a function with respect to a vector, we take the gradient of the function with respect to the vector and solve for the point where every element of the gradient is equal to zero. likewise, we can optimize a functional by solving for the function where the functional derivative at every point is equal to zero. as an example of how this process works, consider the problem of ﬁnding the probability distribution function over x ∈r that has maximal [UNK] entropy. recall that the entropy


by solving for the function where the functional derivative at every point is equal to zero. as an example of how this process works, consider the problem of ﬁnding the probability distribution function over x ∈r that has maximal [UNK] entropy. recall that the entropy of a probability distribution is deﬁned as p x ( ) h p [ ] = −ex log ( ) p x. ( 19. 48 ) for continuous values, the expectation is an integral : h p [ ] = − p x p x dx. ( ) log ( ) ( 19. 49 ) we cannot simply maximize h [ p ] with respect to the function p ( x ), because the result might not be a probability distribution. instead, we need to use lagrange multipliers to add a constraint that p ( x ) integrates to 1. also, the entropy increases without bound as the variance increases. this makes the question of which distribution has the greatest entropy uninteresting. instead, we ask which distribution has maximal entropy for ﬁxed variance σ2. finally, the problem is underdetermined because the distribution can be shifted arbitrarily without changing the entropy. to impose a unique solution, we add a constraint that the mean of the distribution be µ. the


ﬁxed variance σ2. finally, the problem is underdetermined because the distribution can be shifted arbitrarily without changing the entropy. to impose a unique solution, we add a constraint that the mean of the distribution be µ. the lagrangian functional for this optimization problem is l [ ] = p λ1 p x dx ( ) −1 + λ2 ( [ ] ) + e x −µ λ3 e [ ( ) x µ − 2 ] −σ2 + [ ] h p ( 19. 50 ) 646


chapter 19. approximate inference = λ1p x λ ( ) + 2p x x λ ( ) + 3p x x µ ( ) ( − ) 2 −p x p x ( ) log ( ) dx λ − 1 −µλ2 −σ2λ3. ( 19. 51 ) to minimize the lagrangian with respect to p, we set the functional derivatives equal to 0 : [UNK], δ δp x ( ) l = λ1 + λ2x λ + 3 ( ) x µ − 2 − − 1 log ( ) = 0 p x. ( 19. 52 ) this condition now tells us the functional form of p ( x ). by algebraically re - arranging the equation, we obtain p x ( ) = exp λ1 + λ2x λ + 3 ( ) x µ − 2 −1. ( 19. 53 ) we never assumed directly that p ( x ) would take this functional form ; we obtained the expression itself by analytically minimizing a functional. to ﬁnish the minimization problem, we must choose the λ values to ensure that all of our constraints are satisﬁed. we are free to choose any λ values, because the gradient of the lagrangian with


a functional. to ﬁnish the minimization problem, we must choose the λ values to ensure that all of our constraints are satisﬁed. we are free to choose any λ values, because the gradient of the lagrangian with respect to the λ variables is zero so long as the constraints are satisﬁed. to satisfy all of the constraints, we may set λ1 = 1 −log σ √ 2π, λ2 = 0, and λ3 = − 1 2σ2 to obtain p x x µ, σ ( ) = ( n ; 2 ). ( 19. 54 ) this is one reason for using the normal distribution when we do not know the true distribution. because the normal distribution has the maximum entropy, we impose the least possible amount of structure by making this assumption. while examining the critical points of the lagrangian functional for the entropy, we found only one critical point, corresponding to maximizing the entropy for ﬁxed variance. what about the probability distribution function that minimizes the entropy? why did we not ﬁnd a second critical point corresponding to the minimum? the reason is that there is no speciﬁc function that achieves minimal entropy. as functions place more probability density on the two points x =


minimizes the entropy? why did we not ﬁnd a second critical point corresponding to the minimum? the reason is that there is no speciﬁc function that achieves minimal entropy. as functions place more probability density on the two points x = µ + σ and x = µ σ −, and place less probability density on all other values of x, they lose entropy while maintaining the desired variance. however, any function placing exactly zero mass on all but two points does not integrate to one, and is not a valid probability distribution. there thus is no single minimal entropy probability distribution function, much as there is no single minimal positive real number. instead, we can say that there is a sequence of probability distributions converging toward putting mass only on these two points. this degenerate scenario may be 647


chapter 19. approximate inference described as a mixture of dirac distributions. because dirac distributions are not described by a single probability distribution function, no dirac or mixture of dirac distribution corresponds to a single speciﬁc point in function space. these distributions are thus invisible to our method of solving for a speciﬁc point where the functional derivatives are zero. this is a limitation of the method. distributions such as the dirac must be found by other methods, such as guessing the solution and then proving that it is correct. 19. 4. 3 continuous latent variables when our graphical model contains continuous latent variables, we may still perform variational inference and learning by maximizing l. however, we must now use calculus of variations when maximizing with respect to. l q ( ) h v | in most cases, practitioners need not solve any calculus of variations problems themselves. instead, there is a general equation for the mean ﬁeld ﬁxed point updates. if we make the mean ﬁeld approximation q ( ) = h v | i q h ( i | v ), ( 19. 55 ) and ﬁx q ( hj | v ) for all j = i, then the optimal q ( h i | v ) may be obtained by normal


) = h v | i q h ( i | v ), ( 19. 55 ) and ﬁx q ( hj | v ) for all j = i, then the optimal q ( h i | v ) may be obtained by normalizing the unnormalized distribution [UNK] h ( i | v ) = exp [UNK] ( h−i | v ) log [UNK], ( v h ) ( 19. 56 ) so long as p does not assign probability to any joint conﬁguration of variables. 0 carrying out the expectation inside the equation will yield the correct functional form of q ( hi | v ). it is only necessary to derive functional forms of q directly using calculus of variations if one wishes to develop a new form of variational learning ; equation yields the mean ﬁeld approximation for any probabilistic model. 19. 56 equation is a ﬁxed point equation, designed to be iteratively applied for 19. 56 each value of i repeatedly until convergence. however, it also tells us more than that. it tells us the functional form that the optimal solution will take, whether we arrive there by ﬁxed point equations or not. this means we can take the functional form from that equation but regard some of the values that appear in it


more than that. it tells us the functional form that the optimal solution will take, whether we arrive there by ﬁxed point equations or not. this means we can take the functional form from that equation but regard some of the values that appear in it as parameters, that we can optimize with any optimization algorithm we like. as an example, consider a very simple probabilistic model, with latent variables h ∈r2 and just one visible variable, v. suppose that p ( h ) = n ( h ; 0, i ) and p ( v | h ) = n ( v ; wh ; 1 ). we could actually simplify this model by integrating out h ; the result is just a gaussian distribution over v. the model itself is not 648


chapter 19. approximate inference interesting ; we have constructed it only to provide a simple demonstration of how calculus of variations may be applied to probabilistic modeling. the true posterior is given, up to a normalizing constant, by p ( ) h v | ( 19. 57 ) [UNK], ( h v ) ( 19. 58 ) = ( p h1 ) ( p h2 ) ( ) p v h | ( 19. 59 ) [UNK] −1 2 h2 1 + h2 2 + ( v h − 1w1 −h2w2 ) 2 ( 19. 60 ) = exp −1 2 h2 1 + h2 2 + v2 + h2 1w2 1 + h2 2 w2 2 −2vh1w1 −2vh2w2 + 2h1w1h2w2. ( 19. 61 ) due to the presence of the terms multiplying h1 and h2 together, we can see that the true posterior does not factorize over h1 and h2. applying equation, we ﬁnd that 19. 56 [UNK] h ( 1 | v ) ( 19. 62 ) = exp [UNK] ( h2 | v ) log [UNK], ( v h ) (


not factorize over h1 and h2. applying equation, we ﬁnd that 19. 56 [UNK] h ( 1 | v ) ( 19. 62 ) = exp [UNK] ( h2 | v ) log [UNK], ( v h ) ( 19. 63 ) = exp −1 [UNK] ( h2 | v ) h2 1 + h2 2 + v2 + h2 1w2 1 + h2 2w2 2 ( 19. 64 ) −2vh1w1 −2vh2w2 + 2h 1w1h2w2 ]. ( 19. 65 ) from this, we can see that there are [UNK] only two values we need to obtain from q ( h2 | v ) : eh [UNK] | q ( h v ) [ h2 ] and [UNK] | q ( h v ) [ h2 2 ]. writing these as h2and h2 2, we obtain [UNK] h ( 1 | v ) = exp −1 2 h2 1 + h2 2 + v2 + h2 1w2 1 + h2 2w2 2 ( 19. 66 ) −2vh1w1 − 2v h2w2 + 2h 1w1h2w2


+ h2 2 + v2 + h2 1w2 1 + h2 2w2 2 ( 19. 66 ) −2vh1w1 − 2v h2w2 + 2h 1w1h2w2 ]. ( 19. 67 ) from this, we can see that [UNK] has the functional form of a gaussian. we can thus conclude q ( h v | ) = n ( h ; µ β, −1 ) where µ and diagonal β are variational parameters that we can optimize using any technique we choose. it is important to recall that we did not ever assume that q would be gaussian ; its gaussian form was derived automatically by using calculus of variations to maximize q with 649


chapter 19. approximate inference respect to l. using the same approach on a [UNK] model could yield a [UNK] functional form of. q this was of course, just a small case constructed for demonstration purposes. for examples of real applications of variational learning with continuous variables in the context of deep learning, see ( ). goodfellow et al. 2013d 19. 4. 4 interactions between learning and inference using approximate inference as part of a learning algorithm [UNK] the learning process, and this in turn [UNK] the accuracy of the inference algorithm. speciﬁcally, the training algorithm tends to adapt the model in a way that makes the approximating assumptions underlying the approximate inference algorithm become more true. when training the parameters, variational learning increases [UNK] log ( ) p v h,. ( 19. 68 ) for a speciﬁc v, this increases p ( h v | ) for values of h that have high probability under q ( h v | ) and decreases p ( h v | ) for values of h that have low probability under. q ( ) h v | this behavior causes our approximating assumptions to become self - fulﬁlling prophecies. if we train the model with a unimodal approximate posterior, we will obtain a model with a


have low probability under. q ( ) h v | this behavior causes our approximating assumptions to become self - fulﬁlling prophecies. if we train the model with a unimodal approximate posterior, we will obtain a model with a true posterior that is far closer to unimodal than we would have obtained by training the model with exact inference. computing the true amount of harm imposed on a model by a variational approximation is thus very [UNK]. there exist several methods for estimating log p ( v ). we often estimate log p ( v ; θ ) after training the model, and ﬁnd that the gap with l ( v θ,, q ) is small. from this, we can conclude that our variational approximation is accurate for the speciﬁc value of θ that we obtained from the learning process. we should not conclude that our variational approximation is accurate in general or that the variational approximation did little harm to the learning process. to measure the true amount of harm induced by the variational approximation, we would need to know θ∗ = maxθ log p ( v ; θ ). it is possible for l ( v θ,, q ) ≈log p ( v ; θ ) and log p ( v ; θ )


variational approximation, we would need to know θ∗ = maxθ log p ( v ; θ ). it is possible for l ( v θ,, q ) ≈log p ( v ; θ ) and log p ( v ; θ ) log p ( v ; θ∗ ) to hold simultaneously. if maxql ( v θ, ∗, q ) log p ( v ; θ∗ ), because θ∗induces too complicated of a posterior distribution for our q family to capture, then the learning process will never approach θ∗. such a problem is very [UNK] to detect, because we can only know for sure that it happened if we have a superior learning algorithm that can ﬁnd θ∗ for comparison. 650


chapter 19. approximate inference 19. 5 learned approximate inference we have seen that inference can be thought of as an optimization procedure that increases the value of a function l. explicitly performing optimization via iterative procedures such as ﬁxed point equations or gradient - based optimization is often very expensive and time - consuming. many approaches to inference avoid this expense by learning to perform approximate inference. speciﬁcally, we can think of the optimization process as a function f that maps an input v to an approximate distribution q∗ = arg maxq l ( v, q ). once we think of the multi - step iterative optimization process as just being a function, we can approximate it with a neural network that implements an approximation [UNK] f ( ; ) v θ. 19. 5. 1 wake - sleep one of the main [UNK] with training a model to infer h from v is that we do not have a supervised training set with which to train the model. given a v, we do not know the appropriate h. the mapping from v to h depends on the choice of model family, and evolves throughout the learning process as θ changes. the wake - sleep algorithm ( hinton 1995b frey 1996 et al., ; et al., ) resolves this problem by


from v to h depends on the choice of model family, and evolves throughout the learning process as θ changes. the wake - sleep algorithm ( hinton 1995b frey 1996 et al., ; et al., ) resolves this problem by drawing samples of both h and v from the model distribution. for example, in a directed model, this can be done cheaply by performing ancestral sampling beginning at h and ending at v. the inference network can then be trained to perform the reverse mapping : predicting which h caused the present v. the main drawback to this approach is that we will only be able to train the inference network on values of v that have high probability under the model. early in learning, the model distribution will not resemble the data distribution, so the inference network will not have an opportunity to learn on samples that resemble data. in section we saw that one possible explanation for the role of dream sleep 18. 2 in human beings and animals is that dreams could provide the negative phase samples that monte carlo training algorithms use to approximate the negative gradient of the log partition function of undirected models. another possible explanation for biological dreaming is that it is providing samples from p ( h v, ) which can be used to train an inference network to predict h given


algorithms use to approximate the negative gradient of the log partition function of undirected models. another possible explanation for biological dreaming is that it is providing samples from p ( h v, ) which can be used to train an inference network to predict h given v. in some senses, this explanation is more satisfying than the partition function explanation. monte carlo algorithms generally do not perform well if they are run using only the positive phase of the gradient for several steps then with only the negative phase of the gradient for several steps. human beings and animals are usually awake for several consecutive hours then asleep for several consecutive hours. it is 651


chapter 19. approximate inference not readily apparent how this schedule could support monte carlo training of an undirected model. learning algorithms based on maximizing l can be run with prolonged periods of improving q and prolonged periods of improving θ, however. if the role of biological dreaming is to train networks for predicting q, then this explains how animals are able to remain awake for several hours ( the longer they are awake, the greater the gap between l and log p ( v ), but l will remain a lower bound ) and to remain asleep for several hours ( the generative model itself is not modiﬁed during sleep ) without damaging their internal models. of course, these ideas are purely speculative, and there is no hard evidence to suggest that dreaming accomplishes either of these goals. dreaming may also serve reinforcement learning rather than probabilistic modeling, by sampling synthetic experiences from the animal ’ s transition model, on which to train the animal ’ s policy. or sleep may serve some other purpose not yet anticipated by the machine learning community. 19. 5. 2 other forms of learned inference this strategy of learned approximate inference has also been applied to other models. salakhutdinov and larochelle 2010 ( ) showed that a single pass in a learned inference network


learning community. 19. 5. 2 other forms of learned inference this strategy of learned approximate inference has also been applied to other models. salakhutdinov and larochelle 2010 ( ) showed that a single pass in a learned inference network could yield faster inference than iterating the mean ﬁeld ﬁxed point equations in a dbm. the training procedure is based on running the inference network, then applying one step of mean ﬁeld to improve its estimates, and training the inference network to output this reﬁned estimate instead of its original estimate. we have already seen in section that the predictive sparse decomposition 14. 8 model trains a shallow encoder network to predict a sparse code for the input. this can be seen as a hybrid between an autoencoder and sparse coding. it is possible to devise probabilistic semantics for the model, under which the encoder may be viewed as performing learned approximate map inference. due to its shallow encoder, psd is not able to implement the kind of competition between units that we have seen in mean ﬁeld inference. however, that problem can be remedied by training a deep encoder to perform learned approximate inference, as in the ista technique (, ).


to implement the kind of competition between units that we have seen in mean ﬁeld inference. however, that problem can be remedied by training a deep encoder to perform learned approximate inference, as in the ista technique (, ). gregor and lecun 2010b learned approximate inference has recently become one of the dominant approaches to generative modeling, in the form of the variational autoencoder (, ;, ). in this elegant approach, there is no need to kingma 2013 rezende et al. 2014 construct explicit targets for the inference network. instead, the inference network is simply used to deﬁne l, and then the parameters of the inference network are adapted to increase. this model is described in depth later, in section. l 20. 10. 3 652


chapter 19. approximate inference using approximate inference, it is possible to train and use a wide variety of models. many of these models are described in the next chapter. 653


chapter 20 deep generative models in this chapter, we present several of the speciﬁc kinds of generative models that can be built and trained using the techniques presented in chapters –. all of 16 19 these models represent probability distributions over multiple variables in some way. some allow the probability distribution function to be evaluated explicitly. others do not allow the evaluation of the probability distribution function, but support operations that implicitly require knowledge of it, such as drawing samples from the distribution. some of these models are structured probabilistic models described in terms of graphs and factors, using the language of graphical models presented in chapter. others can not easily be described in terms of factors, 16 but represent probability distributions nonetheless. 20. 1 boltzmann machines boltzmann machines were originally introduced as a general “ connectionist ” ap - proach to learning arbitrary probability distributions over binary vectors ( fahlman et al., ; 1983 ackley 1985 hinton 1984 hinton and sejnowski 1986 et al., ; et al., ;, ). variants of the boltzmann machine that include other kinds of variables have long ago surpassed the popularity of the original. in this section we brieﬂy introduce the binary boltzmann machine and discuss the issues that come


., ;, ). variants of the boltzmann machine that include other kinds of variables have long ago surpassed the popularity of the original. in this section we brieﬂy introduce the binary boltzmann machine and discuss the issues that come up when trying to train and perform inference in the model. we deﬁne the boltzmann machine over a d - dimensional binary random vector x ∈ { 0, 1 } d. the boltzmann machine is an energy - based model ( section ), 16. 2. 4 654


chapter 20. deep generative models meaning we deﬁne the joint probability distribution using an energy function : p ( ) = x exp ( ( ) ) −e x z, ( 20. 1 ) where e ( x ) is the energy function and z is the partition function that ensures that x p ( ) = 1 x. the energy function of the boltzmann machine is given by e ( ) = x −xux b − x, ( 20. 2 ) where u is the “ weight ” matrix of model parameters and b is the vector of bias parameters. in the general setting of the boltzmann machine, we are given a set of training examples, each of which are n - dimensional. equation describes the joint 20. 1 probability distribution over the observed variables. while this scenario is certainly viable, it does limit the kinds of interactions between the observed variables to those described by the weight matrix. speciﬁcally, it means that the probability of one unit being on is given by a linear model ( logistic regression ) from the values of the other units. the boltzmann machine becomes more powerful when not all the variables are observed. in this case, the latent variables, can act similarly to hidden units in a multi - layer


model ( logistic regression ) from the values of the other units. the boltzmann machine becomes more powerful when not all the variables are observed. in this case, the latent variables, can act similarly to hidden units in a multi - layer perceptron and model higher - order interactions among the visible units. just as the addition of hidden units to convert logistic regression into an mlp results in the mlp being a universal approximator of functions, a boltzmann machine with hidden units is no longer limited to modeling linear relationships between variables. instead, the boltzmann machine becomes a universal approximator of probability mass functions over discrete variables (, ). le roux and bengio 2008 formally, we decompose the units x into two subsets : the visible units v and the latent ( or hidden ) units. the energy function becomes h e, ( v h v ) = −rv v − wh h − sh b − v c − h. ( 20. 3 ) boltzmann machine learning learning algorithms for boltzmann machines are usually based on maximum likelihood. all boltzmann machines have an intractable partition function, so the maximum likelihood gradient must be ap - proximated using the techniques described


3 ) boltzmann machine learning learning algorithms for boltzmann machines are usually based on maximum likelihood. all boltzmann machines have an intractable partition function, so the maximum likelihood gradient must be ap - proximated using the techniques described in chapter. 18 one interesting property of boltzmann machines when trained with learning rules based on maximum likelihood is that the update for a particular weight connecting two units depends only the statistics of those two units, collected under [UNK] distributions : pmodel ( v ) and [UNK] ( v ) pmodel ( h v | ). the rest of the 655


chapter 20. deep generative models network participates in shaping those statistics, but the weight can be updated without knowing anything about the rest of the network or how those statistics were produced. this means that the learning rule is “ local, ” which makes boltzmann machine learning somewhat biologically plausible. it is conceivable that if each neuron were a random variable in a boltzmann machine, then the axons and dendrites connecting two random variables could learn only by observing the ﬁring pattern of the cells that they actually physically touch. in particular, in the positive phase, two units that frequently activate together have their connection strengthened. this is an example of a hebbian learning rule (, ) often hebb 1949 summarized with the mnemonic “ ﬁre together, wire together. ” hebbian learning rules are among the oldest hypothesized explanations for learning in biological systems and remain relevant today (, ). giudice et al. 2009 other learning algorithms that use more information than local statistics seem to require us to hypothesize the existence of more machinery than this. for example, for the brain to implement back - propagation in a multilayer perceptron, it seems necessary for the brain to maintain a secondary


local statistics seem to require us to hypothesize the existence of more machinery than this. for example, for the brain to implement back - propagation in a multilayer perceptron, it seems necessary for the brain to maintain a secondary communication network for transmitting gradient information backwards through the network. proposals for biologically plausible implementations ( and approximations ) of back - propagation have been made (, ;, ) but remain to be validated, and hinton 2007a bengio 2015 bengio 2015 ( ) links back - propagation of gradients to inference in energy - based models similar to the boltzmann machine ( but with continuous latent variables ). the negative phase of boltzmann machine learning is somewhat harder to explain from a biological point of view. as argued in section, dream sleep 18. 2 may be a form of negative phase sampling. this idea is more speculative though. 20. 2 restricted boltzmann machines invented under the name harmonium (, ), restricted boltzmann smolensky 1986 machines are some of the most common building blocks of deep probabilistic models. we have brieﬂy described rbms previously, in section. here we review the 16. 7. 1 previous information and go into more detail. rb


1986 machines are some of the most common building blocks of deep probabilistic models. we have brieﬂy described rbms previously, in section. here we review the 16. 7. 1 previous information and go into more detail. rbms are undirected probabilistic graphical models containing a layer of observable variables and a single layer of latent variables. rbms may be stacked ( one on top of the other ) to form deeper models. see ﬁgure for some examples. in particular, ﬁgure a shows the 20. 1 20. 1 graph structure of the rbm itself. it is a bipartite graph, with no connections permitted between any variables in the observed layer or between any units in the latent layer. 656


chapter 20. deep generative models h1 h1 h2 h2 h3 h3 v1 v1 v2 v2 v3 v3 h4 h4 h ( 1 ) 1 h ( 1 ) 1 h ( 1 ) 2 h ( 1 ) 2 h ( 1 ) 3 h ( 1 ) 3 v1 v1 v2 v2 v3 v3 h ( 2 ) 1 h ( 2 ) 1 h ( 2 ) 2 h ( 2 ) 2 h ( 2 ) 3 h ( 2 ) 3 h ( 1 ) 4 h ( 1 ) 4 ( a ) ( b ) h ( 1 ) 1 h ( 1 ) 1 h ( 1 ) 2 h ( 1 ) 2 h ( 1 ) 3 h ( 1 ) 3 v1 v1 v2 v2 v3 v3 h ( 2 ) 1 h ( 2 ) 1 h ( 2 ) 2 h ( 2 ) 2 h ( 2 ) 3 h ( 2 ) 3 h ( 1 ) 4 h ( 1 ) 4 ( c ) figure 20. 1 : examples of models that may be built with restricted boltzmann machines. ( a ) the restricted boltzmann machine itself is an undirected graphical model based on a bipartite graph, with visible units in


c ) figure 20. 1 : examples of models that may be built with restricted boltzmann machines. ( a ) the restricted boltzmann machine itself is an undirected graphical model based on a bipartite graph, with visible units in one part of the graph and hidden units in the other part. there are no connections among the visible units, nor any connections among the hidden units. typically every visible unit is connected to every hidden unit but it is possible to construct sparsely connected rbms such as convolutional rbms. a ( b ) deep belief network is a hybrid graphical model involving both directed and undirected connections. like an rbm, it has no intralayer connections. however, a dbn has multiple hidden layers, and thus there are connections between hidden units that are in separate layers. all of the local conditional probability distributions needed by the deep belief network are copied directly from the local conditional probability distributions of its constituent rbms. alternatively, we could also represent the deep belief network with a completely undirected graph, but it would need intralayer connections to capture the dependencies between parents. a deep boltzmann machine is an undirected graphical ( c ) model with several layers of latent variables. like rbms and


a completely undirected graph, but it would need intralayer connections to capture the dependencies between parents. a deep boltzmann machine is an undirected graphical ( c ) model with several layers of latent variables. like rbms and dbns, dbms lack intralayer connections. dbms are less closely tied to rbms than dbns are. when initializing a dbm from a stack of rbms, it is necessary to modify the rbm parameters slightly. some kinds of dbms may be trained without ﬁrst training a set of rbms. 657


chapter 20. deep generative models we begin with the binary version of the restricted boltzmann machine, but as we see later there are extensions to other types of visible and hidden units. more formally, let the observed layer consist of a set of n v binary random variables which we refer to collectively with the vector v. we refer to the latent or hidden layer of nh binary random variables as. h like the general boltzmann machine, the restricted boltzmann machine is an energy - based model with the joint probability distribution speciﬁed by its energy function : p, ( = v v h = ) = h 1 z exp ( ( ) ) −e v h,. ( 20. 4 ) the energy function for an rbm is given by e, ( v h b ) = −v c − h v − wh, ( 20. 5 ) and is the normalizing constant known as the partition function : z z = v h exp ( ) { −e v h, }. ( 20. 6 ) it is apparent from the deﬁnition of the partition function z that the naive method of computing z ( exhaustively summing over all states ) could be computationally intractable, unless a cleverly designed algorithm


}. ( 20. 6 ) it is apparent from the deﬁnition of the partition function z that the naive method of computing z ( exhaustively summing over all states ) could be computationally intractable, unless a cleverly designed algorithm could exploit regularities in the probability distribution to compute z faster. in the case of restricted boltzmann machines, ( ) formally proved that the partition function long and servedio 2010 z is intractable. the intractable partition function z implies that the normalized joint probability distribution is also intractable to evaluate. p ( ) v 20. 2. 1 conditional distributions though p ( v ) is intractable, the bipartite graph structure of the rbm has the very special property that its conditional distributions p ( h v | ) and p ( v h | ) are factorial and relatively simple to compute and to sample from. deriving the conditional distributions from the joint distribution is straightfor - ward : p ( ) = h v | p, ( h v ) p ( ) v ( 20. 7 ) = 1 p ( ) v 1 z exp bv c + h v + wh ( 20. 8 ) = 1 z exp ch v + wh ( 20. 9 ) 658


chapter 20. deep generative models = 1 z exp nh j = 1 cjhj + nh j = 1 vw :, jhj ( 20. 10 ) = 1 z nh j = 1 exp cjhj + vw :, jhj ( 20. 11 ) since we are conditioning on the visible units v, we can treat these as constant with respect to the distribution p ( h v | ). the factorial nature of the conditional p ( h v | ) follows immediately from our ability to write the joint probability over the vector h as the product of ( unnormalized ) distributions over the individual elements, h j. it is now a simple matter of normalizing the distributions over the individual binary hj. p h ( j = 1 ) = | v [UNK] h ( j = 1 ) | v [UNK] p h ( j = 0 ) + | v [UNK] h ( j = 1 ) | v ( 20. 12 ) = exp cj + vw :, j exp 0 + exp { } { cj + vw :, j } ( 20. 13 ) = σ cj + vw :, j. ( 20. 14 ) we can now express the full conditional


##w :, j exp 0 + exp { } { cj + vw :, j } ( 20. 13 ) = σ cj + vw :, j. ( 20. 14 ) we can now express the full conditional over the hidden layer as the factorial distribution : p ( ) = h v | nh j = 1 σ ( 2 1 ) ( + h − c wv ) j. ( 20. 15 ) a similar derivation will show that the other condition of interest to us, p ( v h | ), is also a factorial distribution : p ( ) = v h | nv i = 1 σ ( ( 2 1 ) ( + ) ) v − b w h i. ( 20. 16 ) 20. 2. 2 training restricted boltzmann machines because the rbm admits [UNK] evaluation and [UNK] of [UNK] ( v ) and [UNK] mcmc sampling in the form of block gibbs sampling, it can readily be trained with any of the techniques described in chapter for training models 18 that have intractable partition functions. this includes cd, sml ( pcd ), ratio matching and so on. compared to other undirected models used in deep learning, the rbm is relatively straightforward to train because we can compute


models 18 that have intractable partition functions. this includes cd, sml ( pcd ), ratio matching and so on. compared to other undirected models used in deep learning, the rbm is relatively straightforward to train because we can compute p ( h | v ) 659


chapter 20. deep generative models exactly in closed form. some other deep models, such as the deep boltzmann machine, combine both the [UNK] of an intractable partition function and the [UNK] of intractable inference. 20. 3 deep belief networks deep belief networks ( dbns ) were one of the ﬁrst non - convolutional models to successfully admit training of deep architectures ( hinton 2006 hinton et al., ;, 2007b ). the introduction of deep belief networks in 2006 began the current deep learning renaissance. prior to the introduction of deep belief networks, deep models were considered too [UNK] to optimize. kernel machines with convex objective functions dominated the research landscape. deep belief networks demonstrated that deep architectures can be successful, by outperforming kernelized support vector machines on the mnist dataset (, ). today, deep belief hinton et al. 2006 networks have mostly fallen out of favor and are rarely used, even compared to other unsupervised or generative learning algorithms, but they are still deservedly recognized for their important role in deep learning history. deep belief networks are generative models with several layers of latent variables. the latent variables are typically binary, while the visible units may be binary


##tive learning algorithms, but they are still deservedly recognized for their important role in deep learning history. deep belief networks are generative models with several layers of latent variables. the latent variables are typically binary, while the visible units may be binary or real. there are no intralayer connections. usually, every unit in each layer is connected to every unit in each neighboring layer, though it is possible to construct more sparsely connected dbns. the connections between the top two layers are undirected. the connections between all other layers are directed, with the arrows pointed toward the layer that is closest to the data. see ﬁgure b for an example. 20. 1 a dbn with l hidden layers contains l weight matrices : w ( 1 ),..., w ( ) l. it also contains l + 1 bias vectors : b ( 0 ),..., b ( ) l, with b ( 0 ) providing the biases for the visible layer. the probability distribution represented by the dbn is given by p ( h ( ) l, h ( 1 ) l− ) exp [UNK] b ( ) l h ( ) l + b ( 1 ) l− h ( 1 ) l− + h ( 1 ) l− w


##n is given by p ( h ( ) l, h ( 1 ) l− ) exp [UNK] b ( ) l h ( ) l + b ( 1 ) l− h ( 1 ) l− + h ( 1 ) l− w ( ) l h ( ) l, ( 20. 17 ) p h ( ( ) k i = 1 | h ( + 1 ) k ) = σ b ( ) k i + w ( + 1 ) k :, i h ( + 1 ) k [UNK] ∈ − i, k 1,..., l 2, ( 20. 18 ) p v ( i = 1 | h ( 1 ) ) = σ b ( 0 ) i + w ( 1 ) :, i h ( 1 ) [UNK]. ( 20. 19 ) in the case of real - valued visible units, substitute v [UNK] v b ; ( 0 ) + w ( 1 ) h ( 1 ), β −1 ( 20. 20 ) 660


chapter 20. deep generative models with β diagonal for tractability. generalizations to other exponential family visible units are straightforward, at least in theory. a dbn with only one hidden layer is just an rbm. to generate a sample from a dbn, we ﬁrst run several steps of gibbs sampling on the top two hidden layers. this stage is essentially drawing a sample from the rbm deﬁned by the top two hidden layers. we can then use a single pass of ancestral sampling through the rest of the model to draw a sample from the visible units. deep belief networks incur many of the problems associated with both directed models and undirected models. inference in a deep belief network is intractable due to the explaining away [UNK] within each directed layer, and due to the interaction between the two hidden layers that have undirected connections. evaluating or maximizing the standard evidence lower bound on the log - likelihood is also intractable, because the evidence lower bound takes the expectation of cliques whose size is equal to the network width. evaluating or maximizing the log - likelihood requires not just confronting the problem of intractable inference to marginalize out the latent variables, but also the problem of an intractable partition function within the undirected model of the


to the network width. evaluating or maximizing the log - likelihood requires not just confronting the problem of intractable inference to marginalize out the latent variables, but also the problem of an intractable partition function within the undirected model of the top two layers. to train a deep belief network, one begins by training an rbm to maximize [UNK] log p ( v ) using contrastive divergence or stochastic maximum likelihood. the parameters of the rbm then deﬁne the parameters of the ﬁrst layer of the dbn. next, a second rbm is trained to approximately maximize [UNK] ( 1 ) [UNK] ( 1 ) ( h ( 1 ) | v ) log p ( 2 ) ( h ( 1 ) ) ( 20. 21 ) where p ( 1 ) is the probability distribution represented by the ﬁrst rbm and p ( 2 ) is the probability distribution represented by the second rbm. in other words, the second rbm is trained to model the distribution deﬁned by sampling the hidden units of the ﬁrst rbm, when the ﬁrst rbm is driven by the data. this procedure can be repeated indeﬁnitely, to add as many layers to the dbn as desired, with each new rbm modeling


units of the ﬁrst rbm, when the ﬁrst rbm is driven by the data. this procedure can be repeated indeﬁnitely, to add as many layers to the dbn as desired, with each new rbm modeling the samples of the previous one. each rbm deﬁnes another layer of the dbn. this procedure can be justiﬁed as increasing a variational lower bound on the log - likelihood of the data under the dbn ( hinton et al., ). 2006 in most applications, no [UNK] is made to jointly train the dbn after the greedy layer - wise procedure is complete. however, it is possible to perform generative ﬁne - tuning using the wake - sleep algorithm. 661


chapter 20. deep generative models the trained dbn may be used directly as a generative model, but most of the interest in dbns arose from their ability to improve classiﬁcation models. we can take the weights from the dbn and use them to deﬁne an mlp : h ( 1 ) = σ b ( 1 ) + v w ( 1 ). ( 20. 22 ) h ( ) l = σ b ( ) l i + h ( 1 ) l− w ( ) l [UNK] l 2,..., m, ( 20. 23 ) after initializing this mlp with the weights and biases learned via generative training of the dbn, we may train the mlp to perform a classiﬁcation task. this additional training of the mlp is an example of discriminative ﬁne - tuning. this speciﬁc choice of mlp is somewhat arbitrary, compared to many of the inference equations in chapter that are derived from ﬁrst principles. this mlp 19 is a heuristic choice that seems to work well in practice and is used consistently in the literature. many approximate inference techniques are motivated by their ability to ﬁnd a maximally variational lower bound on the log - likelihood


. this mlp 19 is a heuristic choice that seems to work well in practice and is used consistently in the literature. many approximate inference techniques are motivated by their ability to ﬁnd a maximally variational lower bound on the log - likelihood tight under some set of constraints. one can construct a variational lower bound on the log - likelihood using the hidden unit expectations deﬁned by the dbn ’ s mlp, but this is true of probability distribution over the hidden units, and there is no any reason to believe that this mlp provides a particularly tight bound. in particular, the mlp ignores many important interactions in the dbn graphical model. the mlp propagates information upward from the visible units to the deepest hidden units, but does not propagate any information downward or sideways. the dbn graphical model has explaining away interactions between all of the hidden units within the same layer as well as top - down interactions between layers. while the log - likelihood of a dbn is intractable, it may be approximated with ais ( salakhutdinov and murray 2008, ). this permits evaluating its quality as a generative model. the term “ deep belief network ” is commonly used incorrectly to refer to any kind of deep neural


it may be approximated with ais ( salakhutdinov and murray 2008, ). this permits evaluating its quality as a generative model. the term “ deep belief network ” is commonly used incorrectly to refer to any kind of deep neural network, even networks without latent variable semantics. the term “ deep belief network ” should refer speciﬁcally to models with undirected connections in the deepest layer and directed connections pointing downward between all other pairs of consecutive layers. the term “ deep belief network ” may also cause some confusion because the term “ belief network ” is sometimes used to refer to purely directed models, while deep belief networks contain an undirected layer. deep belief networks also share the acronym dbn with dynamic bayesian networks ( dean and kanazawa 1989, ), which are bayesian networks for representing markov chains. 662


chapter 20. deep generative models h ( 1 ) 1 h ( 1 ) 1 h ( 1 ) 2 h ( 1 ) 2 h ( 1 ) 3 h ( 1 ) 3 v1 v1 v2 v2 v3 v3 h ( 2 ) 1 h ( 2 ) 1 h ( 2 ) 2 h ( 2 ) 2 h ( 2 ) 3 h ( 2 ) 3 h ( 1 ) 4 h ( 1 ) 4 figure 20. 2 : the graphical model for a deep boltzmann machine with one visible layer ( bottom ) and two hidden layers. connections are only between units in neighboring layers. there are no intralayer layer connections. 20. 4 deep boltzmann machines a deep boltzmann machine or dbm ( salakhutdinov and hinton 2009a, ) is another kind of deep, generative model. unlike the deep belief network ( dbn ), it is an entirely undirected model. unlike the rbm, the dbm has several layers of latent variables ( rbms have just one ). but like the rbm, within each layer, each of the variables are mutually independent, conditioned on the variables in the neighboring layers. see ﬁgure for the graph structure. deep boltzmann 20


variables ( rbms have just one ). but like the rbm, within each layer, each of the variables are mutually independent, conditioned on the variables in the neighboring layers. see ﬁgure for the graph structure. deep boltzmann 20. 2 machines have been applied to a variety of tasks including document modeling ( srivastava 2013 et al., ). like rbms and dbns, dbms typically contain only binary units — as we assume for simplicity of our presentation of the model — but it is straightforward to include real - valued visible units. a dbm is an energy - based model, meaning that the the joint probability distribution over the model variables is parametrized by an energy function e. in the case of a deep boltzmann machine with one visible layer, v, and three hidden layers, h ( 1 ), h ( 2 ) and h ( 3 ), the joint probability is given by : p v h, ( 1 ), h ( 2 ), h ( 3 ) = 1 z ( ) θ exp −e, ( v h ( 1 ), h ( 2 ), h ( 3 ) ; ) θ. ( 20. 24 ) to simplify our presentation, we omit the bias parameters below


) = 1 z ( ) θ exp −e, ( v h ( 1 ), h ( 2 ), h ( 3 ) ; ) θ. ( 20. 24 ) to simplify our presentation, we omit the bias parameters below. the dbm energy function is then deﬁned as follows : e, ( v h ( 1 ), h ( 2 ), h ( 3 ) ; ) = θ −vw ( 1 ) h ( 1 ) −h ( 1 ) w ( 2 ) h ( 2 ) −h ( 2 ) w ( 3 ) h ( 3 ). ( 20. 25 ) 663


chapter 20. deep generative models h ( 1 ) 1 h ( 1 ) 1 h ( 1 ) 2 h ( 1 ) 2 h ( 1 ) 3 h ( 1 ) 3 v1 v1 v2 v2 h ( 2 ) 1 h ( 2 ) 1 h ( 2 ) 2 h ( 2 ) 2 h ( 2 ) 3 h ( 2 ) 3 h ( 3 ) 1 h ( 3 ) 1 h ( 3 ) 2 h ( 3 ) 2 v1 v2 h ( 2 ) 1 h ( 2 ) 1 h ( 2 ) 2 h ( 2 ) 2 h ( 2 ) 3 h ( 2 ) 3 h ( 1 ) 1 h ( 1 ) 1 h ( 1 ) 2 h ( 1 ) 2 h ( 1 ) 3 h ( 1 ) 3 h ( 3 ) 1 h ( 3 ) 1 h ( 3 ) 2 h ( 3 ) 2 figure 20. 3 : a deep boltzmann machine, re - arranged to reveal its bipartite graph structure. in comparison to the rbm energy function ( equation ), the dbm energy 20. 5 function includes connections between the hidden units ( latent variables ) in the form of the weight matrices ( w ( 2 ) and w ( 3 ) ). as we will


the rbm energy function ( equation ), the dbm energy 20. 5 function includes connections between the hidden units ( latent variables ) in the form of the weight matrices ( w ( 2 ) and w ( 3 ) ). as we will see, these connections have signiﬁcant consequences for both the model behavior as well as how we go about performing inference in the model. in comparison to fully connected boltzmann machines ( with every unit con - nected to every other unit ), the dbm [UNK] some advantages that are similar to those [UNK] by the rbm. speciﬁcally, as illustrated in ﬁgure, the dbm 20. 3 layers can be organized into a bipartite graph, with odd layers on one side and even layers on the other. this immediately implies that when we condition on the variables in the even layer, the variables in the odd layers become conditionally independent. of course, when we condition on the variables in the odd layers, the variables in the even layers also become conditionally independent. the bipartite structure of the dbm means that we can apply the same equa - tions we have previously used for the conditional distributions of an rbm to determine the conditional distributions in a dbm


even layers also become conditionally independent. the bipartite structure of the dbm means that we can apply the same equa - tions we have previously used for the conditional distributions of an rbm to determine the conditional distributions in a dbm. the units within a layer are conditionally independent from each other given the values of the neighboring layers, so the distributions over binary variables can be fully described by the bernoulli parameters giving the probability of each unit being active. in our example with two hidden layers, the activation probabilities are given by : p v ( i = 1 | h ( 1 ) ) = σ w ( 1 ) i, : h ( 1 ), ( 20. 26 ) 664


chapter 20. deep generative models p h ( ( 1 ) i = 1 | v h, ( 2 ) ) = σ vw ( 1 ) :, i + w ( 2 ) i, : h ( 2 ) ( 20. 27 ) and p h ( ( 2 ) k = 1 | h ( 1 ) ) = σ h ( 1 ) w ( 2 ) :, k. ( 20. 28 ) the bipartite structure makes gibbs sampling in a deep boltzmann machine [UNK]. the naive approach to gibbs sampling is to update only one variable at a time. rbms allow all of the visible units to be updated in one block and all of the hidden units to be updated in a second block. one might naively assume that a dbm with l layers requires l + 1 updates, with each iteration updating a block consisting of one layer of units. instead, it is possible to update all of the units in only two iterations. gibbs sampling can be divided into two blocks of updates, one including all even layers ( including the visible layer ) and the other including all odd layers. due to the bipartite dbm connection pattern, given the even layers, the distribution over the odd layers is factorial and thus can be sampled


, one including all even layers ( including the visible layer ) and the other including all odd layers. due to the bipartite dbm connection pattern, given the even layers, the distribution over the odd layers is factorial and thus can be sampled simultaneously and independently as a block. likewise, given the odd layers, the even layers can be sampled simultaneously and independently as a block. [UNK] sampling is especially important for training with the stochastic maximum likelihood algorithm. 20. 4. 1 interesting properties deep boltzmann machines have many interesting properties. dbms were developed after dbns. compared to dbns, the posterior distribu - tion p ( h v | ) is simpler for dbms. somewhat counterintuitively, the simplicity of this posterior distribution allows richer approximations of the posterior. in the case of the dbn, we perform classiﬁcation using a heuristically motivated approximate inference procedure, in which we guess that a reasonable value for the mean ﬁeld expectation of the hidden units can be provided by an upward pass through the network in an mlp that uses sigmoid activation functions and the same weights as the original dbn. distribution any q ( h ) may be used to obtain a variational lower bound on the


units can be provided by an upward pass through the network in an mlp that uses sigmoid activation functions and the same weights as the original dbn. distribution any q ( h ) may be used to obtain a variational lower bound on the log - likelihood. this heuristic procedure therefore allows us to obtain such a bound. however, the bound is not explicitly optimized in any way, so the bound may be far from tight. in particular, the heuristic estimate of q ignores interactions between hidden units within the same layer as well as the top - down feedback inﬂuence of hidden units in deeper layers on hidden units that are closer to the input. because the heuristic mlp - based inference procedure in the dbn is not able to account for these interactions, the resulting q is presumably far 665


chapter 20. deep generative models from optimal. in dbms, all of the hidden units within a layer are conditionally independent given the other layers. this lack of intralayer interaction makes it possible to use ﬁxed point equations to actually optimize the variational lower bound and ﬁnd the true optimal mean ﬁeld expectations ( to within some numerical tolerance ). the use of proper mean ﬁeld allows the approximate inference procedure for dbms to capture the inﬂuence of top - down feedback interactions. this makes dbms interesting from the point of view of neuroscience, because the human brain is known to use many top - down feedback connections. because of this property, dbms have been used as computational models of real neuroscientiﬁc phenomena (, ; series et al. 2010 reichert 2011 et al., ). one unfortunate property of dbms is that sampling from them is relatively [UNK]. dbns only need to use mcmc sampling in their top pair of layers. the other layers are used only at the end of the sampling process, in one [UNK] ancestral sampling pass. to generate a sample from a dbm, it is necessary to use mcmc across all layers, with every layer of the model participating in every markov chain transition


used only at the end of the sampling process, in one [UNK] ancestral sampling pass. to generate a sample from a dbm, it is necessary to use mcmc across all layers, with every layer of the model participating in every markov chain transition. 20. 4. 2 dbm mean field inference the conditional distribution over one dbm layer given the neighboring layers is factorial. in the example of the dbm with two hidden layers, these distributions are p ( v h | ( 1 ) ), p ( h ( 1 ) | v h, ( 2 ) ) and p ( h ( 2 ) | h ( 1 ) ). the distribution over all hidden layers generally does not factorize because of interactions between layers. in the example with two hidden layers, p ( h ( 1 ), h ( 2 ) | v ) does not factorize due due to the interaction weights w ( 2 ) between h ( 1 ) and h ( 2 ) which render these variables mutually dependent. as was the case with the dbn, we are left to seek out methods to approximate the dbm posterior distribution. however, unlike the dbn, the dbm posterior distribution over their hidden units — while complicated — is easy to approximate with a variational approximation ( as discussed


dbn, we are left to seek out methods to approximate the dbm posterior distribution. however, unlike the dbn, the dbm posterior distribution over their hidden units — while complicated — is easy to approximate with a variational approximation ( as discussed in section ), speciﬁcally a 19. 4 mean ﬁeld approximation. the mean ﬁeld approximation is a simple form of variational inference, where we restrict the approximating distribution to fully factorial distributions. in the context of dbms, the mean ﬁeld equations capture the bidirectional interactions between layers. in this section we derive the iterative approximate inference procedure originally introduced in salakhutdinov and hinton ( ). 2009a in variational approximations to inference, we approach the task of approxi - 666


chapter 20. deep generative models mating a particular target distribution — in our case, the posterior distribution over the hidden units given the visible units — by some reasonably simple family of dis - tributions. in the case of the mean ﬁeld approximation, the approximating family is the set of distributions where the hidden units are conditionally independent. we now develop the mean ﬁeld approach for the example with two hidden layers. let q ( h ( 1 ), h ( 2 ) | v ) be the approximation of p ( h ( 1 ), h ( 2 ) | v ). the mean ﬁeld assumption implies that q ( h ( 1 ), h ( 2 ) | v ) = j q h ( ( 1 ) j | v ) k q h ( ( 2 ) k | v ). ( 20. 29 ) the mean ﬁeld approximation attempts to ﬁnd a member of this family of distributions that best ﬁts the true posterior p ( h ( 1 ), h ( 2 ) | v ). importantly, the inference process must be run again to ﬁnd a [UNK] distribution q every time we use a new value of. v one can conceive of many ways of measuring how well q ( h v | ) ﬁts p (


). importantly, the inference process must be run again to ﬁnd a [UNK] distribution q every time we use a new value of. v one can conceive of many ways of measuring how well q ( h v | ) ﬁts p ( h v | ). the mean ﬁeld approach is to minimize kl ( ) = q p h q ( h ( 1 ), h ( 2 ) | v ) log q ( h ( 1 ), h ( 2 ) | v ) p ( h ( 1 ), h ( 2 ) | v ). ( 20. 30 ) in general, we do not have to provide a parametric form of the approximating distribution beyond enforcing the independence assumptions. the variational approximation procedure is generally able to recover a functional form of the approximate distribution. however, in the case of a mean ﬁeld assumption on binary hidden units ( the case we are developing here ) there is no loss of generality resulting from ﬁxing a parametrization of the model in advance. we parametrize q as a product of bernoulli distributions, that is we associate the probability of each element of h ( 1 ) with a parameter. speciﬁcally, for each j, [UNK] ( 1 ) j = q (


we parametrize q as a product of bernoulli distributions, that is we associate the probability of each element of h ( 1 ) with a parameter. speciﬁcally, for each j, [UNK] ( 1 ) j = q ( h ( 1 ) j = 1 | v ), where [UNK] ( 1 ) j ∈ [ 0, 1 ] and for each k, [UNK] ( 2 ) k = q ( h ( 2 ) k = 1 | v ), where [UNK] ( 2 ) k ∈ [ 0 1 ],. thus we have the following approximation to the posterior : q ( h ( 1 ), h ( 2 ) | v ) = j q h ( ( 1 ) j | v ) k q h ( ( 2 ) k | v ) ( 20. 31 ) = j ( [UNK] ( 1 ) j ) h ( 1 ) j ( 1 [UNK] ( 1 ) j ) ( 1−h ( 1 ) j ) × k ( [UNK] ( 2 ) k ) h ( 2 ) k ( 1 [UNK] ( 2 ) k ) ( 1−h ( 2 ) k ). ( 20. 32 ) of course, for dbms with more layers the approximate posterior parametrization can be extended in the obvious way, exploiting the


k ( 1 [UNK] ( 2 ) k ) ( 1−h ( 2 ) k ). ( 20. 32 ) of course, for dbms with more layers the approximate posterior parametrization can be extended in the obvious way, exploiting the bipartite structure of the graph 667


chapter 20. deep generative models to update all of the even layers simultaneously and then to update all of the odd layers simultaneously, following the same schedule as gibbs sampling. now that we have speciﬁed our family of approximating distributions q, it remains to specify a procedure for choosing the member of this family that best ﬁts p. the most straightforward way to do this is to use the mean ﬁeld equations speciﬁed by equation. these equations were derived by solving for where the 19. 56 derivatives of the variational lower bound are zero. they describe in an abstract manner how to optimize the variational lower bound for any model, simply by taking expectations with respect to. q applying these general equations, we obtain the update rules ( again, ignoring bias terms ) : [UNK] ( 1 ) j = σ i viw ( 1 ) i, j + k w ( 2 ) j, [UNK] ( 2 ) k, j [UNK] ( 20. 33 ) [UNK] ( 2 ) k = σ j w ( 2 ) j, [UNK] ( 1 ) j, k. [UNK] ( 20. 34 ) at a ﬁxed point of this system of equations, we have a local maximum of the variational lower bound l ( q ). thus these ﬁxed


( 2 ) j, [UNK] ( 1 ) j, k. [UNK] ( 20. 34 ) at a ﬁxed point of this system of equations, we have a local maximum of the variational lower bound l ( q ). thus these ﬁxed point update equations deﬁne an iterative algorithm where we alternate updates of [UNK] ( 1 ) j ( using equation ) and 20. 33 updates of [UNK] ( 2 ) k ( using equation ). on small problems such as mnist, as few 20. 34 as ten iterations can be [UNK] to ﬁnd an approximate positive phase gradient for learning, and ﬁfty usually [UNK] to obtain a high quality representation of a single speciﬁc example to be used for high - accuracy classiﬁcation. extending approximate variational inference to deeper dbms is straightforward. 20. 4. 3 dbm parameter learning learning in the dbm must confront both the challenge of an intractable partition function, using the techniques from chapter, and the challenge of an intractable 18 posterior distribution, using the techniques from chapter. 19 as described in section, variational inference allows the construction of 20. 4. 2 a distribution q ( h v | ) that approximates the intractable p ( h v | ). learning then proceeds


distribution, using the techniques from chapter. 19 as described in section, variational inference allows the construction of 20. 4. 2 a distribution q ( h v | ) that approximates the intractable p ( h v | ). learning then proceeds by maximizing l ( v θ, q, ), the variational lower bound on the intractable log - likelihood,. log ( ; ) p v θ 668


chapter 20. deep generative models for a deep boltzmann machine with two hidden layers, is given by l l ( ) = q, θ i j viw ( 1 ) i, [UNK] ( 1 ) j + j k [UNK] ( 1 ) j w ( 2 ) j, [UNK] ( 2 ) k− h log ( ) + z θ ( ) q. ( 20. 35 ) this expression still contains the log partition function, log z ( θ ). because a deep boltzmann machine contains restricted boltzmann machines as components, the hardness results for computing the partition function and sampling that apply to restricted boltzmann machines also apply to deep boltzmann machines. this means that evaluating the probability mass function of a boltzmann machine requires approximate methods such as annealed importance sampling. likewise, training the model requires approximations to the gradient of the log partition function. see chapter for a general description of these methods. dbms are typically trained 18 using stochastic maximum likelihood. many of the other techniques described in chapter are not applicable. techniques such as pseudolikelihood require the 18 ability to evaluate the unnormalized probabilities, rather than merely obtain a variational lower bound on them. contrastive divergence is slow for


other techniques described in chapter are not applicable. techniques such as pseudolikelihood require the 18 ability to evaluate the unnormalized probabilities, rather than merely obtain a variational lower bound on them. contrastive divergence is slow for deep boltzmann machines because they do not allow [UNK] sampling of the hidden units given the visible units — instead, contrastive divergence would require burning in a markov chain every time a new negative phase sample is needed. the non - variational version of stochastic maximum likelihood algorithm was discussed earlier, in section. variational stochastic maximum likelihood as 18. 2 applied to the dbm is given in algorithm. recall that we describe a simpliﬁed 20. 1 varient of the dbm that lacks bias parameters ; including them is trivial. 20. 4. 4 layer - wise pretraining unfortunately, training a dbm using stochastic maximum likelihood ( as described above ) from a random initialization usually results in failure. in some cases, the model fails to learn to represent the distribution adequately. in other cases, the dbm may represent the distribution well, but with no higher likelihood than could be obtained with just an rbm. a dbm with very small weights in all but


, the model fails to learn to represent the distribution adequately. in other cases, the dbm may represent the distribution well, but with no higher likelihood than could be obtained with just an rbm. a dbm with very small weights in all but the ﬁrst layer represents approximately the same distribution as an rbm. various techniques that permit joint training have been developed and are described in section. however, the original and most popular method for 20. 4. 5 overcoming the joint training problem of dbms is greedy layer - wise pretraining. in this method, each layer of the dbm is trained in isolation as an rbm. the ﬁrst layer is trained to model the input data. each subsequent rbm is trained to model samples from the previous rbm ’ s posterior distribution. after all of the 669


chapter 20. deep generative models algorithm 20. 1 the variational stochastic maximum likelihood algorithm for training a dbm with two hidden layers. set, the step size, to a small positive number set k, the number of gibbs steps, high enough to allow a markov chain of p ( v h, ( 1 ), h ( 2 ) ; θ + ∆θ ) to burn in, starting from samples from p ( v h, ( 1 ), h ( 2 ) ; θ ). initialize three matrices, [UNK], [UNK] h ( 1 ) and [UNK] h ( 2 ) each with m rows set to random values ( e. g., from bernoulli distributions, possibly with marginals matched to the model ’ s marginals ). while not converged ( learning loop ) do sample a minibatch of m examples from the training data and arrange them as the rows of a design matrix. v initialize matrices [UNK] h ( 1 ) and [UNK] h ( 2 ), possibly to the model ’ s marginals. while not converged ( mean ﬁeld inference loop ) do [UNK] h ( 1 ) ←σ v w ( 1 ) + [UNK] h ( 2 ) w ( 2 ). [UNK] h ( 2 ) ←σ [UNK] h


model ’ s marginals. while not converged ( mean ﬁeld inference loop ) do [UNK] h ( 1 ) ←σ v w ( 1 ) + [UNK] h ( 2 ) w ( 2 ). [UNK] h ( 2 ) ←σ [UNK] h ( 1 ) w ( 2 ). end while ∆w ( 1 ) ←1 mv [UNK] h ( 1 ) ∆w ( 2 ) ←1 m [UNK] h ( 1 ) [UNK] ( 2 ) for do l k = 1 to ( gibbs sampling ) gibbs block 1 : [UNK], j, [UNK] i, j sampled from p ( [UNK], j = 1 ) = σ w ( 1 ) j, : [UNK] ( 1 ) i, :. [UNK], j, [UNK] ( 2 ) i, j sampled from p ( [UNK] ( 2 ) i, j = 1 ) = σ [UNK] h ( 1 ) i, : w ( 2 ) :, j. gibbs block 2 : [UNK], j, [UNK] ( 1 ) i, j sampled from p ( [UNK] ( 1 ) i, j = 1 ) = σ [UNK], : w ( 1 ) :, j + [UNK] h ( 2 ) i, : w ( 2 ) j, :. end for ∆w ( 1 ) ←∆w ( 1 ) −1


i, j = 1 ) = σ [UNK], : w ( 1 ) :, j + [UNK] h ( 2 ) i, : w ( 2 ) j, :. end for ∆w ( 1 ) ←∆w ( 1 ) −1 mv [UNK] h ( 1 ) ∆w ( 2 ) ←∆w ( 2 ) −1 m [UNK] h ( 1 ) [UNK] h ( 2 ) w ( 1 ) ←w ( 1 ) + ∆w ( 1 ) ( this is a cartoon illustration, in practice use a more [UNK] algorithm, such as momentum with a decaying learning rate ) w ( 2 ) ←w ( 2 ) + ∆ w ( 2 ) end while 670


chapter 20. deep generative models rbms have been trained in this way, they can be combined to form a dbm. the dbm may then be trained with pcd. typically pcd training will make only a small change in the model ’ s parameters and its performance as measured by the log - likelihood it assigns to the data, or its ability to classify inputs. see ﬁgure 20. 4 for an illustration of the training procedure. this greedy layer - wise training procedure is not just coordinate ascent. it bears some passing resemblance to coordinate ascent because we optimize one subset of the parameters at each step. the two methods [UNK] because the greedy layer - wise training procedure uses a [UNK] objective function at each step. greedy layer - wise pretraining of a dbm [UNK] from greedy layer - wise pre - training of a dbn. the parameters of each individual rbm may be copied to the corresponding dbn directly. in the case of the dbm, the rbm parameters must be modiﬁed before inclusion in the dbm. a layer in the middle of the stack of rbms is trained with only bottom - up input, but after the stack is combined to form the dbm, the layer will have both bottom - up and top


##ed before inclusion in the dbm. a layer in the middle of the stack of rbms is trained with only bottom - up input, but after the stack is combined to form the dbm, the layer will have both bottom - up and top - down input. to account for this [UNK], salakhutdinov and hinton 2009a ( ) advocate dividing the weights of all but the top and bottom rbm in half before inserting them into the dbm. additionally, the bottom rbm must be trained using two “ copies ” of each visible unit and the weights tied to be equal between the two copies. this means that the weights are [UNK] doubled during the upward pass. similarly, the top rbm should be trained with two copies of the topmost layer. obtaining the state of the art results with the deep boltzmann machine requires a modiﬁcation of the standard sml algorithm, which is to use a small amount of mean ﬁeld during the negative phase of the joint pcd training step ( salakhutdinov and hinton 2009a, ). speciﬁcally, the expectation of the energy gradient should be computed with respect to the mean ﬁeld distribution in which all of the units are independent from each other. the


( salakhutdinov and hinton 2009a, ). speciﬁcally, the expectation of the energy gradient should be computed with respect to the mean ﬁeld distribution in which all of the units are independent from each other. the parameters of this mean ﬁeld distribution should be obtained by running the mean ﬁeld ﬁxed point equations for just one step. see ( ) for a comparison of the performance of centered goodfellow et al. 2013b dbms with and without the use of partial mean ﬁeld in the negative phase. 20. 4. 5 jointly training deep boltzmann machines classic dbms require greedy unsupervised pretraining, and to perform classiﬁcation well, require a separate mlp - based classiﬁer on top of the hidden features they extract. this has some undesirable properties. it is hard to track performance during training because we cannot evaluate properties of the full dbm while training the ﬁrst rbm. thus, it is hard to tell how well our hyperparameters 671


chapter 20. deep generative models d ) a ) b ) c ) figure 20. 4 : the deep boltzmann machine training procedure used to classify the mnist dataset ( salakhutdinov and hinton 2009a srivastava 2014, ; et al., ). train an rbm ( a ) by using cd to approximately maximize log p ( v ). train a second rbm that models ( b ) h ( 1 ) and target class y by using cd - k to approximately maximize log p ( h ( 1 ), y ) where h ( 1 ) is drawn from the ﬁrst rbm ’ s posterior conditioned on the data. increasek from 1 to 20 during learning. combine the two rbms into a dbm. train it to approximately ( c ) maximize log p ( v, y ) using stochastic maximum likelihood with k = 5. delete ( d ) y from the model. deﬁne a new set of features h ( 1 ) and h ( 2 ) that are obtained by running mean ﬁeld inference in the model lacking y. use these features as input to an mlp whose structure is the same as an additional pass of mean ﬁeld, with an additional output layer for the estimate of y


2 ) that are obtained by running mean ﬁeld inference in the model lacking y. use these features as input to an mlp whose structure is the same as an additional pass of mean ﬁeld, with an additional output layer for the estimate of y. initialize the mlp ’ s weights to be the same as the dbm ’ s weights. train the mlp to approximately maximize log p ( y | v ) using stochastic gradient descent and dropout. figure reprinted from (, ). goodfellow et al. 2013b 672


chapter 20. deep generative models are working until quite late in the training process. software implementations of dbms need to have many [UNK] components for cd training of individual rbms, pcd training of the full dbm, and training based on back - propagation through the mlp. finally, the mlp on top of the boltzmann machine loses many of the advantages of the boltzmann machine probabilistic model, such as being able to perform inference when some input values are missing. there are two main ways to resolve the joint training problem of the deep boltzmann machine. the ﬁrst is the centered deep boltzmann machine ( montavon and muller 2012, ), which reparametrizes the model in order to make the hessian of the cost function better - conditioned at the beginning of the learning process. this yields a model that can be trained without a greedy layer - wise pretraining stage. the resulting model obtains excellent test set log - likelihood and produces high quality samples. unfortunately, it remains unable to compete with appropriately regularized mlps as a classiﬁer. the second way to jointly train a deep boltzmann machine is to use a multi - prediction deep boltzmann machine ( goodfellow 2013


. unfortunately, it remains unable to compete with appropriately regularized mlps as a classiﬁer. the second way to jointly train a deep boltzmann machine is to use a multi - prediction deep boltzmann machine ( goodfellow 2013b et al., ). this model uses an alternative training criterion that allows the use of the back - propagation algorithm in order to avoid the problems with mcmc estimates of the gradient. unfortunately, the new criterion does not lead to good likelihood or samples, but, compared to the mcmc approach, it does lead to superior classiﬁcation performance and ability to reason well about missing inputs. the centering trick for the boltzmann machine is easiest to describe if we return to the general view of a boltzmann machine as consisting of a set of units x with a weight matrix u and biases b. recall from equation that he energy 20. 2 function is given by e ( ) = x −xux b − x. ( 20. 36 ) using [UNK] sparsity patterns in the weight matrix u, we can implement structures of boltzmann machines, such as rbms, or dbms with [UNK] numbers of layers. this is accomplished by partitioning x into visible and hidden units and zeroing