the eigenvalues that are not equal to to decay to zero. 1 under some additional mild conditions, a is guaranteed to have only one eigenvector with eigenvalue. the process thus converges to a 1 stationary distribution, sometimes also called the. at convergence, equilibrium distribution v = = av v, ( 17. 23 ) and this same condition holds for every additional step. this is an eigenvector equation. to be a stationary point, v must be an eigenvector with corresponding eigenvalue. this condition guarantees that once we have reached the stationary 1 distribution, repeated applications of the transition sampling procedure do not change the over the states of all the various markov chains ( although distribution transition operator does change each individual state, of course ). if we have chosen t correctly, then the stationary distribution q will be equal to the distribution p we wish to sample from. we will describe how to choose t shortly, in section. 17. 4 most properties of markov chains with countable states can be generalized to continuous variables. in this situation, some authors call the markov chain a harris chain but we use the term markov chain to describe both conditions. in general, a markov chain with transition operator t will


with countable states can be generalized to continuous variables. in this situation, some authors call the markov chain a harris chain but we use the term markov chain to describe both conditions. in general, a markov chain with transition operator t will converge, under mild conditions, to a ﬁxed point described by the equation q ( x ) = [UNK] ( x | x ), ( 17. 24 ) which in the discrete case is just rewriting equation. when 17. 23 x is discrete, the expectation corresponds to a sum, and when x is continuous, the expectation corresponds to an integral. 597

/home/ricoiban/GEMMA/mnlp_chatsplaining/RAG/Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville (z-lib.org).pdf


chapter 17. monte carlo methods regardless of whether the state is continuous or discrete, all markov chain methods consist of repeatedly applying stochastic updates until eventually the state begins to yield samples from the equilibrium distribution. running the markov chain until it reaches its equilibrium distribution is called “ burning in ” the markov chain. after the chain has reached equilibrium, a sequence of inﬁnitely many samples may be drawn from from the equilibrium distribution. they are identically distributed but any two successive samples will be highly correlated with each other. a ﬁnite sequence of samples may thus not be very representative of the equilibrium distribution. one way to mitigate this problem is to return only every n successive samples, so that our estimate of the statistics of the equilibrium distribution is not as biased by the correlation between an mcmc sample and the next several samples. markov chains are thus expensive to use because of the time required to burn in to the equilibrium distribution and the time required to transition from one sample to another reasonably decorrelated sample after reaching equilibrium. if one desires truly independent samples, one can run multiple markov chains in parallel. this approach uses extra parallel computation to eliminate latency. the strategy of using only a single markov chain to generate all samples and the strategy


sample after reaching equilibrium. if one desires truly independent samples, one can run multiple markov chains in parallel. this approach uses extra parallel computation to eliminate latency. the strategy of using only a single markov chain to generate all samples and the strategy of using one markov chain for each desired sample are two extremes ; deep learning practitioners usually use a number of chains that is similar to the number of examples in a minibatch and then draw as many samples as are needed from this ﬁxed set of markov chains. a commonly used number of markov chains is 100. another [UNK] is that we do not know in advance how many steps the markov chain must run before reaching its equilibrium distribution. this length of time is called the mixing time. it is also very [UNK] to test whether a markov chain has reached equilibrium. we do not have a precise enough theory for guiding us in answering this question. theory tells us that the chain will converge, but not much more. if we analyze the markov chain from the point of view of a matrix a acting on a vector of probabilities v, then we know that the chain mixes when at has [UNK] lost all of the eigenvalues from a besides the unique eigenvalue of. 1 this means


point of view of a matrix a acting on a vector of probabilities v, then we know that the chain mixes when at has [UNK] lost all of the eigenvalues from a besides the unique eigenvalue of. 1 this means that the magnitude of the second largest eigenvalue will determine the mixing time. however, in practice, we cannot actually represent our markov chain in terms of a matrix. the number of states that our probabilistic model can visit is exponentially large in the number of variables, so it is infeasible to represent v, a, or the eigenvalues of a. due to these and other obstacles, we usually do not know whether a markov chain has mixed. instead, we simply run the markov chain for an amount of time that we roughly estimate to be [UNK], and use heuristic methods to determine whether the chain has mixed. these heuristic methods include manually inspecting samples or measuring correlations between 598


chapter 17. monte carlo methods successive samples. 17. 4 gibbs sampling so far we have described how to draw samples from a distribution q ( x ) by repeatedly updating x x ← [UNK] ( x | x ). however, we have not described how to ensure that q ( x ) is a useful distribution. two basic approaches are considered in this book. the ﬁrst one is to derive t from a given learned pmodel, described below with the case of sampling from ebms. the second one is to directly parametrize t and learn it, so that its stationary distribution implicitly deﬁnes the pmodel of interest. examples of this second approach are discussed in sections and. 20. 12 20. 13 in the context of deep learning, we commonly use markov chains to draw samples from an energy - based model deﬁning a distribution pmodel ( x ). in this case, we want the q ( x ) for the markov chain to be pmodel ( x ). to obtain the desired q ( ) x, we must choose an appropriate t ( x | x ). a conceptually simple and [UNK] approach to building a markov chain that samples from pmodel ( x ) is to use gibbs sampling,


. to obtain the desired q ( ) x, we must choose an appropriate t ( x | x ). a conceptually simple and [UNK] approach to building a markov chain that samples from pmodel ( x ) is to use gibbs sampling, in which sampling from t ( x | x ) is accomplished by selecting one variable xi and sampling it from pmodel conditioned on its neighbors in the undirected graph g deﬁning the structure of the energy - based model. it is also possible to sample several variables at the same time so long as they are conditionally independent given all of their neighbors. as shown in the rbm example in section, all of the hidden units of an 16. 7. 1 rbm may be sampled simultaneously because they are conditionally independent from each other given all of the visible units. likewise, all of the visible units may be sampled simultaneously because they are conditionally independent from each other given all of the hidden units. gibbs sampling approaches that update many variables simultaneously in this way are called block gibbs sampling. alternate approaches to designing markov chains to sample from pmodel are possible. for example, the metropolis - hastings algorithm is widely used in other disciplines. in the context of the deep learning approach to undirected modeling,


called block gibbs sampling. alternate approaches to designing markov chains to sample from pmodel are possible. for example, the metropolis - hastings algorithm is widely used in other disciplines. in the context of the deep learning approach to undirected modeling, it is rare to use any approach other than gibbs sampling. improved sampling techniques are one possible research frontier. 17. 5 the challenge of mixing between separated modes the primary [UNK] involved with mcmc methods is that they have a tendency to mix poorly. ideally, successive samples from a markov chain designed to sample 599


chapter 17. monte carlo methods from p ( x ) would be completely independent from each other and would visit many [UNK] regions in x space proportional to their probability. instead, especially in high dimensional cases, mcmc samples become very correlated. we refer to such behavior as slow mixing or even failure to mix. mcmc methods with slow mixing can be seen as inadvertently performing something resembling noisy gradient descent on the energy function, or equivalently noisy hill climbing on the probability, with respect to the state of the chain ( the random variables being sampled ). the chain tends to take small steps ( in the space of the state of the markov chain ), from a conﬁguration x ( 1 ) t− to a conﬁguration x ( ) t, with the energy e ( x ( ) t ) generally lower or approximately equal to the energy e ( x ( 1 ) t− ), with a preference for moves that yield lower energy conﬁgurations. when starting from a rather improbable conﬁguration ( higher energy than the typical ones from p ( x ) ), the chain tends to gradually reduce the energy of the state and only occasionally move to another mode. once the chain has found a region of low energy ( for example, if


##guration ( higher energy than the typical ones from p ( x ) ), the chain tends to gradually reduce the energy of the state and only occasionally move to another mode. once the chain has found a region of low energy ( for example, if the variables are pixels in an image, a region of low energy might be a connected manifold of images of the same object ), which we call a mode, the chain will tend to walk around that mode ( following a kind of random walk ). once in a while it will step out of that mode and generally return to it or ( if it ﬁnds an escape route ) move towards another mode. the problem is that successful escape routes are rare for many interesting distributions, so the markov chain will continue to sample the same mode longer than it should. this is very clear when we consider the gibbs sampling algorithm ( section ). 17. 4 in this context, consider the probability of going from one mode to a nearby mode within a given number of steps. what will determine that probability is the shape of the “ energy barrier ” between these modes. transitions between two modes that are separated by a high energy barrier ( a region of low probability ) are exponentially less likely ( in terms of the height of the energy barrier )


that probability is the shape of the “ energy barrier ” between these modes. transitions between two modes that are separated by a high energy barrier ( a region of low probability ) are exponentially less likely ( in terms of the height of the energy barrier ). this is illustrated in ﬁgure. the problem arises when there are multiple modes with 17. 1 high probability that are separated by regions of low probability, especially when each gibbs sampling step must update only a small subset of variables whose values are largely determined by the other variables. as a simple example, consider an energy - based model over two variables a and b, which are both binary with a sign, taking on values −1 1 and. if e ( a b, ) = −wab for some large positive number w, then the model expresses a strong belief that a and b have the same sign. consider updating b using a gibbs sampling step with a = 1. the conditional distribution over b is given by p ( b = 1 | a = 1 ) = σ ( w ). if w is large, the sigmoid saturates, and the probability of also assigning b to be 1 is close to 1. likewise, if a = −1, the probability of assigning b to


= σ ( w ). if w is large, the sigmoid saturates, and the probability of also assigning b to be 1 is close to 1. likewise, if a = −1, the probability of assigning b to be −1 is close to 1. according to pmodel ( a b, ), both signs of both variables are equally likely. 600


chapter 17. monte carlo methods figure 17. 1 : paths followed by gibbs sampling for three distributions, with the markov chain initialized at the mode in both cases. ( left ) a multivariate normal distribution with two independent variables. gibbs sampling mixes well because the variables are independent. a multivariate normal distribution with highly correlated variables. ( center ) the correlation between variables makes it [UNK] for the markov chain to mix. because the update for each variable must be conditioned on the other variable, the correlation reduces the rate at which the markov chain can move away from the starting point. ( right ) a mixture of gaussians with widely separated modes that are not axis - aligned. gibbs sampling mixes very slowly because it is [UNK] to change modes while altering only one variable at a time. according to pmodel ( a b | ), both variables should have the same sign. this means that gibbs sampling will only very rarely ﬂip the signs of these variables. in more practical scenarios, the challenge is even greater because we care not only about making transitions between two modes but more generally between all the many modes that a real model might contain. if several such transitions are [UNK] because of the [UNK] of mixing between modes, then it becomes very expensive to obtain a


greater because we care not only about making transitions between two modes but more generally between all the many modes that a real model might contain. if several such transitions are [UNK] because of the [UNK] of mixing between modes, then it becomes very expensive to obtain a reliable set of samples covering most of the modes, and convergence of the chain to its stationary distribution is very slow. sometimes this problem can be resolved by ﬁnding groups of highly dependent units and updating all of them simultaneously in a block. unfortunately, when the dependencies are complicated, it can be computationally intractable to draw a sample from the group. after all, the problem that the markov chain was originally introduced to solve is this problem of sampling from a large group of variables. in the context of models with latent variables, which deﬁne a joint distribution pmodel ( x h, ), we often draw samples of x by alternating between sampling from pmodel ( x h | ) and sampling from pmodel ( h x | ). from the point of view of mixing 601


chapter 17. monte carlo methods figure 17. 2 : an illustration of the slow mixing problem in deep probabilistic models. each panel should be read left to right, top to bottom. ( left ) consecutive samples from gibbs sampling applied to a deep boltzmann machine trained on the mnist dataset. consecutive samples are similar to each other. because the gibbs sampling is performed in a deep graphical model, this similarity is based more on semantic rather than raw visual features, but it is still [UNK] for the gibbs chain to transition from one mode of the distribution to another, for example by changing the digit identity. consecutive ( right ) ancestral samples from a generative adversarial network. because ancestral sampling generates each sample independently from the others, there is no mixing problem. rapidly, we would like pmodel ( h x | ) to have very high entropy. however, from the point of view of learning a useful representation of h, we would like h to encode enough information about x to reconstruct it well, which implies that h and x should have very high mutual information. these two goals are at odds with each other. we often learn generative models that very precisely encode x into h but are not able to mix very well. this


it well, which implies that h and x should have very high mutual information. these two goals are at odds with each other. we often learn generative models that very precisely encode x into h but are not able to mix very well. this situation arises frequently with boltzmann machines — the sharper the distribution a boltzmann machine learns, the harder it is for a markov chain sampling from the model distribution to mix well. this problem is illustrated in ﬁgure. 17. 2 all this could make mcmc methods less useful when the distribution of interest has a manifold structure with a separate manifold for each class : the distribution is concentrated around many modes and these modes are separated by vast regions of high energy. this type of distribution is what we expect in many classiﬁcation problems and would make mcmc methods converge very slowly because of poor mixing between modes. 602


chapter 17. monte carlo methods 17. 5. 1 tempering to mix between modes when a distribution has sharp peaks of high probability surrounded by regions of low probability, it is [UNK] to mix between the [UNK] modes of the distribution. several techniques for faster mixing are based on constructing alternative versions of the target distribution in which the peaks are not as high and the surrounding valleys are not as low. energy - based models provide a particularly simple way to do so. so far, we have described an energy - based model as deﬁning a probability distribution p e. ( ) exp ( x [UNK] − ( ) ) x ( 17. 25 ) energy - based models may be augmented with an extra parameter β controlling how sharply peaked the distribution is : pβ ( ) exp ( ( ) ) x [UNK] −βe x. ( 17. 26 ) the β parameter is often described as being the reciprocal of the temperature, reﬂecting the origin of energy - based models in statistical physics. when the temperature falls to zero and rises to inﬁnity, the energy - based model becomes β deterministic. when the temperature rises to inﬁnity and β falls to zero, the distribution ( for discrete ) becomes uniform. x typically, a model is trained to be


and rises to inﬁnity, the energy - based model becomes β deterministic. when the temperature rises to inﬁnity and β falls to zero, the distribution ( for discrete ) becomes uniform. x typically, a model is trained to be evaluated at β = 1. however, we can make use of other temperatures, particularly those where β < 1. tempering is a general strategy of mixing between modes of p1 rapidly by drawing samples with. β < 1 markov chains based on tempered transitions (, ) temporarily neal 1994 sample from higher - temperature distributions in order to mix to [UNK] modes, then resume sampling from the unit temperature distribution. these techniques have been applied to models such as rbms ( salakhutdinov 2010, ). another approach is to use parallel tempering (, ), in which the markov chain iba 2001 simulates many [UNK] states in parallel, at [UNK] temperatures. the highest temperature states mix slowly, while the lowest temperature states, at temperature 1, provide accurate samples from the model. the transition operator includes stochastically swapping states between two [UNK] temperature levels, so that a [UNK] high - probability sample from a high - temperature slot can jump into a lower temperature slot. this approach has also been applied to rb


the model. the transition operator includes stochastically swapping states between two [UNK] temperature levels, so that a [UNK] high - probability sample from a high - temperature slot can jump into a lower temperature slot. this approach has also been applied to rbms ( desjardins et al. et al., ; 2010 cho, ). although tempering is a promising approach, at 2010 this point it has not allowed researchers to make a strong advance in solving the challenge of sampling from complex ebms. one possible reason is that there are critical temperatures around which the temperature transition must be very slow ( as the temperature is gradually reduced ) in order for tempering to be [UNK]. 603


chapter 17. monte carlo methods 17. 5. 2 depth may help mixing when drawing samples from a latent variable model p ( h x, ), we have seen that if p ( h x | ) encodes x too well, then sampling from p ( x h | ) will not change x very much and mixing will be poor. one way to resolve this problem is to make h be a deep representation, that encodes into in such a way that a markov chain in x h the space of h can mix more easily. many representation learning algorithms, such as autoencoders and rbms, tend to yield a marginal distribution over h that is more uniform and more unimodal than the original data distribution over x. it can be argued that this arises from trying to minimize reconstruction error while using all of the available representation space, because minimizing reconstruction error over the training examples will be better achieved when [UNK] training examples are easily distinguishable from each other in h - space, and thus well separated. bengio 2013a et al. ( ) observed that deeper stacks of regularized autoencoders or rbms yield marginal distributions in the top - level h - space that appeared more spread out and more uniform, with less of a gap between the


bengio 2013a et al. ( ) observed that deeper stacks of regularized autoencoders or rbms yield marginal distributions in the top - level h - space that appeared more spread out and more uniform, with less of a gap between the regions corresponding to [UNK] modes ( categories, in the experiments ). training an rbm in that higher - level space allowed gibbs sampling to mix faster between modes. it remains however unclear how to exploit this observation to help better train and sample from deep generative models. despite the [UNK] of mixing, monte carlo techniques are useful and are often the best tool available. indeed, they are the primary tool used to confront the intractable partition function of undirected models, discussed next. 604


chapter 18 confronting the partition function in section we saw that many probabilistic models ( commonly known as undi - 16. 2. 2 rected graphical models ) are deﬁned by an unnormalized probability distribution [UNK] ( x ; θ ). we must normalize [UNK] by dividing by a partition function z ( θ ) in order to obtain a valid probability distribution : p ( ; ) = x θ 1 z ( ) θ [UNK]. ( ; ) x θ ( 18. 1 ) the partition function is an integral ( for continuous variables ) or sum ( for discrete variables ) over the unnormalized probability of all states : [UNK] d ( ) x x ( 18. 2 ) or x [UNK]. ( ) x ( 18. 3 ) this operation is intractable for many interesting models. as we will see in chapter, several deep learning models are designed to 20 have a tractable normalizing constant, or are designed to be used in ways that do not involve computing p ( x ) at all. however, other models directly confront the challenge of intractable partition functions. in this chapter, we describe techniques used for training and evaluating models that have intractable partition functions. 605


chapter 18. confronting the partition function 18. 1 the log - likelihood gradient what makes learning undirected models by maximum likelihood particularly [UNK] is that the partition function depends on the parameters. the gradient of the log - likelihood with respect to the parameters has a term corresponding to the gradient of the partition function : ∇θ log ( ; ) = p x θ ∇θ log [UNK] ( ; ) x θ −∇θ log ( ) z θ. ( 18. 4 ) this is a well - known decomposition into the positive phase and negative phase of learning. for most undirected models of interest, the negative phase is [UNK]. models with no latent variables or with few interactions between latent variables typically have a tractable positive phase. the quintessential example of a model with a straightforward positive phase and [UNK] negative phase is the rbm, which has hidden units that are conditionally independent from each other given the visible units. the case where the positive phase is [UNK], with complicated interactions between latent variables, is primarily covered in chapter. this chapter focuses 19 on the [UNK] of the negative phase. let us look more closely at the gradient of : log z ∇θ log z ( 18. 5 ) = ∇θz z ( 18. 6 ) = ∇


primarily covered in chapter. this chapter focuses 19 on the [UNK] of the negative phase. let us look more closely at the gradient of : log z ∇θ log z ( 18. 5 ) = ∇θz z ( 18. 6 ) = ∇θ x [UNK] ( ) x z ( 18. 7 ) = x [UNK] ( ) x z. ( 18. 8 ) for models that guarantee p ( x ) > 0 for all x, we can substitute exp ( log [UNK] ( ) ) x for [UNK] ( ) x : x ∇θexp ( log [UNK] ( ) ) x z ( 18. 9 ) = x exp ( log [UNK] ( ) ) x ∇θ log [UNK] ( ) x z ( 18. 10 ) = x [UNK] ( ) x ∇θ log [UNK] ( ) x z ( 18. 11 ) = x p ( ) x ∇θ log [UNK] ( ) x ( 18. 12 ) 606


chapter 18. confronting the partition function = ex x [UNK] ( ) ∇θ log [UNK]. ( ) x ( 18. 13 ) this derivation made use of summation over discrete x, but a similar result applies using integration over continuous x. in the continuous version of the derivation, we use leibniz ’ s rule for [UNK] under the integral sign to obtain the identity ∇θ [UNK] d ( ) x x = ∇θ [UNK] d. ( ) x x ( 18. 14 ) this identity is applicable only under certain regularity conditions on [UNK] and [UNK] ( x ). in measure theoretic terms, the conditions are : ( i ) the unnormalized distribution [UNK] must be a lebesgue - integrable function of x for every value of θ ; ( ii ) the gradient [UNK] ( x ) must exist for all θ and almost all x ; ( iii ) there must exist an integrable function r ( x ) that bounds [UNK] ( x ) in the sense that maxi | ∂ ∂θ i [UNK] ( x ) | ≤r ( x ) for all θ and almost all x. fortunately, most machine learning models of interest have these properties. this identity ∇θ log = z ex x [UNK] ( ) ∇θ log [UNK] ( ) x ( 18


( x ) | ≤r ( x ) for all θ and almost all x. fortunately, most machine learning models of interest have these properties. this identity ∇θ log = z ex x [UNK] ( ) ∇θ log [UNK] ( ) x ( 18. 15 ) is the basis for a variety of monte carlo methods for approximately maximizing the likelihood of models with intractable partition functions. the monte carlo approach to learning undirected models provides an intuitive framework in which we can think of both the positive phase and the negative phase. in the positive phase, we increase log [UNK] ( x ) for x drawn from the data. in the negative phase, we decrease the partition function by decreasing log [UNK] ( x ) drawn from the model distribution. in the deep learning literature, it is common to parametrize log [UNK] in terms of an energy function ( equation ). in this case, we can interpret the positive 16. 7 phase as pushing down on the energy of training examples and the negative phase as pushing up on the energy of samples drawn from the model, as illustrated in ﬁgure. 18. 1 18. 2 stochastic maximum likelihood and contrastive divergence the naive way of implementing equation is to compute it by burning in 18. 15 a set of markov


samples drawn from the model, as illustrated in ﬁgure. 18. 1 18. 2 stochastic maximum likelihood and contrastive divergence the naive way of implementing equation is to compute it by burning in 18. 15 a set of markov chains from a random initialization every time the gradient is needed. when learning is performed using stochastic gradient descent, this means the chains must be burned in once per gradient step. this approach leads to the 607


chapter 18. confronting the partition function training procedure presented in algorithm. the high cost of burning in the 18. 1 markov chains in the inner loop makes this procedure computationally infeasible, but this procedure is the starting point that other more practical algorithms aim to approximate. algorithm 18. 1 a naive mcmc algorithm for maximizing the log - likelihood with an intractable partition function using gradient ascent. set, the step size, to a small positive number. set k, the number of gibbs steps, high enough to allow burn in. perhaps 100 to train an rbm on a small image patch. while not converged do sample a minibatch of examples m { x ( 1 ),..., x ( ) m } from the training set. g ← 1 m m i = 1 ∇θ log [UNK] ( x ( ) i ; ) θ. initialize a set of m samples { [UNK] ( 1 ),..., [UNK] ( ) m } to random values ( e. g., from a uniform or normal distribution, or possibly a distribution with marginals matched to the model ’ s marginals ). for do i k = 1 to for do j m = 1 to [UNK] ( ) j ←gibbs _ update (


., from a uniform or normal distribution, or possibly a distribution with marginals matched to the model ’ s marginals ). for do i k = 1 to for do j m = 1 to [UNK] ( ) j ←gibbs _ update ( [UNK] ( ) j ). end for end for g g ← − 1 m m i = 1 ∇θ log [UNK] ( [UNK] ( ) i ; ) θ. θ θ ← +. g end while we can view the mcmc approach to maximum likelihood as trying to achieve balance between two forces, one pushing up on the model distribution where the data occurs, and another pushing down on the model distribution where the model samples occur. figure illustrates this process. the two forces correspond to 18. 1 maximizing log [UNK] and minimizing log z. several approximations to the negative phase are possible. each of these approximations can be understood as making the negative phase computationally cheaper but also making it push down in the wrong locations. because the negative phase involves drawing samples from the model ’ s distri - bution, we can think of it as ﬁnding points that the model believes in strongly. because the negative phase acts to reduce the probability of those points, they are generally considered to represent the model ’ s incorrect


model ’ s distri - bution, we can think of it as ﬁnding points that the model believes in strongly. because the negative phase acts to reduce the probability of those points, they are generally considered to represent the model ’ s incorrect beliefs about the world. they are frequently referred to in the literature as “ hallucinations ” or “ fantasy particles. ” in fact, the negative phase has been proposed as a possible explanation 608


chapter 18. confronting the partition function x p ( x ) the positive phase pmodel ( ) x pdata ( ) x x p ( x ) the negative phase pmodel ( ) x pdata ( ) x figure 18. 1 : the view of algorithm as having a “ positive phase ” and “ negative phase. ” 18. 1 ( left ) in the positive phase, we sample points from the data distribution, and push up on their unnormalized probability. this means points that are likely in the data get pushed up on more. ( right ) in the negative phase, we sample points from the model distribution, and push down on their unnormalized probability. this counteracts the positive phase ’ s tendency to just add a large constant to the unnormalized probability everywhere. when the data distribution and the model distribution are equal, the positive phase has the same chance to push up at a point as the negative phase has to push down. when this occurs, there is no longer any gradient ( in expectation ) and training must terminate. for dreaming in humans and other animals ( crick and mitchison 1983, ), the idea being that the brain maintains a probabilistic model of the world and follows the gradient of log [UNK]


gradient ( in expectation ) and training must terminate. for dreaming in humans and other animals ( crick and mitchison 1983, ), the idea being that the brain maintains a probabilistic model of the world and follows the gradient of log [UNK] while experiencing real events while awake and follows the negative gradient of log [UNK] to minimize log z while sleeping and experiencing events sampled from the current model. this view explains much of the language used to describe algorithms with a positive and negative phase, but it has not been proven to be correct with neuroscientiﬁc experiments. in machine learning models, it is usually necessary to use the positive and negative phase simultaneously, rather than in separate time periods of wakefulness and rem sleep. as we will see in section, other machine learning algorithms draw samples from the model 19. 5 distribution for other purposes and such algorithms could also provide an account for the function of dream sleep. given this understanding of the role of the positive and negative phase of learning, we can attempt to design a less expensive alternative to algorithm. 18. 1 the main cost of the naive mcmc algorithm is the cost of burning in the markov chains from a random initialization at each step. a natural solution is to initialize the markov chains from


a less expensive alternative to algorithm. 18. 1 the main cost of the naive mcmc algorithm is the cost of burning in the markov chains from a random initialization at each step. a natural solution is to initialize the markov chains from a distribution that is very close to the model distribution, 609


chapter 18. confronting the partition function so that the burn in operation does not take as many steps. the contrastive divergence ( cd, or cd - k to indicate cd with k gibbs steps ) algorithm initializes the markov chain at each step with samples from the data distribution ( hinton 2000 2010,, ). this approach is presented as algorithm. 18. 2 obtaining samples from the data distribution is free, because they are already available in the data set. initially, the data distribution is not close to the model distribution, so the negative phase is not very accurate. fortunately, the positive phase can still accurately increase the model ’ s probability of the data. after the positive phase has had some time to act, the model distribution is closer to the data distribution, and the negative phase starts to become accurate. algorithm 18. 2 the contrastive divergence algorithm, using gradient ascent as the optimization procedure. set, the step size, to a small positive number. set k, the number of gibbs steps, high enough to allow a markov chain sampling from p ( x ; θ ) to mix when initialized from pdata. perhaps 1 - 20 to train an rbm on a small image patch. while not converged do sample a minibatch of examples


to allow a markov chain sampling from p ( x ; θ ) to mix when initialized from pdata. perhaps 1 - 20 to train an rbm on a small image patch. while not converged do sample a minibatch of examples m { x ( 1 ),..., x ( ) m } from the training set. g ← 1 m m i = 1 ∇θ log [UNK] ( x ( ) i ; ) θ. for do i m = 1 to [UNK] ( ) i ←x ( ) i. end for for do i k = 1 to for do j m = 1 to [UNK] ( ) j ←gibbs _ update ( [UNK] ( ) j ). end for end for g g ← − 1 m m i = 1 ∇θ log [UNK] ( [UNK] ( ) i ; ) θ. θ θ ← +. g end while of course, cd is still an approximation to the correct negative phase. the main way that cd qualitatively fails to implement the correct negative phase is that it fails to suppress regions of high probability that are far from actual training examples. these regions that have high probability under the model but low probability under the data generating distribution are called spurious modes. figure illustrates why this happens. essentially, it


is that it fails to suppress regions of high probability that are far from actual training examples. these regions that have high probability under the model but low probability under the data generating distribution are called spurious modes. figure illustrates why this happens. essentially, it is because modes in the 18. 2 model distribution that are far from the data distribution will not be visited by 610


chapter 18. confronting the partition function x p ( x ) pmodel ( ) x pdata ( ) x figure 18. 2 : an illustration of how the negative phase of contrastive divergence ( algo - rithm ) can fail to suppress spurious modes. a spurious mode is a mode that is 18. 2 present in the model distribution but absent in the data distribution. because contrastive divergence initializes its markov chains from data points and runs the markov chain for only a few steps, it is unlikely to visit modes in the model that are far from the data points. this means that when sampling from the model, we will sometimes get samples that do not resemble the data. it also means that due to wasting some of its probability mass on these modes, the model will struggle to place high probability mass on the correct modes. for the purpose of visualization, this ﬁgure uses a somewhat simpliﬁed concept of distance — the spurious mode is far from the correct mode along the number line in r. this corresponds to a markov chain based on making local moves with a single x variable in r. for most deep probabilistic models, the markov chains are based on gibbs sampling and can make non


mode along the number line in r. this corresponds to a markov chain based on making local moves with a single x variable in r. for most deep probabilistic models, the markov chains are based on gibbs sampling and can make non - local moves of individual variables but cannot move all of the variables simultaneously. for these problems, it is usually better to consider the edit distance between modes, rather than the euclidean distance. however, edit distance in a high dimensional space is [UNK] to depict in a 2 - d plot. markov chains initialized at training points, unless is very large. k carreira - perpinan and hinton 2005 ( ) showed experimentally that the cd estimator is biased for rbms and fully visible boltzmann machines, in that it converges to [UNK] points than the maximum likelihood estimator. they argue that because the bias is small, cd could be used as an inexpensive way to initialize a model that could later be ﬁne - tuned via more expensive mcmc methods. bengio and delalleau 2009 ( ) showed that cd can be interpreted as discarding the smallest terms of the correct mcmc update gradient, which explains the bias. cd is useful for training shallow models like rbms. these can


##mc methods. bengio and delalleau 2009 ( ) showed that cd can be interpreted as discarding the smallest terms of the correct mcmc update gradient, which explains the bias. cd is useful for training shallow models like rbms. these can in turn be stacked to initialize deeper models like dbns or dbms. however, cd does not provide much help for training deeper models directly. this is because it is [UNK] 611


chapter 18. confronting the partition function to obtain samples of the hidden units given samples of the visible units. since the hidden units are not included in the data, initializing from training points cannot solve the problem. even if we initialize the visible units from the data, we will still need to burn in a markov chain sampling from the distribution over the hidden units conditioned on those visible samples. the cd algorithm can be thought of as penalizing the model for having a markov chain that changes the input rapidly when the input comes from the data. this means training with cd somewhat resembles autoencoder training. even though cd is more biased than some of the other training methods, it can be useful for pretraining shallow models that will later be stacked. this is because the earliest models in the stack are encouraged to copy more information up to their latent variables, thereby making it available to the later models. this should be thought of more of as an often - exploitable side [UNK] of cd training rather than a principled design advantage. sutskever and tieleman 2010 ( ) showed that the cd update direction is not the gradient of any function. this allows for situations where cd could cycle forever, but in practice this is not a serious problem. a [UNK]


##d design advantage. sutskever and tieleman 2010 ( ) showed that the cd update direction is not the gradient of any function. this allows for situations where cd could cycle forever, but in practice this is not a serious problem. a [UNK] strategy that resolves many of the problems with cd is to initial - ize the markov chains at each gradient step with their states from the previous gradient step. this approach was ﬁrst discovered under the name stochastic max - imum likelihood ( sml ) in the applied mathematics and statistics community ( younes 1998, ) and later independently rediscovered under the name persistent contrastive divergence ( pcd, or pcd - k to indicate the use of k gibbs steps per update ) in the deep learning community (, ). see algorithm. tieleman 2008 18. 3 the basic idea of this approach is that, so long as the steps taken by the stochastic gradient algorithm are small, then the model from the previous step will be similar to the model from the current step. it follows that the samples from the previous model ’ s distribution will be very close to being fair samples from the current model ’ s distribution, so a markov chain initialized with these samples will not require much time to mix


from the current step. it follows that the samples from the previous model ’ s distribution will be very close to being fair samples from the current model ’ s distribution, so a markov chain initialized with these samples will not require much time to mix. because each markov chain is continually updated throughout the learning process, rather than restarted at each gradient step, the chains are free to wander far enough to ﬁnd all of the model ’ s modes. sml is thus considerably more resistant to forming models with spurious modes than cd is. moreover, because it is possible to store the state of all of the sampled variables, whether visible or latent, sml provides an initialization point for both the hidden and visible units. cd is only able to provide an initialization for the visible units, and therefore requires burn - in for deep models. sml is able to train deep models [UNK]. 612


chapter 18. confronting the partition function marlin 2010 et al. ( ) compared sml to many of the other criteria presented in this chapter. they found that sml results in the best test set log - likelihood for an rbm, and that if the rbm ’ s hidden units are used as features for an svm classiﬁer, sml results in the best classiﬁcation accuracy. sml is vulnerable to becoming inaccurate if the stochastic gradient algorithm can move the model faster than the markov chain can mix between steps. this can happen if k is too small or is too large. the permissible range of values is unfortunately highly problem - dependent. there is no known way to test formally whether the chain is successfully mixing between steps. subjectively, if the learning rate is too high for the number of gibbs steps, the human operator will be able to observe that there is much more variance in the negative phase samples across gradient steps rather than across [UNK] markov chains. for example, a model trained on mnist might sample exclusively 7s on one step. the learning process will then push down strongly on the mode corresponding to 7s, and the model might sample exclusively 9s on the next step. algorithm 18. 3 the stoch


model trained on mnist might sample exclusively 7s on one step. the learning process will then push down strongly on the mode corresponding to 7s, and the model might sample exclusively 9s on the next step. algorithm 18. 3 the stochastic maximum likelihood / persistent contrastive divergence algorithm using gradient ascent as the optimization procedure. set, the step size, to a small positive number. set k, the number of gibbs steps, high enough to allow a markov chain sampling from p ( x ; θ + g ) to burn in, starting from samples from p ( x ; θ ). perhaps 1 for rbm on a small image patch, or 5 - 50 for a more complicated model like a dbm. initialize a set of m samples { [UNK] ( 1 ),..., [UNK] ( ) m } to random values ( e. g., from a uniform or normal distribution, or possibly a distribution with marginals matched to the model ’ s marginals ). while not converged do sample a minibatch of examples m { x ( 1 ),..., x ( ) m } from the training set. g ← 1 m m i = 1 ∇θ log [UNK] ( x ( ) i ; ) θ.


do sample a minibatch of examples m { x ( 1 ),..., x ( ) m } from the training set. g ← 1 m m i = 1 ∇θ log [UNK] ( x ( ) i ; ) θ. for do i k = 1 to for do j m = 1 to [UNK] ( ) j ←gibbs _ update ( [UNK] ( ) j ). end for end for g g ← − 1 m m i = 1 ∇θ log [UNK] ( [UNK] ( ) i ; ) θ. θ θ ← +. g end while care must be taken when evaluating the samples from a model trained with sml. it is necessary to draw the samples starting from a fresh markov chain 613


chapter 18. confronting the partition function initialized from a random starting point after the model is done training. the samples present in the persistent negative chains used for training have been inﬂuenced by several recent versions of the model, and thus can make the model appear to have greater capacity than it actually does. berglund and raiko 2013 ( ) performed experiments to examine the bias and variance in the estimate of the gradient provided by cd and sml. cd proves to have lower variance than the estimator based on exact sampling. sml has higher variance. the cause of cd ’ s low variance is its use of the same training points in both the positive and negative phase. if the negative phase is initialized from [UNK] training points, the variance rises above that of the estimator based on exact sampling. all of these methods based on using mcmc to draw samples from the model can in principle be used with almost any variant of mcmc. this means that techniques such as sml can be improved by using any of the enhanced mcmc techniques described in chapter, such as parallel tempering (, 17 desjardins et al. 2010 cho 2010 ; et al., ). one approach to accelerating mixing during learning relies not on changing the monte carlo sampling


of the enhanced mcmc techniques described in chapter, such as parallel tempering (, 17 desjardins et al. 2010 cho 2010 ; et al., ). one approach to accelerating mixing during learning relies not on changing the monte carlo sampling technology but rather on changing the parametrization of the model and the cost function. fast pcd or fpcd (, tieleman and hinton 2009 ) involves replacing the parameters θ of a traditional model with an expression θ θ = ( ) slow + θ ( ) fast. ( 18. 16 ) there are now twice as many parameters as before, and they are added together element - wise to provide the parameters used by the original model deﬁnition. the fast copy of the parameters is trained with a much larger learning rate, allowing it to adapt rapidly in response to the negative phase of learning and push the markov chain to new territory. this forces the markov chain to mix rapidly, though this [UNK] only occurs during learning while the fast weights are free to change. typically one also applies signiﬁcant weight decay to the fast weights, encouraging them to converge to small values, after only transiently taking on large values long enough to encourage the markov chain to change modes. one key ben


to change. typically one also applies signiﬁcant weight decay to the fast weights, encouraging them to converge to small values, after only transiently taking on large values long enough to encourage the markov chain to change modes. one key beneﬁt to the mcmc - based methods described in this section is that they provide an estimate of the gradient of log z, and thus we can essentially decompose the problem into the log [UNK] contribution and the log z contribution. we can then use any other method to tackle log [UNK] ( x ), and just add our negative phase gradient onto the other method ’ s gradient. in particular, this means that our positive phase can make use of methods that provide only a lower bound on [UNK]. most of the other methods of dealing with log z presented in this chapter are 614


chapter 18. confronting the partition function incompatible with bound - based positive phase methods. 18. 3 pseudolikelihood monte carlo approximations to the partition function and its gradient directly confront the partition function. other approaches sidestep the issue, by training the model without computing the partition function. most of these approaches are based on the observation that it is easy to compute ratios of probabilities in an undirected probabilistic model. this is because the partition function appears in both the numerator and the denominator of the ratio and cancels out : p ( ) x p ( ) y = 1 z [UNK] ( ) x 1 z [UNK] ( ) y = [UNK] ( ) x [UNK] ( ) y. ( 18. 17 ) the pseudolikelihood is based on the observation that conditional probabilities take this ratio - based form, and thus can be computed without knowledge of the partition function. suppose that we partition x into a, b and c, where a contains the variables we want to ﬁnd the conditional distribution over, b contains the variables we want to condition on, and c contains the variables that are not part of our query. p ( ) = a b | p, ( a b ) p ( ) b = p, (


##nd the conditional distribution over, b contains the variables we want to condition on, and c contains the variables that are not part of our query. p ( ) = a b | p, ( a b ) p ( ) b = p, ( a b ) a c, p,, ( a b c ) = [UNK], ( a b ) a c, [UNK],, ( a b c ). ( 18. 18 ) this quantity requires marginalizing out a, which can be a very [UNK] operation provided that a and c do not contain very many variables. in the extreme case, a can be a single variable and c can be empty, making this operation require only as many evaluations of [UNK] as there are values of a single random variable. unfortunately, in order to compute the log - likelihood, we need to marginalize out large sets of variables. if there are n variables total, we must marginalize a set of size. by the chain rule of probability, n −1 log ( ) = log ( p x p x1 ) + log ( p x2 | x1 ) + + ( · · · p xn | x1 : 1 n− ). ( 18. 19 ) in this case, we have made a maximally


( p x p x1 ) + log ( p x2 | x1 ) + + ( · · · p xn | x1 : 1 n− ). ( 18. 19 ) in this case, we have made a maximally small, but c can be as large as x2 : n. what if we simply move c into b to reduce the computational cost? this yields the pseudolikelihood (, ) objective function, based on predicting the value besag 1975 of feature xi given all of the other features x−i : n i = 1 log ( p xi | x−i ). ( 18. 20 ) 615


chapter 18. confronting the partition function if each random variable has k [UNK] values, this requires onlyk n × evaluations of [UNK] to compute, as opposed to the kn evaluations needed to compute the partition function. this may look like an unprincipled hack, but it can be proven that estimation by maximizing the pseudolikelihood is asymptotically consistent (, ). mase 1995 of course, in the case of datasets that do not approach the large sample limit, pseudolikelihood may display [UNK] behavior from the maximum likelihood estimator. it is possible to trade computational complexity for deviation from maximum likelihood behavior by using the generalized pseudolikelihood estimator ( huang and ogata 2002, ). the generalized pseudolikelihood estimator uses m [UNK] sets s ( ) i, i = 1,..., m of indices of variables that appear together on the left side of the conditioning bar. in the extreme case of m = 1 and s ( 1 ) = 1,..., n the generalized pseudolikelihood recovers the log - likelihood. in the extreme case of m = n and s ( ) i = { } i, the generalized pseudolikelihood recovers the


1 ) = 1,..., n the generalized pseudolikelihood recovers the log - likelihood. in the extreme case of m = n and s ( ) i = { } i, the generalized pseudolikelihood recovers the pseudolikelihood. the generalized pseudolikelihood objective function is given by m i = 1 log ( p xs ( ) i | x−s ( ) i ). ( 18. 21 ) the performance of pseudolikelihood - based approaches depends largely on how the model will be used. pseudolikelihood tends to perform poorly on tasks that require a good model of the full joint p ( x ), such as density estimation and sampling. however, it can perform better than maximum likelihood for tasks that require only the conditional distributions used during training, such as ﬁlling in small amounts of missing values. generalized pseudolikelihood techniques are especially powerful if the data has regular structure that allows the s index sets to be designed to capture the most important correlations while leaving out groups of variables that only have negligible correlation. for example, in natural images, pixels that are widely separated in space also have weak correlation, so the generalized pseudolikelihood can be applied with each set being a small,


while leaving out groups of variables that only have negligible correlation. for example, in natural images, pixels that are widely separated in space also have weak correlation, so the generalized pseudolikelihood can be applied with each set being a small, spatially localized window. s one weakness of the pseudolikelihood estimator is that it cannot be used with other approximations that provide only a lower bound on [UNK] ( x ), such as variational inference, which will be covered in chapter. this is because 19 [UNK] appears in the denominator. a lower bound on the denominator provides only an upper bound on the expression as a whole, and there is no beneﬁt to maximizing an upper bound. this makes it [UNK] to apply pseudolikelihood approaches to deep models such as deep boltzmann machines, since variational methods are one of the dominant approaches to approximately marginalizing out the many layers of hidden variables 616


chapter 18. confronting the partition function that interact with each other. however, pseudolikelihood is still useful for deep learning, because it can be used to train single layer models, or deep models using approximate inference methods that are not based on lower bounds. pseudolikelihood has a much greater cost per gradient step than sml, due to its explicit computation of all of the conditionals. however, generalized pseudo - likelihood and similar criteria can still perform well if only one randomly selected conditional is computed per example ( goodfellow 2013b et al., ), thereby bringing the computational cost down to match that of sml. though the pseudolikelihood estimator does not explicitly minimize log z, it can still be thought of as having something resembling a negative phase. the denominators of each conditional distribution result in the learning algorithm suppressing the probability of all states that have only one variable [UNK] from a training example. see marlin and de freitas 2011 ( ) for a theoretical analysis of the asymptotic [UNK] of pseudolikelihood. 18. 4 score matching and ratio matching score matching (, ) provides another consistent means of training a hyvarinen 2005 model without estimating z or its derivatives. the name score matching


the asymptotic [UNK] of pseudolikelihood. 18. 4 score matching and ratio matching score matching (, ) provides another consistent means of training a hyvarinen 2005 model without estimating z or its derivatives. the name score matching comes from terminology in which the derivatives of a log density with respect to its argument, ∇x log p ( x ), are called its score. the strategy used by score matching is to minimize the expected squared [UNK] between the derivatives of the model ’ s log density with respect to the input and the derivatives of the data ’ s log density with respect to the input : l, ( x θ ) = 1 2 | | ∇x log pmodel ( ; ) x θ −∇x log p data ( ) x | | 2 2 ( 18. 22 ) j ( ) = θ 1 2epdata ( ) x l, ( x θ ) ( 18. 23 ) θ ∗ = min θ j ( ) θ ( 18. 24 ) this objective function avoids the [UNK] associated with [UNK] the partition function z because z is not a function of x and therefore ∇xz = 0. initially, score matching appears to have a new [UNK] : computing the score of the data distribution requires knowledge of the true distribution generating


the [UNK] associated with [UNK] the partition function z because z is not a function of x and therefore ∇xz = 0. initially, score matching appears to have a new [UNK] : computing the score of the data distribution requires knowledge of the true distribution generating the training data, pdata. fortunately, minimizing the expected value of is l, ( x θ ) 617


chapter 18. confronting the partition function equivalent to minimizing the expected value of [UNK], ( x θ ) = n j = 1 ∂2 ∂x2 j log pmodel ( ; ) + x θ 1 2 ∂ ∂xj log pmodel ( ; ) x θ 2 ( 18. 25 ) where is the dimensionality of. n x because score matching requires taking derivatives with respect to x, it is not applicable to models of discrete data. however, the latent variables in the model may be discrete. like the pseudolikelihood, score matching only works when we are able to evaluate log [UNK] ( x ) and its derivatives directly. it is not compatible with methods that only provide a lower bound on log [UNK] ( x ), because score matching requires the derivatives and second derivatives of log [UNK] ( x ) and a lower bound conveys no information about its derivatives. this means that score matching cannot be applied to estimating models with complicated interactions between the hidden units, such as sparse coding models or deep boltzmann machines. while score matching can be used to pretrain the ﬁrst hidden layer of a larger model, it has not been applied as a pretraining strategy for the deeper layers of a larger model. this is probably because the


deep boltzmann machines. while score matching can be used to pretrain the ﬁrst hidden layer of a larger model, it has not been applied as a pretraining strategy for the deeper layers of a larger model. this is probably because the hidden layers of such models usually contain some discrete variables. while score matching does not explicitly have a negative phase, it can be viewed as a version of contrastive divergence using a speciﬁc kind of markov chain (, ). the markov chain in this case is not gibbs sampling, but hyvarinen 2007a rather a [UNK] approach that makes local moves guided by the gradient. score matching is equivalent to cd with this type of markov chain when the size of the local moves approaches zero. lyu 2009 ( ) generalized score matching to the discrete case ( but made an error in their derivation that was corrected by ( ) ). marlin et al. 2010 marlin et al. ( ) found that 2010 generalized score matching ( gsm ) does not work in high dimensional discrete spaces where the observed probability of many events is 0. a more successful approach to extending the basic ideas of score matching to discrete data is ratio matching (, ). ratio matching applies hyvarinen 2007b speci


not work in high dimensional discrete spaces where the observed probability of many events is 0. a more successful approach to extending the basic ideas of score matching to discrete data is ratio matching (, ). ratio matching applies hyvarinen 2007b speciﬁcally to binary data. ratio matching consists of minimizing the average over examples of the following objective function : l ( ) rm ( ) = x θ, n j = 1 1 1 + pmodel ( ; ) x θ pmodel ( ( ) ) ; ) f x, j θ 2, ( 18. 26 ) 618


chapter 18. confronting the partition function where returns with the bit at position ﬂipped. ratio matching avoids f, j ( x ) x j the partition function using the same trick as the pseudolikelihood estimator : in a ratio of two probabilities, the partition function cancels out. ( ) marlin et al. 2010 found that ratio matching outperforms sml, pseudolikelihood and gsm in terms of the ability of models trained with ratio matching to denoise test set images. like the pseudolikelihood estimator, ratio matching requires n evaluations of [UNK] per data point, making its computational cost per update roughly n times higher than that of sml. as with the pseudolikelihood estimator, ratio matching can be thought of as pushing down on all fantasy states that have only one variable [UNK] from a training example. since ratio matching applies speciﬁcally to binary data, this means that it acts on all fantasy states within hamming distance 1 of the data. ratio matching can also be useful as the basis for dealing with high - dimensional sparse data, such as word count vectors. this kind of data poses a challenge for mcmc - based methods because the data is extremely expensive to represent in dense


the data. ratio matching can also be useful as the basis for dealing with high - dimensional sparse data, such as word count vectors. this kind of data poses a challenge for mcmc - based methods because the data is extremely expensive to represent in dense format, yet the mcmc sampler does not yield sparse values until the model has learned to represent the sparsity in the data distribution. dauphin and bengio ( ) overcame this issue by designing an unbiased stochastic approximation to 2013 ratio matching. the approximation evaluates only a randomly selected subset of the terms of the objective, and does not require the model to generate complete fantasy samples. see marlin and de freitas 2011 ( ) for a theoretical analysis of the asymptotic [UNK] of ratio matching. 18. 5 denoising score matching in some cases we may wish to regularize score matching, by ﬁtting a distribution psmoothed ( ) = x p data ( ) ( ) y q x y | dy ( 18. 27 ) rather than the true pdata. the distribution q ( x y | ) is a corruption process, usually one that forms by adding a small amount of noise to. x y denoising score matching is especially useful because in practice we usually


27 ) rather than the true pdata. the distribution q ( x y | ) is a corruption process, usually one that forms by adding a small amount of noise to. x y denoising score matching is especially useful because in practice we usually do not have access to the true pdata but rather only an empirical distribution deﬁned by samples from it. any consistent estimator will, given enough capacity, make pmodel into a set of dirac distributions centered on the training points. smoothing by q helps to reduce this problem, at the loss of the asymptotic consistency property 619


chapter 18. confronting the partition function described in section. ( ) introduced a procedure for 5. 4. 5 kingma and lecun 2010 performing regularized score matching with the smoothing distribution q being normally distributed noise. recall from section that several autoencoder training algorithms are 14. 5. 1 equivalent to score matching or denoising score matching. these autoencoder training algorithms are therefore a way of overcoming the partition function problem. 18. 6 noise - contrastive estimation most techniques for estimating models with intractable partition functions do not provide an estimate of the partition function. sml and cd estimate only the gradient of the log partition function, rather than the partition function itself. score matching and pseudolikelihood avoid computing quantities related to the partition function altogether. noise - contrastive estimation ( nce ) ( gutmann and hyvarinen 2010, ) takes a [UNK] strategy. in this approach, the probability distribution estimated by the model is represented explicitly as log pmodel ( ) = log [UNK] x pmodel ( ; ) + x θ c, ( 18. 28 ) where c is explicitly introduced as an approximation of −log z ( θ ). rather than estimating only θ, the noise contrastive estimation procedure treats c as just


x pmodel ( ; ) + x θ c, ( 18. 28 ) where c is explicitly introduced as an approximation of −log z ( θ ). rather than estimating only θ, the noise contrastive estimation procedure treats c as just another parameter and estimates θ and c simultaneously, using the same algorithm for both. the resulting log pmodel ( x ) thus may not correspond exactly to a valid probability distribution, but will become closer and closer to being valid as the estimate of improves. c 1 such an approach would not be possible using maximum likelihood as the criterion for the estimator. the maximum likelihood criterion would choose to set c c arbitrarily high, rather than setting to create a valid probability distribution. nce works by reducing the unsupervised learning problem of estimating p ( x ) to that of learning a probabilistic binary classiﬁer in which one of the categories corresponds to the data generated by the model. this supervised learning problem is constructed in such a way that maximum likelihood estimation in this supervised 1nce is also applicable to problems with a tractable partition function, where there is no need to introduce the extra parameter c. however, it has generated the most interest as a means of estimating models with


way that maximum likelihood estimation in this supervised 1nce is also applicable to problems with a tractable partition function, where there is no need to introduce the extra parameter c. however, it has generated the most interest as a means of estimating models with [UNK] partition functions. 620


chapter 18. confronting the partition function learning problem deﬁnes an asymptotically consistent estimator of the original problem. speciﬁcally, we introduce a second distribution, the noise distribution pnoise ( x ). the noise distribution should be tractable to evaluate and to sample from. we can now construct a model over both x and a new, binary class variable y. in the new joint model, we specify that pjoint ( = 1 ) = y 1 2, ( 18. 29 ) pjoint ( = 1 ) = x | y pmodel ( ) x, ( 18. 30 ) and pjoint ( = 0 ) = x | y pnoise ( ) x. ( 18. 31 ) in other words, y is a switch variable that determines whether we will generate x from the model or from the noise distribution. we can construct a similar joint model of training data. in this case, the switch variable determines whether we draw x from the data or from the noise distribution. formally, ptrain ( y = 1 ) = 1 2, ptrain ( x | y = 1 ) = pdata ( x ), and ptrain ( = 0 ) = x | y pnoise ( )


from the noise distribution. formally, ptrain ( y = 1 ) = 1 2, ptrain ( x | y = 1 ) = pdata ( x ), and ptrain ( = 0 ) = x | y pnoise ( ) x. we can now just use standard maximum likelihood learning on the supervised learning problem of ﬁtting pjoint to ptrain : θ, c = arg max θ, c ex, p [UNK] log pjoint ( ) y | x. ( 18. 32 ) the distribution pjoint is essentially a logistic regression model applied to the [UNK] in log probabilities of the model and the noise distribution : pjoint ( = 1 ) = y | x pmodel ( ) x p model ( ) + x pnoise ( ) x ( 18. 33 ) = 1 1 + pnoise ( ) x pmodel ( ) x ( 18. 34 ) = 1 1 + exp log pnoise ( ) x pmodel ( ) x ( 18. 35 ) = σ −log pnoise ( ) x pmodel ( ) x ( 18. 36 ) = ( log σ pmodel ( ) log x − pnoise ( ) ) x. ( 18


) x ( 18. 35 ) = σ −log pnoise ( ) x pmodel ( ) x ( 18. 36 ) = ( log σ pmodel ( ) log x − pnoise ( ) ) x. ( 18. 37 ) 621


chapter 18. confronting the partition function nce is thus simple to apply so long as log [UNK] is easy to back - propagate through, and, as speciﬁed above, pnoise is easy to evaluate ( in order to evaluate pjoint ) and sample from ( in order to generate the training data ). nce is most successful when applied to problems with few random variables, but can work well even if those random variables can take on a high number of values. for example, it has been successfully applied to modeling the conditional distribution over a word given the context of the word ( mnih and kavukcuoglu, 2013 ). though the word may be drawn from a large vocabulary, there is only one word. when nce is applied to problems with many random variables, it becomes less [UNK]. the logistic regression classiﬁer can reject a noise sample by identifying any one variable whose value is unlikely. this means that learning slows down greatly after pmodel has learned the basic marginal statistics. imagine learning a model of images of faces, using unstructured gaussian noise as pnoise. if pmodel learns about eyes, it can reject almost all unstructured noise samples without having learned anything about


the basic marginal statistics. imagine learning a model of images of faces, using unstructured gaussian noise as pnoise. if pmodel learns about eyes, it can reject almost all unstructured noise samples without having learned anything about other facial features, such as mouths. the constraint that pnoise must be easy to evaluate and easy to sample from can be overly restrictive. when pnoise is simple, most samples are likely to be too obviously distinct from the data to force pmodel to improve noticeably. like score matching and pseudolikelihood, nce does not work if only a lower bound on [UNK] is available. such a lower bound could be used to construct a lower bound on pjoint ( y = 1 | x ), but it can only be used to construct an upper bound on pjoint ( y = 0 | x ), which appears in half the terms of the nce objective. likewise, a lower bound on p noise is not useful, because it provides only an upper bound on pjoint ( = 1 ) y | x. when the model distribution is copied to deﬁne a new noise distribution before each gradient step, nce deﬁnes a procedure called self - contrastive estimation, whose expected


an upper bound on pjoint ( = 1 ) y | x. when the model distribution is copied to deﬁne a new noise distribution before each gradient step, nce deﬁnes a procedure called self - contrastive estimation, whose expected gradient is equivalent to the expected gradient of maximum likelihood (, ). the special case of nce where the noise samples goodfellow 2014 are those generated by the model suggests that maximum likelihood can be interpreted as a procedure that forces a model to constantly learn to distinguish reality from its own evolving beliefs, while noise contrastive estimation achieves some reduced computational cost by only forcing the model to distinguish reality from a ﬁxed baseline ( the noise model ). using the supervised task of classifying between training samples and generated samples ( with the model energy function used in deﬁning the classiﬁer ) to provide a gradient on the model was introduced earlier in various forms ( welling et al., 2003b bengio 2009 ;, ). 622


chapter 18. confronting the partition function noise contrastive estimation is based on the idea that a good generative model should be able to distinguish data from noise. a closely related idea is that a good generative model should be able to generate samples that no classiﬁer can distinguish from data. this idea yields generative adversarial networks ( section ). 20. 10. 4 18. 7 estimating the partition function while much of this chapter is dedicated to describing methods that avoid needing to compute the intractable partition function z ( θ ) associated with an undirected graphical model, in this section we discuss several methods for directly estimating the partition function. estimating the partition function can be important because we require it if we wish to compute the normalized likelihood of data. this is often important in evaluating the model, monitoring training performance, and comparing models to each other. for example, imagine we have two models : model ma deﬁning a probabil - ity distribution pa ( x ; θa ) = 1 za [UNK] ( x ; θa ) and model mb deﬁning a probability distribution pb ( x ; θb ) = 1 zb [UNK] ( x ; θb ). a common way to compare the models is to evaluate


) = 1 za [UNK] ( x ; θa ) and model mb deﬁning a probability distribution pb ( x ; θb ) = 1 zb [UNK] ( x ; θb ). a common way to compare the models is to evaluate and compare the likelihood that both models assign to an i. i. d. test dataset. suppose the test set consists of m examples { x ( 1 ),..., x ( ) m }. if i pa ( x ( ) i ; θa ) > i pb ( x ( ) i ; θb ) or equivalently if i log pa ( x ( ) i ; θa ) − i log pb ( x ( ) i ; θb ) 0 >, ( 18. 38 ) then we say that ma is a better model than mb ( or, at least, it is a better model of the test set ), in the sense that it has a better test log - likelihood. unfortunately, testing whether this condition holds requires knowledge of the partition function. unfortunately, equation seems to require evaluating the log probability that 18. 38 the model assigns to each point, which in turn requires evaluating the partition function. we can simplify the situation slightly by re - arranging equation 18.


knowledge of the partition function. unfortunately, equation seems to require evaluating the log probability that 18. 38 the model assigns to each point, which in turn requires evaluating the partition function. we can simplify the situation slightly by re - arranging equation 18. 38 into a form where we need to know only the ratio of the two model ’ s partition functions : i log pa ( x ( ) i ; θa ) − i log pb ( x ( ) i ; θb ) = i log [UNK] ( x ( ) i ; θa ) [UNK] ( x ( ) i ; θb ) −m log z ( θa ) z ( θb ). ( 18. 39 ) 623


chapter 18. confronting the partition function we can thus determine whether ma is a better model than mb without knowing the partition function of either model but only their ratio. as we will see shortly, we can estimate this ratio using importance sampling, provided that the two models are similar. if, however, we wanted to compute the actual probability of the test data under either ma or mb, we would need to compute the actual value of the partition functions. that said, if we knew the ratio of two partition functions, r = z ( θb ) z ( θa ), and we knew the actual value of just one of the two, say z ( θa ), we could compute the value of the other : z ( θb ) = ( rz θa ) = z ( θb ) z ( θa ) z ( θa ). ( 18. 40 ) a simple way to estimate the partition function is to use a monte carlo method such as simple importance sampling. we present the approach in terms of continuous variables using integrals, but it can be readily applied to discrete variables by replacing the integrals with summation. we use a proposal distribution p0 ( x ) = 1 z 0 [UNK] ( x ) which supports tractable sampling and tractable


continuous variables using integrals, but it can be readily applied to discrete variables by replacing the integrals with summation. we use a proposal distribution p0 ( x ) = 1 z 0 [UNK] ( x ) which supports tractable sampling and tractable evaluation of both the partition function z0 and the unnormalized distribution [UNK] ( ) x. z1 = [UNK] ( ) x dx ( 18. 41 ) = p0 ( ) x p0 ( ) x [UNK] ( ) x dx ( 18. 42 ) = z0 p0 ( ) x [UNK] ( ) x [UNK] ( ) x dx ( 18. 43 ) [UNK] = z0 k k k = 1 [UNK] ( x ( ) k ) [UNK] ( x ( ) k ) s t :.. x ( ) k [UNK] ( 18. 44 ) in the last line, we make a monte carlo estimator, [UNK], of the integral using samples drawn from p0 ( x ) and then weight each sample with the ratio of the unnormalized [UNK] and the proposal p0. we see also that this approach allows us to estimate the ratio between the partition functions as 1 k k k = 1 [UNK] ( x ( ) k ) [UNK] ( x ( ) k ) s t :


##normalized [UNK] and the proposal p0. we see also that this approach allows us to estimate the ratio between the partition functions as 1 k k k = 1 [UNK] ( x ( ) k ) [UNK] ( x ( ) k ) s t :.. x ( ) k [UNK]. ( 18. 45 ) this value can then be used directly to compare two models as described in equation. 18. 39 624


chapter 18. confronting the partition function if the distribution p0 is close to p1, equation can be an [UNK] way of 18. 44 estimating the partition function ( minka 2005, ). unfortunately, most of the time p1 is both complicated ( usually multimodal ) and deﬁned over a high dimensional space. it is [UNK] to ﬁnd a tractable p0 that is simple enough to evaluate while still being close enough to p1 to result in a high quality approximation. if p0 and p1 are not close, most samples from p0 will have low probability under p 1 and therefore make ( relatively ) negligible contribution to the sum in equation. 18. 44 having few samples with signiﬁcant weights in this sum will result in an estimator that is of poor quality due to high variance. this can be understood quantitatively through an estimate of the variance of our estimate [UNK] : [UNK] var [UNK] = z0 k 2 k k = 1 [UNK] ( x ( ) k ) [UNK] ( x ( ) k ) [UNK] 2. ( 18. 46 ) this quantity is largest when there is signiﬁcant deviation in the values of the importance weights [UNK] ( x ( ) k ) [UNK] ( x ( ) k )


) [UNK] ( x ( ) k ) [UNK] 2. ( 18. 46 ) this quantity is largest when there is signiﬁcant deviation in the values of the importance weights [UNK] ( x ( ) k ) [UNK] ( x ( ) k ). we now turn to two related strategies developed to cope with the challeng - ing task of estimating partition functions for complex distributions over high - dimensional spaces : annealed importance sampling and bridge sampling. both start with the simple importance sampling strategy introduced above and both attempt to overcome the problem of the proposal p0 being too far from p1 by introducing intermediate distributions that attempt to between bridge the gap p0 and p1. 18. 7. 1 annealed importance sampling in situations where dkl ( p0p1 ) is large ( i. e., where there is little overlap between p0 and p1 ), a strategy called annealed importance sampling ( ais ) attempts to bridge the gap by introducing intermediate distributions (, ;, jarzynski 1997 neal 2001 ). consider a sequence of distributions pη0,..., pηn, with 0 = η0 < η1 < < · · · ηn−1 < ηn = 1 so that the ﬁrst


##zynski 1997 neal 2001 ). consider a sequence of distributions pη0,..., pηn, with 0 = η0 < η1 < < · · · ηn−1 < ηn = 1 so that the ﬁrst and last distributions in the sequence are p0 and p1 respectively. this approach allows us to estimate the partition function of a multimodal distribution deﬁned over a high - dimensional space ( such as the distribution deﬁned by a trained rbm ). we begin with a simpler model with a known partition function ( such as an rbm with zeroes for weights ) and estimate the ratio between the two model ’ s partition functions. the estimate of this ratio is based on the estimate of the ratios of a sequence of many similar distributions, such as the sequence of rbms with weights interpolating between zero and the learned weights. 625


chapter 18. confronting the partition function we can now write the ratio z1 z0 as z1 z0 = z1 z0 zη 1 zη 1 · · · zηn−1 zηn−1 ( 18. 47 ) = zη 1 z0 zη 2 zη 1 · · · zηn−1 zηn−2 z1 zηn−1 ( 18. 48 ) = n−1 j = 0 zηj + 1 zηj ( 18. 49 ) provided the distributions pηj and pηj + 1, for all 0 ≤ ≤ − j n 1, are [UNK] close, we can reliably estimate each of the factors zηj + 1 zηj using simple importance sampling and then use these to obtain an estimate of z1 z0. where do these intermediate distributions come from? just as the original proposal distribution p0 is a design choice, so is the sequence of distributions pη1... pηn−1. that is, it can be speciﬁcally constructed to suit the problem domain. one general - purpose and popular choice for the intermediate distributions is to use the weighted geometric average of the target distribution p1 and the starting proposal distribution ( for which the partition


, it can be speciﬁcally constructed to suit the problem domain. one general - purpose and popular choice for the intermediate distributions is to use the weighted geometric average of the target distribution p1 and the starting proposal distribution ( for which the partition function is known ) p0 : pη j [UNK] 1 p1−ηj 0 ( 18. 50 ) in order to sample from these intermediate distributions, we deﬁne a series of markov chain transition functionstη j ( x | x ) that deﬁne the conditional probability distribution of transitioning to xgiven we are currently at x. the transition operator tηj ( x | x ) is deﬁned to leave pηj ( ) x invariant : pηj ( ) = x pηj ( x ) tηj ( x x | ) dx ( 18. 51 ) these transitions may be constructed as any markov chain monte carlo method ( e. g., metropolis - hastings, gibbs ), including methods involving multiple passes through all of the random variables or other kinds of iterations. the ais sampling strategy is then to generate samples from p0 and then use the transition operators to sequentially generate samples from the intermediate distributions until we arrive at samples from the target


passes through all of the random variables or other kinds of iterations. the ais sampling strategy is then to generate samples from p0 and then use the transition operators to sequentially generate samples from the intermediate distributions until we arrive at samples from the target distribution p1 : • for k... k = 1 – sample x ( ) k η1 [UNK] ( ) x 626


chapter 18. confronting the partition function – sample x ( ) k η2 [UNK] ( x ( ) k η2 | x ( ) k η1 ) –... – sample x ( ) k ηn−1 [UNK] ( x ( ) k ηn−1 | x ( ) k ηn−2 ) – sample x ( ) k ηn [UNK] ( x ( ) k ηn | x ( ) k ηn−1 ) • end for sample k, we can derive the importance weight by chaining together the importance weights for the jumps between the intermediate distributions given in equation : 18. 49 w ( ) k = [UNK] ( x ( ) k η1 ) [UNK] ( x ( ) k η1 ) [UNK] ( x ( ) k η2 ) [UNK] ( x ( ) k η2 )... [UNK] ( x ( ) k 1 ) [UNK] ηn−1 ( x ( ) k ηn ). ( 18. 52 ) to avoid numerical issues such as overﬂow, it is probably best to compute log w ( ) k by adding and subtracting log probabilities, rather than computing w ( ) k by multiplying and dividing probabilities. with the sampling procedure thus deﬁned and the importance weights given in equation,


compute log w ( ) k by adding and subtracting log probabilities, rather than computing w ( ) k by multiplying and dividing probabilities. with the sampling procedure thus deﬁned and the importance weights given in equation, the estimate of the ratio of partition functions is given by : 18. 52 z1 z0 ≈1 k k k = 1 w ( ) k ( 18. 53 ) in order to verify that this procedure deﬁnes a valid importance sampling scheme, we can show (, ) that the ais procedure corresponds to simple neal 2001 importance sampling on an extended state space with points sampled over the product space [ xη1,..., xηn−1, x1 ]. to do this, we deﬁne the distribution over the extended space as : [UNK] ( xη1,..., xηn−1, x1 ) ( 18. 54 ) = [UNK] ( x1 ) [UNK] ( x ηn−1 | x1 ) [UNK] ( xηn−2 | xηn−1 )... [UNK] 1 ( xη1 | xη2 ), ( 18. 55 ) where [UNK] is the reverse of the transition operator deﬁned by ta ( via an application of


xηn−2 | xηn−1 )... [UNK] 1 ( xη1 | xη2 ), ( 18. 55 ) where [UNK] is the reverse of the transition operator deﬁned by ta ( via an application of bayes ’ rule ) : [UNK] ( x | x ) = pa ( x ) pa ( ) x ta ( x x | ) = [UNK] ( x ) [UNK] ( ) x ta ( x x | ). ( 18. 56 ) plugging the above into the expression for the joint distribution on the extended state space given in equation, we get : 18. 55 [UNK] ( xη1,..., xηn−1, x1 ) ( 18. 57 ) 627


chapter 18. confronting the partition function = [UNK] ( x1 ) [UNK] n−1 ( xηn−1 ) [UNK] n−1 ( x1 ) tηn−1 ( x 1 | xηn−1 ) n−2 i = 1 [UNK] ( xηi ) [UNK] ( xηi + 1 ) tη i ( xηi + 1 | xηi ) ( 18. 58 ) = [UNK] ( x1 ) [UNK] ( x1 ) tηn−1 ( x1 | xηn−1 ) [UNK] ( xη 1 ) n−2 i = 1 [UNK] + 1 ( xηi + 1 ) [UNK] ( xηi + 1 ) tη i ( xη i + 1 | xηi ). ( 18. 59 ) we now have means of generating samples from the joint proposal distribution q over the extended sample via a sampling scheme given above, with the joint distribution given by : q ( xη 1,..., xηn−1, x1 ) = p0 ( xη1 ) tη1 ( xη2 | xη1 )... tηn−1 ( x1 | x ηn−1 ). ( 18. 60 ) we have a joint distribution on the extended space given


##0 ( xη1 ) tη1 ( xη2 | xη1 )... tηn−1 ( x1 | x ηn−1 ). ( 18. 60 ) we have a joint distribution on the extended space given by equation. taking 18. 59 q ( xη1,..., xη n−1, x1 ) as the proposal distribution on the extended state space from which we will draw samples, it remains to determine the importance weights : w ( ) k = [UNK] ( xη1,..., xηn−1, x 1 ) q ( xη1,..., xη n−1, x1 ) = [UNK] ( x ( ) k 1 ) [UNK] ( x ( ) k ηn−1 )... [UNK] ( x ( ) k η2 ) [UNK] ( x ( ) k η1 ) [UNK] ( x ( ) k η1 ) [UNK] 0 ( x ( ) k 0 ). ( 18. 61 ) these weights are the same as proposed for ais. thus we can interpret ais as simple importance sampling applied to an extended state and its validity follows immediately from the validity of importance sampling. annealed importance sampling ( ais ) was ﬁrst discovered by


are the same as proposed for ais. thus we can interpret ais as simple importance sampling applied to an extended state and its validity follows immediately from the validity of importance sampling. annealed importance sampling ( ais ) was ﬁrst discovered by ( ) jarzynski 1997 and then again, independently, by ( ). it is currently the most common neal 2001 way of estimating the partition function for undirected probabilistic models. the reasons for this may have more to do with the publication of an inﬂuential paper ( salakhutdinov and murray 2008, ) describing its application to estimating the partition function of restricted boltzmann machines and deep belief networks than with any inherent advantage the method has over the other method described below. a discussion of the properties of the ais estimator ( e. g.. its variance and [UNK] ) can be found in ( ). neal 2001 18. 7. 2 bridge sampling bridge sampling ( ) is another method that, like ais, addresses the bennett 1976 shortcomings of importance sampling. rather than chaining together a series of 628


chapter 18. confronting the partition function intermediate distributions, bridge sampling relies on a single distribution p∗, known as the bridge, to interpolate between a distribution with known partition function, p0, and a distribution p1 for which we are trying to estimate the partition function z1. bridge sampling estimates the ratio z1 / z0 as the ratio of the expected impor - tance weights between [UNK] and [UNK] between [UNK] and [UNK] : z1 z0 ≈ k k = 1 [UNK] ( x ( ) k 0 ) [UNK] ( x ( ) k 0 ) k k = 1 [UNK] ( x ( ) k 1 ) [UNK] ( x ( ) k 1 ) ( 18. 62 ) if the bridge distribution p∗is chosen carefully to have a large overlap of support with both p0 and p1, then bridge sampling can allow the distance between two distributions ( or more formally, dkl ( p0p1 ) ) to be much larger than with standard importance sampling. it can be shown that the optimal bridging distribution is given by p ( ) opt ∗ ( x ) [UNK] [UNK] ( ) [UNK] x p1 ( ) x [UNK] ( ) + [UNK] x p1 ( ) x where r = z1 / z0. at ﬁrst, this


##idging distribution is given by p ( ) opt ∗ ( x ) [UNK] [UNK] ( ) [UNK] x p1 ( ) x [UNK] ( ) + [UNK] x p1 ( ) x where r = z1 / z0. at ﬁrst, this appears to be an unworkable solution as it would seem to require the very quantity we are trying to estimate, z1 / z0. however, it is possible to start with a coarse estimate of r and use the resulting bridge distribution to reﬁne our estimate iteratively (, ). that is, we neal 2005 iteratively re - estimate the ratio and use each iteration to update the value of. r linked importance sampling both ais and bridge sampling have their ad - vantages. if dkl ( p 0p1 ) is not too large ( because p0 and p1 are [UNK] close ) bridge sampling can be a more [UNK] means of estimating the ratio of partition functions than ais. if, however, the two distributions are too far apart for a single distribution p∗to bridge the gap then one can at least use ais with potentially many intermediate distributions to span the distance between p0 and p1. neal ( ) showed how his linked importance sampling method leveraged the power


far apart for a single distribution p∗to bridge the gap then one can at least use ais with potentially many intermediate distributions to span the distance between p0 and p1. neal ( ) showed how his linked importance sampling method leveraged the power of 2005 the bridge sampling strategy to bridge the intermediate distributions used in ais to signiﬁcantly improve the overall partition function estimates. estimating the partition function while training while ais has become accepted as the standard method for estimating the partition function for many undirected models, it is [UNK] computationally intensive that it remains infeasible to use during training. however, alternative strategies that have been explored to maintain an estimate of the partition function throughout training using a combination of bridge sampling, short - chain ais and parallel tempering, desjardins 2011 et al. ( ) devised a scheme to track the partition function of an 629


chapter 18. confronting the partition function rbm throughout the training process. the strategy is based on the maintenance of independent estimates of the partition functions of the rbm at every temperature operating in the parallel tempering scheme. the authors combined bridge sampling estimates of the ratios of partition functions of neighboring chains ( i. e. from parallel tempering ) with ais estimates across time to come up with a low variance estimate of the partition functions at every iteration of learning. the tools described in this chapter provide many [UNK] ways of overcoming the problem of intractable partition functions, but there can be several other [UNK] involved in training and using generative models. foremost among these is the problem of intractable inference, which we confront next. 630


chapter 19 approximate inference many probabilistic models are [UNK] to train because it is [UNK] to perform inference in them. in the context of deep learning, we usually have a set of visible variables v and a set of latent variables h. the challenge of inference usually refers to the [UNK] problem of computing p ( h v | ) or taking expectations with respect to it. such operations are often necessary for tasks like maximum likelihood learning. many simple graphical models with only one hidden layer, such as restricted boltzmann machines and probabilistic pca, are deﬁned in a way that makes inference operations like computing p ( h v | ), or taking expectations with respect to it, simple. unfortunately, most graphical models with multiple layers of hidden variables have intractable posterior distributions. exact inference requires an exponential amount of time in these models. even some models with only a single layer, such as sparse coding, have this problem. in this chapter, we introduce several of the techniques for confronting these intractable inference problems. later, in chapter, we will describe how to use 20 these techniques to train probabilistic models that would otherwise be intractable, such as deep belief networks and deep boltzmann machines. intractable inference problems in deep learning usually arise


. later, in chapter, we will describe how to use 20 these techniques to train probabilistic models that would otherwise be intractable, such as deep belief networks and deep boltzmann machines. intractable inference problems in deep learning usually arise from interactions between latent variables in a structured graphical model. see ﬁgure for some 19. 1 examples. these interactions may be due to direct interactions in undirected models or “ explaining away ” interactions between mutual ancestors of the same visible unit in directed models. 631


chapter 19. approximate inference figure 19. 1 : intractable inference problems in deep learning are usually the result of interactions between latent variables in a structured graphical model. these can be due to edges directly connecting one latent variable to another, or due to longer paths that are activated when the child of a v - structure is observed. ( left ) asemi - restricted boltzmann machine (, ) with connections between hidden osindero and hinton 2008 units. these direct connections between latent variables make the posterior distribution intractable due to large cliques of latent variables. a deep boltzmann machine, ( center ) organized into layers of variables without intra - layer connections, still has an intractable posterior distribution due to the connections between layers. this directed model ( right ) has interactions between latent variables when the visible variables are observed, because every two latent variables are co - parents. some probabilistic models are able to provide tractable inference over the latent variables despite having one of the graph structures depicted above. this is possible if the conditional probability distributions are chosen to introduce additional independences beyond those described by the graph. for example, probabilistic pca has the graph structure shown in the right, yet still has simple inference due


structures depicted above. this is possible if the conditional probability distributions are chosen to introduce additional independences beyond those described by the graph. for example, probabilistic pca has the graph structure shown in the right, yet still has simple inference due to special properties of the speciﬁc conditional distributions it uses ( linear - gaussian conditionals with mutually orthogonal basis vectors ). 632


chapter 19. approximate inference 19. 1 inference as optimization many approaches to confronting the problem of [UNK] inference make use of the observation that exact inference can be described as an optimization problem. approximate inference algorithms may then be derived by approximating the underlying optimization problem. to construct the optimization problem, assume we have a probabilistic model consisting of observed variables v and latent variables h. we would like to compute the log probability of the observed data, log p ( v ; θ ). sometimes it is too [UNK] to compute log p ( v ; θ ) if it is costly to marginalize out h. instead, we can compute a lower bound l ( v θ,, q ) on log p ( v ; θ ). this bound is called the evidence lower bound ( elbo ). another commonly used name for this lower bound is the negative variational free energy. speciﬁcally, the evidence lower bound is deﬁned to be l − ( ) = log ( ; ) v θ,, q p v θ dkl ( ( ) ( ; ) ) q h v | p h v | θ ( 19. 1 ) where is an arbitrary probability distribution over. q h because the [UNK] between log p ( v ) and l ( v


p v θ dkl ( ( ) ( ; ) ) q h v | p h v | θ ( 19. 1 ) where is an arbitrary probability distribution over. q h because the [UNK] between log p ( v ) and l ( v θ,, q ) is given by the kl divergence and because the kl divergence is always non - negative, we can see that l always has at most the same value as the desired log probability. the two are equal if and only if is the same distribution as. q p ( ) h v | surprisingly, l can be considerably easier to compute for some distributions q. simple algebra shows that we can rearrange l into a much more convenient form : l − ( ) = log ( ; ) v θ,, q p v θ dkl ( ( ) ( ; ) ) q h v | p h v | θ ( 19. 2 ) = log ( ; ) p v θ [UNK] log q ( ) h v | p ( ) h v | ( 19. 3 ) = log ( ; ) p v θ [UNK] log q ( ) h v | p, ( h v θ ; ) p ( ; ) v θ ( 19. 4 ) = log ( ; ) p v


h v | ( 19. 3 ) = log ( ; ) p v θ [UNK] log q ( ) h v | p, ( h v θ ; ) p ( ; ) v θ ( 19. 4 ) = log ( ; ) p v θ [UNK] [ log ( ) log ( ; ) + log ( ; ) ] q h v | − p h v, θ p v θ ( 19. 5 ) = [UNK] [ log ( ) log ( ; ) ] q h v | − p h v, θ. ( 19. 6 ) this yields the more canonical deﬁnition of the evidence lower bound, l ( ) = v θ,, q [UNK] [ log ( ) ] + ( ) p h v, h q. ( 19. 7 ) for an appropriate choice of q, l is tractable to compute. for any choice of q, l provides a lower bound on the likelihood. for q ( h v | ) that are better 633


chapter 19. approximate inference approximations of p ( h v | ), the lower bound l will be tighter, in other words, closer to log p ( v ). when q ( h v | ) = p ( h v | ), the approximation is perfect, and l ( ) = log ( ; ) v θ,, q p v θ. we can thus think of inference as the procedure for ﬁnding the q that maximizes l. exact inference maximizes l perfectly by searching over a family of functions q that includes p ( h v | ). throughout this chapter, we will show how to derive [UNK] forms of approximate inference by using approximate optimization to ﬁnd q. we can make the optimization procedure less expensive but approximate by restricting the family of distributions q the optimization is allowed to search over or by using an imperfect optimization procedure that may not completely maximize but merely increase it by a signiﬁcant amount. l no matter what choice of q we use, l is a lower bound. we can get tighter or looser bounds that are cheaper or more expensive to compute depending on how we choose to approach this optimization problem. we can obtain a poorly matched q but reduce the computational cost by using an imperfect optimization procedure, or by using a perfect optimization


get tighter or looser bounds that are cheaper or more expensive to compute depending on how we choose to approach this optimization problem. we can obtain a poorly matched q but reduce the computational cost by using an imperfect optimization procedure, or by using a perfect optimization procedure over a restricted family of q distributions. 19. 2 expectation maximization the ﬁrst algorithm we introduce based on maximizing a lower bound l is the expectation maximization ( em ) algorithm, a popular training algorithm for models with latent variables. we describe here a view on the em algorithm developed by ( ). unlike most of the other algorithms we neal and hinton 1999 describe in this chapter, em is not an approach to approximate inference, but rather an approach to learning with an approximate posterior. the em algorithm consists of alternating between two steps until convergence : • the e - step ( expectation step ) : let θ ( 0 ) denote the value of the parameters at the beginning of the step. set q ( h ( ) i | v ) = p ( h ( ) i | v ( ) i ; θ ( 0 ) ) for all indices i of the training examples v ( ) i we want to train on ( both batch and minibatch variants are valid ). by this we mean q is deﬁ


) i | v ( ) i ; θ ( 0 ) ) for all indices i of the training examples v ( ) i we want to train on ( both batch and minibatch variants are valid ). by this we mean q is deﬁned in terms of the current parameter value of θ ( 0 ) ; if we vary θ then p ( h v | ; θ ) will change but q p ( ) h v | will remain equal to ( ; h v | θ ( 0 ) ). • the ( maximization step ) : completely or partially maximize m - step i l ( v ( ) i,, q θ ) ( 19. 8 ) 634


chapter 19. approximate inference with respect to using your optimization algorithm of choice. θ this can be viewed as a coordinate ascent algorithm to maximize l. on one step, we maximize l with respect to q, and on the other, we maximize l with respect to. θ stochastic gradient ascent on latent variable models can be seen as a special case of the em algorithm where the m step consists of taking a single gradient step. other variants of the em algorithm can make much larger steps. for some model families, the m step can even be performed analytically, jumping all the way to the optimal solution for given the current. θ q even though the e - step involves exact inference, we can think of the em algorithm as using approximate inference in some sense. speciﬁcally, the m - step assumes that the same value of q can be used for all values of θ. this will introduce a gap between l and the true log p ( v ) as the m - step moves further and further away from the value θ ( 0 ) used in the e - step. fortunately, the e - step reduces the gap to zero again as we enter the loop for the next time. the em algorithm contains a few [UNK] insights. first, there is the basic