chapter 12. applications computing speed have come from parallelization across cores ( either in cpus or gpus ). this is very [UNK] from the situation of the 1990s ( the previous neural network era ) where the hardware implementations of neural networks ( which might take two years from inception to availability of a chip ) could not keep up with the rapid progress and low prices of general - purpose cpus. building specialized hardware is thus a way to push the envelope further, at a time when new hardware designs are being developed for low - power devices such as phones, aiming for general - public applications of deep learning ( e. g., with speech, computer vision or natural language ). recent work on low - precision implementations of backprop - based neural nets ( vanhoucke 2011 courbariaux 2015 gupta 2015 et al., ; et al., ; et al., ) suggests that between 8 and 16 bits of precision can [UNK] for using or training deep neural networks with back - propagation. what is clear is that more precision is required during training than at inference time, and that some forms of dynamic ﬁxed point representation of numbers can be used to reduce how many bits are required per number. traditional ﬁxed point numbers are restricted to a ﬁxed range (


that more precision is required during training than at inference time, and that some forms of dynamic ﬁxed point representation of numbers can be used to reduce how many bits are required per number. traditional ﬁxed point numbers are restricted to a ﬁxed range ( which corresponds to a given exponent in a ﬂoating point representation ). dynamic ﬁxed point representations share that range among a set of numbers ( such as all the weights in one layer ). using ﬁxed point rather than ﬂoating point representations and using less bits per number reduces the hardware surface area, power requirements and computing time needed for performing multiplications, and multiplications are the most demanding of the operations needed to use or train a modern deep network with backprop. 12. 2 computer vision computer vision has traditionally been one of the most active research areas for deep learning applications, because vision is a task that is [UNK] for humans and many animals but challenging for computers (, ). many of ballard et al. 1983 the most popular standard benchmark tasks for deep learning algorithms are forms of object recognition or optical character recognition. computer vision is a very broad ﬁeld encompassing a wide variety of ways of processing images, and an amazing diversity of applications. applications of computer vision range from reproducing human visual

/home/ricoiban/GEMMA/mnlp_chatsplaining/RAG/Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville (z-lib.org).pdf


deep learning algorithms are forms of object recognition or optical character recognition. computer vision is a very broad ﬁeld encompassing a wide variety of ways of processing images, and an amazing diversity of applications. applications of computer vision range from reproducing human visual abilities, such as recognizing faces, to creating entirely new categories of visual abilities. as an example of the latter category, one recent computer vision application is to recognize sound waves from the vibrations they induce in objects visible in a video (, davis et al. 2014 ). most deep learning research on computer vision has not focused on such 452


chapter 12. applications exotic applications that expand the realm of what is possible with imagery but rather a small core of ai goals aimed at replicating human abilities. most deep learning for computer vision is used for object recognition or detection of some form, whether this means reporting which object is present in an image, annotating an image with bounding boxes around each object, transcribing a sequence of symbols from an image, or labeling each pixel in an image with the identity of the object it belongs to. because generative modeling has been a guiding principle of deep learning research, there is also a large body of work on image synthesis using deep models. while image synthesis is usually not considered a ex nihilo computer vision endeavor, models capable of image synthesis are usually useful for image restoration, a computer vision task involving repairing defects in images or removing objects from images. 12. 2. 1 preprocessing many application areas require sophisticated preprocessing because the original input comes in a form that is [UNK] for many deep learning architectures to represent. computer vision usually requires relatively little of this kind of pre - processing. the images should be standardized so that their pixels all lie in the same, reasonable range, like [ 0, 1 ] or [ - 1, 1


learning architectures to represent. computer vision usually requires relatively little of this kind of pre - processing. the images should be standardized so that their pixels all lie in the same, reasonable range, like [ 0, 1 ] or [ - 1, 1 ]. mixing images that lie in [ 0, 1 ] with images that lie in [ 0, 255 ] will usually result in failure. formatting images to have the same scale is the only kind of preprocessing that is strictly necessary. many computer vision architectures require images of a standard size, so images must be cropped or scaled to ﬁt that size. even this rescaling is not always strictly necessary. some convolutional models accept variably - sized inputs and dynamically adjust the size of their pooling regions to keep the output size constant ( waibel et al., 1989 ). other convolutional models have variable - sized output that automatically scales in size with the input, such as models that denoise or label each pixel in an image (, ). hadsell et al. 2007 dataset augmentation may be seen as a way of preprocessing the training set only. dataset augmentation is an excellent way to reduce the general


label each pixel in an image (, ). hadsell et al. 2007 dataset augmentation may be seen as a way of preprocessing the training set only. dataset augmentation is an excellent way to reduce the generalization error of most computer vision models. a related idea applicable at test time is to show the model many [UNK] versions of the same input ( for example, the same image cropped at slightly [UNK] locations ) and have the [UNK] instantiations of the model vote to determine the output. this latter idea can be interpreted as an ensemble approach, and helps to reduce generalization error. other kinds of preprocessing are applied to both the train and the test set with the goal of putting each example into a more canonical form in order to reduce the amount of variation that the model needs to account for. reducing the amount of 453


chapter 12. applications variation in the data can both reduce generalization error and reduce the size of the model needed to ﬁt the training set. simpler tasks may be solved by smaller models, and simpler solutions are more likely to generalize well. preprocessing of this kind is usually designed to remove some kind of variability in the input data that is easy for a human designer to describe and that the human designer is conﬁdent has no relevance to the task. when training with large datasets and large models, this kind of preprocessing is often unnecessary, and it is best to just let the model learn which kinds of variability it should become invariant to. for example, the alexnet system for classifying imagenet only has one preprocessing step : subtracting the mean across training examples of each pixel ( krizhevsky et al., ). 2012 12. 2. 1. 1 contrast normalization one of the most obvious sources of variation that can be safely removed for many tasks is the amount of contrast in the image. contrast simply refers to the magnitude of the [UNK] between the bright and the dark pixels in an image. there are many ways of quantifying the contrast of an image. in the context of deep learning,


is the amount of contrast in the image. contrast simply refers to the magnitude of the [UNK] between the bright and the dark pixels in an image. there are many ways of quantifying the contrast of an image. in the context of deep learning, contrast usually refers to the standard deviation of the pixels in an image or region of an image. suppose we have an image represented by a tensor x ∈rr c × ×3, with xi, j, 1 being the red intensity at row i and column j, xi, j, 2 giving the green intensity and xi, j, 3 giving the blue intensity. then the contrast of the entire image is given by 1 3rc r i = 1 c j = 1 3 k = 1 xi, j, k [UNK] 2 ( 12. 1 ) where [UNK] is the mean intensity of the entire image : [UNK] = 1 3rc r i = 1 c j = 1 3 k = 1 xi, j, k. ( 12. 2 ) global contrast normalization ( gcn ) aims to prevent images from having varying amounts of contrast by subtracting the mean from each image, then rescaling it so that the standard deviation across its pixels is equal to some constant s. this approach is complicated by the fact that


##n ) aims to prevent images from having varying amounts of contrast by subtracting the mean from each image, then rescaling it so that the standard deviation across its pixels is equal to some constant s. this approach is complicated by the fact that no scaling factor can change the contrast of a zero - contrast image ( one whose pixels all have equal intensity ). images with very low but non - zero contrast often have little information content. dividing by the true standard deviation usually accomplishes nothing 454


chapter 12. applications more than amplifying sensor noise or compression artifacts in such cases. this motivates introducing a small, positive regularization parameter λ to bias the estimate of the standard deviation. alternately, one can constrain the denominator to be at least. given an input image x, gcn produces an output image x, deﬁned such that x i, j, k = s xi, j, k [UNK] max, λ + 1 3rc r i = 1 c j = 1 3 k = 1 xi, j, k [UNK]. ( 12. 3 ) datasets consisting of large images cropped to interesting objects are unlikely to contain any images with nearly constant intensity. in these cases, it is safe to practically ignore the small denominator problem by setting λ = 0 and avoid division by 0 in extremely rare cases by setting to an extremely low value like 10−8. this is the approach used by ( ) on the cifar - 10 goodfellow et al. 2013a dataset. small images cropped randomly are more likely to have nearly constant intensity, making aggressive regularization more useful. ( ) used coates et al. 2011 λ = 0 and = 10 on small, randomly selected patches drawn from ci


2013a dataset. small images cropped randomly are more likely to have nearly constant intensity, making aggressive regularization more useful. ( ) used coates et al. 2011 λ = 0 and = 10 on small, randomly selected patches drawn from cifar - 10. the scale parameter s can usually be set to, as done by ( ), 1 coates et al. 2011 or chosen to make each individual pixel have standard deviation across examples close to 1, as done by ( ). goodfellow et al. 2013a the standard deviation in equation is just a rescaling of the 12. 3 l2 norm of the image ( assuming the mean of the image has already been removed ). it is preferable to deﬁne gcn in terms of standard deviation rather than l2 norm because the standard deviation includes division by the number of pixels, so gcn based on standard deviation allows the same s to be used regardless of image size. however, the observation that the l2 norm is proportional to the standard deviation can help build a useful intuition. one can understand gcn as mapping examples to a spherical shell. see ﬁgure for an illustration. this can be a 12. 1 useful property because neural networks are often better at responding


to the standard deviation can help build a useful intuition. one can understand gcn as mapping examples to a spherical shell. see ﬁgure for an illustration. this can be a 12. 1 useful property because neural networks are often better at responding to directions in space rather than exact locations. responding to multiple distances in the same direction requires hidden units with collinear weight vectors but [UNK] biases. such coordination can be [UNK] for the learning algorithm to discover. additionally, many shallow graphical models have problems with representing multiple separated modes along the same line. gcn avoids these problems by reducing each example to a direction rather than a direction and a distance. counterintuitively, there is a preprocessing operation known as sphering and it is not the same operation as gcn. sphering does not refer to making the data lie on a spherical shell, but rather to rescaling the principal components to have 455


chapter 12. applications −1 5 0 0 1 5... x0 −1 5. 0 0. 1 5. x1 raw input −1 5 0 0 1 5... x0 gcn, = 10 λ −2 −1 5 0 0 1 5... x0 gcn, = 0 λ figure 12. 1 : gcn maps examples onto a sphere. ( left ) raw input data may have any norm. ( center ) gcn with λ = 0 maps all non - zero examples perfectly onto a sphere. here we use s = 1 and = 10−8. because we use gcn based on normalizing the standard deviation rather than the l2 norm, the resulting sphere is not the unit sphere. ( right ) regularized gcn, with λ > 0, draws examples toward the sphere but does not completely discard the variation in their norm. we leave and the same as before. s equal variance, so that the multivariate normal distribution used by pca has spherical contours. sphering is more commonly known as. whitening global contrast normalization will often fail to highlight image features we would like to stand out, such as edges and corners. if we have a


normal distribution used by pca has spherical contours. sphering is more commonly known as. whitening global contrast normalization will often fail to highlight image features we would like to stand out, such as edges and corners. if we have a scene with a large dark area and a large bright area ( such as a city square with half the image in the shadow of a building ) then global contrast normalization will ensure there is a large [UNK] between the brightness of the dark area and the brightness of the light area. it will not, however, ensure that edges within the dark region stand out. this motivates local contrast normalization. local contrast normalization ensures that the contrast is normalized across each small window, rather than over the image as a whole. see ﬁgure for a comparison of global and local contrast 12. 2 normalization. various deﬁnitions of local contrast normalization are possible. in all cases, one modiﬁes each pixel by subtracting a mean of nearby pixels and dividing by a standard deviation of nearby pixels. in some cases, this is literally the mean and standard deviation of all pixels in a rectangular window centered on the pixel to be modiﬁed (, ). in other cases, this is a


and dividing by a standard deviation of nearby pixels. in some cases, this is literally the mean and standard deviation of all pixels in a rectangular window centered on the pixel to be modiﬁed (, ). in other cases, this is a weighted mean pinto et al. 2008 and weighted standard deviation using gaussian weights centered on the pixel to be modiﬁed. in the case of color images, some strategies process [UNK] color 456


chapter 12. applications input image gcn lcn figure 12. 2 : a comparison of global and local contrast normalization. visually, the [UNK] of global contrast normalization are subtle. it places all images on roughly the same scale, which reduces the burden on the learning algorithm to handle multiple scales. local contrast normalization modiﬁes the image much more, discarding all regions of constant intensity. this allows the model to focus on just the edges. regions of ﬁne texture, such as the houses in the second row, may lose some detail due to the bandwidth of the normalization kernel being too high. channels separately while others combine information from [UNK] channels to normalize each pixel (, ). sermanet et al. 2012 local contrast normalization can usually be implemented [UNK] by using separable convolution ( see section ) to compute feature maps of local means and 9. 8 local standard deviations, then using element - wise subtraction and element - wise division on [UNK] feature maps. local contrast normalization is a [UNK] operation and can also be used as a nonlinearity applied to the hidden layers of a network, as well as a preprocessing operation applied to the input. as with global contrast normalization, we typically need to


contrast normalization is a [UNK] operation and can also be used as a nonlinearity applied to the hidden layers of a network, as well as a preprocessing operation applied to the input. as with global contrast normalization, we typically need to regularize local contrast normalization to avoid division by zero. in fact, because local contrast normalization typically acts on smaller windows, it is even more important to regularize. smaller windows are more likely to contain values that are all nearly the same as each other, and thus more likely to have zero standard deviation. 457


chapter 12. applications 12. 2. 1. 2 dataset augmentation as described in section, it is easy to improve the generalization of a classiﬁer 7. 4 by increasing the size of the training set by adding extra copies of the training examples that have been modiﬁed with transformations that do not change the class. object recognition is a classiﬁcation task that is especially amenable to this form of dataset augmentation because the class is invariant to so many transformations and the input can be easily transformed with many geometric operations. as described before, classiﬁers can beneﬁt from random translations, rotations, and in some cases, ﬂips of the input to augment the dataset. in specialized computer vision applications, more advanced transformations are commonly used for dataset augmentation. these schemes include random perturbation of the colors in an image (, ) and nonlinear geometric distortions of krizhevsky et al. 2012 the input (, ). lecun et al. 1998b 12. 3 speech recognition the task of speech recognition is to map an acoustic signal containing a spoken natural language utterance into the corresponding sequence of words intended by the speaker. let x = ( x (


). lecun et al. 1998b 12. 3 speech recognition the task of speech recognition is to map an acoustic signal containing a spoken natural language utterance into the corresponding sequence of words intended by the speaker. let x = ( x ( 1 ), x ( 2 ),..., x ( ) t ) denote the sequence of acoustic input vectors ( traditionally produced by splitting the audio into 20ms frames ). most speech recognition systems preprocess the input using specialized hand - designed features, but some (, ) deep learning systems learn features jaitly and hinton 2011 from raw input. let y = ( y1, y2,..., yn ) denote the target output sequence ( usually a sequence of words or characters ). the automatic speech recognition ( asr ) task consists of creating a function f ∗ asr that computes the most probable linguistic sequence given the acoustic sequence : y x f∗ asr ( ) = arg max x y p ∗ ( = ) y x | x ( 12. 4 ) where p ∗is the true conditional distribution relating the inputs x to the targets y. since the 1980s and until about 2009 – 2012, state - of - the art speech recognition systems primarily combined hidden markov


y x | x ( 12. 4 ) where p ∗is the true conditional distribution relating the inputs x to the targets y. since the 1980s and until about 2009 – 2012, state - of - the art speech recognition systems primarily combined hidden markov models ( hmms ) and gaussian mixture models ( gmms ). gmms modeled the association between acoustic features and phonemes (, ), while hmms modeled the sequence of phonemes. bahl et al. 1987 the gmm - hmm model family treats acoustic waveforms as being generated by the following process : ﬁrst an hmm generates a sequence of phonemes and discrete sub - phonemic states ( such as the beginning, middle, and end of each 458


chapter 12. applications phoneme ), then a gmm transforms each discrete symbol into a brief segment of audio waveform. although gmm - hmm systems dominated asr until recently, speech recognition was actually one of the ﬁrst areas where neural networks were applied, and numerous asr systems from the late 1980s and early 1990s used neural nets ( bourlard and wellekens 1989 waibel 1989 robinson and, ; et al., ; fallside 1991 bengio 1991 1992 konig 1996, ; et al.,, ; et al., ). at the time, the performance of asr based on neural nets approximately matched the performance of gmm - hmm systems. for example, robinson and fallside 1991 ( ) achieved 26 % phoneme error rate on the timit (, ) corpus ( with 39 garofolo et al. 1993 phonemes to discriminate between ), which was better than or comparable to hmm - based systems. since then, timit has been a benchmark for phoneme recognition, playing a role similar to the role mnist plays for object recognition. however, because of the complex engineering involved in software systems for speech recognition and the [UNK] that had been invested in building these systems on the basis of gmm - hmm


##me recognition, playing a role similar to the role mnist plays for object recognition. however, because of the complex engineering involved in software systems for speech recognition and the [UNK] that had been invested in building these systems on the basis of gmm - hmms, the industry did not see a compelling argument for switching to neural networks. as a consequence, until the late 2000s, both academic and industrial research in using neural nets for speech recognition mostly focused on using neural nets to learn extra features for gmm - hmm systems. later, with much larger and deeper models and much larger datasets, recognition accuracy was dramatically improved by using neural networks to replace gmms for the task of associating acoustic features to phonemes ( or sub - phonemic states ). starting in 2009, speech researchers applied a form of deep learning based on unsupervised learning to speech recognition. this approach to deep learning was based on training undirected probabilistic models called restricted boltzmann machines ( rbms ) to model the input data. rbms will be described in part. iii to solve speech recognition tasks, unsupervised pretraining was used to build deep feedforward networks whose layers were each initialized by training an rbm. these networks


the input data. rbms will be described in part. iii to solve speech recognition tasks, unsupervised pretraining was used to build deep feedforward networks whose layers were each initialized by training an rbm. these networks take spectral acoustic representations in a ﬁxed - size input window ( around a center frame ) and predict the conditional probabilities of hmm states for that center frame. training such deep networks helped to signiﬁcantly improve the recognition rate on timit (,, ), bringing down the mohamed et al. 2009 2012a phoneme error rate from about 26 % to 20. 7 %. see ( ) for an mohamed et al. 2012b analysis of reasons for the success of these models. extensions to the basic phone recognition pipeline included the addition of speaker - adaptive features ( mohamed et al., ) that further reduced the error rate. this was quickly followed up 2011 by work to expand the architecture from phoneme recognition ( which is what timit is focused on ) to large - vocabulary speech recognition (, ), dahl et al. 2012 which involves not just recognizing phonemes but also recognizing sequences of words from a large vocabulary. deep networks for speech recognition eventually 459


chapter 12. applications shifted from being based on pretraining and boltzmann machines to being based on techniques such as rectiﬁed linear units and dropout (, ; zeiler et al. 2013 dahl et al., ). by that time, several of the major speech groups in industry had 2013 started exploring deep learning in collaboration with academic researchers. hinton et al. ( ) describe the breakthroughs achieved by these collaborators, which 2012a are now deployed in products such as mobile phones. later, as these groups explored larger and larger labeled datasets and incorpo - rated some of the methods for initializing, training, and setting up the architecture of deep nets, they realized that the unsupervised pretraining phase was either unnecessary or did not bring any signiﬁcant improvement. these breakthroughs in recognition performance for word error rate in speech recognition were unprecedented ( around 30 % improvement ) and were following a long period of about ten years during which error rates did not improve much with the traditional gmm - hmm technology, in spite of the continuously growing size of training sets ( see ﬁgure 2. 4 of deng and yu 2014 ( ) ). this created a rapid shift in the speech recognition community towards deep


improve much with the traditional gmm - hmm technology, in spite of the continuously growing size of training sets ( see ﬁgure 2. 4 of deng and yu 2014 ( ) ). this created a rapid shift in the speech recognition community towards deep learning. in a matter of roughly two years, most of the industrial products for speech recognition incorporated deep neural networks and this success spurred a new wave of research into deep learning algorithms and architectures for asr, which is still ongoing today. one of these innovations was the use of convolutional networks (, sainath et al. 2013 ) that replicate weights across time and frequency, improving over the earlier time - delay neural networks that replicated weights only across time. the new two - dimensional convolutional models regard the input spectrogram not as one long vector but as an image, with one axis corresponding to time and the other to frequency of spectral components. another important push, still ongoing, has been towards end - to - end deep learning speech recognition systems that completely remove the hmm. the ﬁrst major breakthrough in this direction came from graves 2013 et al. ( ) who trained a deep lstm rnn ( see section ), using map inference over the frame - to - 10.


recognition systems that completely remove the hmm. the ﬁrst major breakthrough in this direction came from graves 2013 et al. ( ) who trained a deep lstm rnn ( see section ), using map inference over the frame - to - 10. 10 phoneme alignment, as in ( ) and in the ctc framework ( lecun et al. 1998b graves et al., ; 2006 graves 2012 graves 2013, ). a deep rnn ( et al., ) has state variables from several layers at each time step, giving the unfolded graph two kinds of depth : ordinary depth due to a stack of layers, and depth due to time unfolding. this work brought the phoneme error rate on timit to a record low of 17. 7 %. see pascanu 2014a chung 2014 et al. ( ) and et al. ( ) for other variants of deep rnns, applied in other settings. another contemporary step toward end - to - end deep learning asr is to let the system learn how to “ align ” the acoustic - level information with the phonetic - level 460


chapter 12. applications information (, ;, ). chorowski et al. 2014 lu et al. 2015 12. 4 natural language processing natural language processing ( nlp ) is the use of human languages, such as english or french, by a computer. computer programs typically read and emit specialized languages designed to allow [UNK] and unambiguous parsing by simple programs. more naturally occurring languages are often ambiguous and defy formal description. natural language processing includes applications such as machine translation, in which the learner must read a sentence in one human language and emit an equivalent sentence in another human language. many nlp applications are based on language models that deﬁne a probability distribution over sequences of words, characters or bytes in a natural language. as with the other applications discussed in this chapter, very generic neural network techniques can be successfully applied to natural language processing. however, to achieve excellent performance and to scale well to large applications, some domain - speciﬁc strategies become important. to build an [UNK] model of natural language, we must usually use techniques that are specialized for processing sequential data. in many cases, we choose to regard natural language as a sequence of words, rather than a sequence of individual characters or bytes. because the total number of possible


model of natural language, we must usually use techniques that are specialized for processing sequential data. in many cases, we choose to regard natural language as a sequence of words, rather than a sequence of individual characters or bytes. because the total number of possible words is so large, word - based language models must operate on an extremely high - dimensional and sparse discrete space. several strategies have been developed to make models of such a space [UNK], both in a computational and in a statistical sense. 12. 4. 1 - grams n a language model deﬁnes a probability distribution over sequences of tokens in a natural language. depending on how the model is designed, a token may be a word, a character, or even a byte. tokens are always discrete entities. the earliest successful language models were based on models of ﬁxed - length sequences of tokens called - grams. an - gram is a sequence of tokens. n n n models based on n - grams deﬁne the conditional probability of the n - th token given the preceding n −1 tokens. the model uses products of these conditional distributions to deﬁne the probability distribution over longer sequences : p x ( 1,..., xτ ) = ( p x1,...


token given the preceding n −1 tokens. the model uses products of these conditional distributions to deﬁne the probability distribution over longer sequences : p x ( 1,..., xτ ) = ( p x1,..., xn−1 ) τ t n = p x ( t | xt n − + 1,..., xt−1 ). ( 12. 5 ) 461


chapter 12. applications this decomposition is justiﬁed by the chain rule of probability. the probability distribution over the initial sequencep ( x1,..., xn−1 ) may be modeled by a [UNK] model with a smaller value of. n training n - gram models is straightforward because the maximum likelihood estimate can be computed simply by counting how many times each possible n gram occurs in the training set. models based on n - grams have been the core building block of statistical language modeling for many decades ( jelinek and mercer 1980 katz 1987 chen and goodman 1999, ;, ;, ). for small values of n, models have particular names : unigram for n = 1, bigram for n = 2, and trigram for n = 3. these names derive from the latin preﬁxes for the corresponding numbers and the greek [UNK] “ - gram ” denoting something that is written. usually we train both an n - gram model and an n−1 gram model simultaneously. this makes it easy to compute p x ( t | xt n − + 1,..., xt−1 ) = pn ( xt n − + 1,..., xt ) pn−1 ( xt n − + 1,


compute p x ( t | xt n − + 1,..., xt−1 ) = pn ( xt n − + 1,..., xt ) pn−1 ( xt n − + 1,..., xt−1 ) ( 12. 6 ) simply by looking up two stored probabilities. for this to exactly reproduce inference in pn, we must omit the ﬁnal character from each sequence when we train pn−1. as an example, we demonstrate how a trigram model computes the probability of the sentence “ the dog ran away. ” the ﬁrst words of the sentence cannot be handled by the default formula based on conditional probability because there is no context at the beginning of the sentence. instead, we must use the marginal prob - ability over words at the start of the sentence. we thus evaluate p3 ( the dog ran ). finally, the last word may be predicted using the typical case, of using the condi - tional distribution p ( away dog ran | ). putting this together with equation, 12. 6 we obtain : p p ( ) = the dog ran away 3 ( ) the dog ran p3 ( ) dog ran away / p 2 ( )


tional distribution p ( away dog ran | ). putting this together with equation, 12. 6 we obtain : p p ( ) = the dog ran away 3 ( ) the dog ran p3 ( ) dog ran away / p 2 ( ) dog ran. ( 12. 7 ) a fundamental limitation of maximum likelihood for n - gram models is that pn as estimated from training set counts is very likely to be zero in many cases, even though the tuple ( x t n − + 1,..., xt ) may appear in the test set. this can cause two [UNK] kinds of catastrophic outcomes. when pn−1 is zero, the ratio is undeﬁned, so the model does not even produce a sensible output. when pn−1 is non - zero but pn is zero, the test log - likelihood is −∞. to avoid such catastrophic outcomes, most n - gram models employ some form of smoothing. smoothing techniques 462


chapter 12. applications shift probability mass from the observed tuples to unobserved ones that are similar. see ( ) for a review and empirical comparisons. one basic chen and goodman 1999 technique consists of adding non - zero probability mass to all of the possible next symbol values. this method can be justiﬁed as bayesian inference with a uniform or dirichlet prior over the count parameters. another very popular idea is to form a mixture model containing higher - order and lower - order n - gram models, with the higher - order models providing more capacity and the lower - order models being more likely to avoid counts of zero. back - [UNK] look - up the lower - order n - grams if the frequency of the context xt−1,..., xt n − + 1 is too small to use the higher - order model. more formally, they estimate the distribution over xt by using contexts xt n k − +,..., xt−1, for increasing k, until a [UNK] reliable estimate is found. classical n - gram models are particularly vulnerable to the curse of dimension - ality. there are | | v n possible n - grams and | | v is often very large. even with a massive training set and modest n


reliable estimate is found. classical n - gram models are particularly vulnerable to the curse of dimension - ality. there are | | v n possible n - grams and | | v is often very large. even with a massive training set and modest n, most n - grams will not occur in the training set. one way to view a classical n - gram model is that it is performing nearest - neighbor lookup. in other words, it can be viewed as a local non - parametric predictor, similar to k - nearest neighbors. the statistical problems facing these extremely local predictors are described in section. the problem for a language model 5. 11. 2 is even more severe than usual, because any two [UNK] words have the same dis - tance from each other in one - hot vector space. it is thus [UNK] to leverage much information from any “ neighbors ” — only training examples that repeat literally the same context are useful for local generalization. to overcome these problems, a language model must be able to share knowledge between one word and other semantically similar words. to improve the statistical [UNK] of n - gram models, class - based language models ( brown 1992 ney and kneser 1993 niesler 1998 et al., ;, ; et


share knowledge between one word and other semantically similar words. to improve the statistical [UNK] of n - gram models, class - based language models ( brown 1992 ney and kneser 1993 niesler 1998 et al., ;, ; et al., ) introduce the notion of word categories and then share statistical strength between words that are in the same category. the idea is to use a clustering algorithm to partition the set of words into clusters or classes, based on their co - occurrence frequencies with other words. the model can then use word class ids rather than individual word ids to represent the context on the right side of the conditioning bar. composite models combining word - based and class - based models via mixing or back - [UNK] also possible. although word classes provide a way to generalize between sequences in which some word is replaced by another of the same class, much information is lost in this representation. 463


chapter 12. applications 12. 4. 2 neural language models neural language models or nlms are a class of language model designed to overcome the curse of dimensionality problem for modeling natural language sequences by using a distributed representation of words (, ). bengio et al. 2001 unlike class - based n - gram models, neural language models are able to recognize that two words are similar without losing the ability to encode each word as distinct from the other. neural language models share statistical strength between one word ( and its context ) and other similar words and contexts. the distributed representation the model learns for each word enables this sharing by allowing the model to treat words that have features in common similarly. for example, if the word dog and the word cat map to representations that share many attributes, then sentences that contain the wordcat can inform the predictions that will be made by the model for sentences that contain the word dog, and vice - versa. because there are many such attributes, there are many ways in which generalization can happen, transferring information from each training sentence to an exponentially large number of semantically related sentences. the curse of dimensionality requires the model to generalize to a number of sentences that is exponential in the sentence length. the model counters this curse by relating each


transferring information from each training sentence to an exponentially large number of semantically related sentences. the curse of dimensionality requires the model to generalize to a number of sentences that is exponential in the sentence length. the model counters this curse by relating each training sentence to an exponential number of similar sentences. we sometimes call these word representations word embeddings. in this interpretation, we view the raw symbols as points in a space of dimension equal to the vocabulary size. the word representations embed those points in a feature space of lower dimension. in the original space, every word is represented by a one - hot vector, so every pair of words is at euclidean distance √ 2 from each other. in the embedding space, words that frequently appear in similar contexts ( or any pair of words sharing some “ features ” learned by the model ) are close to each other. this often results in words with similar meanings being neighbors. figure zooms in on speciﬁc areas of a learned word embedding space to show 12. 3 how semantically similar words map to representations that are close to each other. neural networks in other domains also deﬁne embeddings. for example, a hidden layer of a convolutional network provides an “


show 12. 3 how semantically similar words map to representations that are close to each other. neural networks in other domains also deﬁne embeddings. for example, a hidden layer of a convolutional network provides an “ image embedding. ” usually nlp practitioners are much more interested in this idea of embeddings because natural language does not originally lie in a real - valued vector space. the hidden layer has provided a more qualitatively dramatic change in the way the data is represented. the basic idea of using distributed representations to improve models for natural language processing is not restricted to neural networks. it may also be used with graphical models that have distributed representations in the form of 464


chapter 12. applications multiple latent variables ( mnih and hinton 2007, ). − − − − − 34 32 30 28 26 −14 −13 −12 −11 −10 −9 −8 −7 −6 canada europe ontario north english canadian union african africa british france russian china germany french assembly eu japan iraq south european 35 0 35 5 36 0 36 5 37 0 37 5 38 0....... 17 18 19 20 21 22 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 figure 12. 3 : two - dimensional visualizations of word embeddings obtained from a neural machine translation model (, ), zooming in on speciﬁc areas where bahdanau et al. 2015 semantically related words have embedding vectors that are close to each other. countries appear on the left and numbers on the right. keep in mind that these embeddings are 2 - d for the purpose of visualization. in real applications, embeddings typically have higher dimensionality and can simultaneously capture many kinds of similarity between words. 12. 4. 3 high - dimensional outputs in many natural language applications, we often want our models to produce words ( rather than characters ) as the fundamental


##beddings typically have higher dimensionality and can simultaneously capture many kinds of similarity between words. 12. 4. 3 high - dimensional outputs in many natural language applications, we often want our models to produce words ( rather than characters ) as the fundamental unit of the output. for large vocabularies, it can be very computationally expensive to represent an output distribution over the choice of a word, because the vocabulary size is large. in many applications, v contains hundreds of thousands of words. the naive approach to representing such a distribution is to apply an [UNK] transformation from a hidden representation to the output space, then apply the softmax function. suppose we have a vocabulary v with size | | v. the weight matrix describing the linear component of this [UNK] transformation is very large, because its output dimension is | | v. this imposes a high memory cost to represent the matrix, and a high computational cost to multiply by it. because the softmax is normalized across all | | v outputs, it is necessary to perform the full matrix multiplication at training time as well as test time — we cannot calculate only the dot product with the weight vector for the correct output. the high computational costs of the output layer thus arise both at training time ( to compute the likelihood


to perform the full matrix multiplication at training time as well as test time — we cannot calculate only the dot product with the weight vector for the correct output. the high computational costs of the output layer thus arise both at training time ( to compute the likelihood and its gradient ) and at test time ( to compute probabilities for all or selected words ). for specialized 465


chapter 12. applications loss functions, the gradient can be computed [UNK] (, ), but vincent et al. 2015 the standard cross - entropy loss applied to a traditional softmax output layer poses many [UNK]. suppose that h is the top hidden layer used to predict the output probabilities [UNK]. if we parametrize the transformation from h to [UNK] with learned weights w and learned biases b, then the [UNK] - softmax output layer performs the following computations : ai = bi + j wijhj [UNK] { | | } i 1,..., v, ( 12. 8 ) [UNK] = eai | | v i = 1 eai. ( 12. 9 ) if h contains nh elements then the above operation is o ( | | v n h ). with nh in the thousands and | | v in the hundreds of thousands, this operation dominates the computation of most neural language models. 12. 4. 3. 1 use of a short list the ﬁrst neural language models (,, ) dealt with the high cost bengio et al. 2001 2003 of using a softmax over a large number of output words by limiting the vocabulary size to 10, 000 or 20, 000 words. schwenk and gauvain 2002


) dealt with the high cost bengio et al. 2001 2003 of using a softmax over a large number of output words by limiting the vocabulary size to 10, 000 or 20, 000 words. schwenk and gauvain 2002 schwenk 2007 ( ) and ( ) built upon this approach by splitting the vocabulary v into a shortlist l of most frequent words ( handled by the neural net ) and a tail t = v l \ of more rare words ( handled by an n - gram model ). to be able to combine the two predictions, the neural net also has to predict the probability that a word appearing after context c belongs to the tail list. this may be achieved by adding an extra sigmoid output unit to provide an estimate of p ( i c ∈ | t ). the extra output can then be used to achieve an estimate of the probability distribution over all words in as follows : v p y i c ( = | ) = 1i∈lp y i c, i p i c ( = | ∈ − l ) ( 1 ( ∈ | t ) ) + 1i∈tp y i c, i p i c ( = | ∈t ) ( ∈ | t ) ( 12. 10 ) where p (


p i c ( = | ∈ − l ) ( 1 ( ∈ | t ) ) + 1i∈tp y i c, i p i c ( = | ∈t ) ( ∈ | t ) ( 12. 10 ) where p ( y = i c, i | ∈l ) is provided by the neural language model and p ( y = i | c, i ∈t ) is provided by the n - gram model. with slight modiﬁcation, this approach can also work using an extra output value in the neural language model ’ s softmax layer, rather than a separate sigmoid unit. an obvious disadvantage of the short list approach is that the potential gener - alization advantage of the neural language models is limited to the most frequent 466


chapter 12. applications words, where, arguably, it is the least useful. this disadvantage has stimulated the exploration of alternative methods to deal with high - dimensional outputs, described below. 12. 4. 3. 2 hierarchical softmax a classical approach (, ) to reducing the computational burden goodman 2001 of high - dimensional output layers over large vocabulary sets v is to decompose probabilities hierarchically. instead of necessitating a number of computations proportional to | | v ( and also proportional to the number of hidden units, nh ), the | | v factor can be reduced to as low as log | | v. ( ) and bengio 2002 morin and bengio 2005 ( ) introduced this factorized approach to the context of neural language models. one can think of this hierarchy as building categories of words, then categories of categories of words, then categories of categories of categories of words, etc. these nested categories form a tree, with words at the leaves. in a balanced tree, the tree has depth o ( log | | v ). the probability of a choosing a word is given by the product of the probabilities of choosing the branch leading to that word at every node on a path from the root of the tree to the leaf containing


depth o ( log | | v ). the probability of a choosing a word is given by the product of the probabilities of choosing the branch leading to that word at every node on a path from the root of the tree to the leaf containing the word. figure illustrates a simple example. ( ) also describe 12. 4 mnih and hinton 2009 how to use multiple paths to identify a single word in order to better model words that have multiple meanings. computing the probability of a word then involves summation over all of the paths that lead to that word. to predict the conditional probabilities required at each node of the tree, we typically use a logistic regression model at each node of the tree, and provide the same context c as input to all of these models. because the correct output is encoded in the training set, we can use supervised learning to train the logistic regression models. this is typically done using a standard cross - entropy loss, corresponding to maximizing the log - likelihood of the correct sequence of decisions. because the output log - likelihood can be computed [UNK] ( as low as log | | v rather than | | v ), its gradients may also be computed [UNK]. this includes not only the gradient with respect to the output parameters but


of decisions. because the output log - likelihood can be computed [UNK] ( as low as log | | v rather than | | v ), its gradients may also be computed [UNK]. this includes not only the gradient with respect to the output parameters but also the gradients with respect to the hidden layer activations. it is possible but usually not practical to optimize the tree structure to minimize the expected number of computations. tools from information theory specify how to choose the optimal binary code given the relative frequencies of the words. to do so, we could structure the tree so that the number of bits associated with a word is approximately equal to the logarithm of the frequency of that word. however, in 467


chapter 12. applications ( 1 ) ( 0 ) ( 0, 0, 0 ) ( 0, 0, 1 ) ( 0, 1, 0 ) ( 0, 1, 1 ) ( 1, 0, 0 ) ( 1, 0, 1 ) ( 1, 1, 0 ) ( 1, 1, 1 ) ( 1, 1 ) ( 1, 0 ) ( 0, 1 ) ( 0, 0 ) w0 w0 w1 w1 w2 w2 w3 w3 w4 w4 w5 w5 w6 w6 w7 w7 figure 12. 4 : illustration of a simple hierarchy of word categories, with 8 wordsw0,..., w7 organized into a three level hierarchy. the leaves of the tree represent actual speciﬁc words. internal nodes represent groups of words. any node can be indexed by the sequence of binary decisions ( 0 = left, 1 = right ) to reach the node from the root. super - class ( 0 ) contains the classes ( 0, 0 ) ( 0 and, 1 ), which respectively contain the sets of words { w 0, w1 } and { w2, w3 }, and similarly super - class contains the classes ( 1 ) ( 1


the classes ( 0, 0 ) ( 0 and, 1 ), which respectively contain the sets of words { w 0, w1 } and { w2, w3 }, and similarly super - class contains the classes ( 1 ) ( 1, 0 ) ( 1 and, 1 ), which respectively contain the words ( w4, w5 ) ( and w6, w7 ). if the tree is [UNK] balanced, the maximum depth ( number of binary decisions ) is on the order of the logarithm of the number of words | | v : the choice of one out of | | v words can be obtained by doing o ( log | | v ) operations ( one for each of the nodes on the path from the root ). in this example, computing the probability of a word y can be done by multiplying three probabilities, associated with the binary decisions to move left or right at each node on the path from the root to a node y. let bi ( y ) be the i - th binary decision when traversing the tree towards the value y. the probability of sampling an output y decomposes into a product of conditional probabilities, using the chain rule for conditional probabilities, with each node indexed


be the i - th binary decision when traversing the tree towards the value y. the probability of sampling an output y decomposes into a product of conditional probabilities, using the chain rule for conditional probabilities, with each node indexed by the preﬁx of these bits. for example, node ( 1, 0 ) corresponds to the preﬁx ( b0 ( w4 ) = 1, b1 ( w4 ) = 0 ), and the probability of w4 can be decomposed as follows : p w ( = y 4 ) = ( p b0 = 1, b1 = 0, b2 = 0 ) ( 12. 11 ) = ( p b0 = 1 ) ( p b1 = 0 | b0 = 1 ) ( p b2 = 0 | b0 = 1, b1 = 0 ). ( 12. 12 ) 468


chapter 12. applications practice, the computational savings are typically not worth the [UNK] because the computation of the output probabilities is only one part of the total computation in the neural language model. for example, suppose there are l fully connected hidden layers of width nh. let nb be the weighted average of the number of bits required to identify a word, with the weighting given by the frequency of these words. in this example, the number of operations needed to compute the hidden activations grows as as o ( ln2 h ) while the output computations grow as o ( nhnb ). as long as nb ≤lnh, we can reduce computation more by shrinking nh than by shrinking nb. indeed, nb is often small. because the size of the vocabulary rarely exceeds a million words and log2 ( 106 ) ≈20, it is possible to reduce nb to about, 20 but nh is often much larger, around 103 or more. rather than carefully optimizing a tree with a branching factor of, one can instead deﬁne a tree with depth two 2 and a branching factor of | | v. such a tree corresponds to simply deﬁning a set of mutually exclusive word classes. the simple approach based on a tree of depth


factor of, one can instead deﬁne a tree with depth two 2 and a branching factor of | | v. such a tree corresponds to simply deﬁning a set of mutually exclusive word classes. the simple approach based on a tree of depth two captures most of the computational beneﬁt of the hierarchical strategy. one question that remains somewhat open is how to best deﬁne these word classes, or how to deﬁne the word hierarchy in general. early work used existing hierarchies (, ) but the hierarchy can also be learned, ideally morin and bengio 2005 jointly with the neural language model. learning the hierarchy is [UNK]. an exact optimization of the log - likelihood appears intractable because the choice of a word hierarchy is a discrete one, not amenable to gradient - based optimization. however, one could use discrete optimization to approximately optimize the partition of words into word classes. an important advantage of the hierarchical softmax is that it brings computa - tional beneﬁts both at training time and at test time, if at test time we want to compute the probability of speciﬁc words. of course, computing the probability of all | | v words will remain expensive even with the hierarchical softmax.


##ﬁts both at training time and at test time, if at test time we want to compute the probability of speciﬁc words. of course, computing the probability of all | | v words will remain expensive even with the hierarchical softmax. another important operation is selecting the most likely word in a given context. unfortunately the tree structure does not provide an [UNK] and exact solution to this problem. a disadvantage is that in practice the hierarchical softmax tends to give worse test results than sampling - based methods we will describe next. this may be due to a poor choice of word classes. 12. 4. 3. 3 importance sampling one way to speed up the training of neural language models is to avoid explicitly computing the contribution of the gradient from all of the words that do not appear 469


chapter 12. applications in the next position. every incorrect word should have low probability under the model. it can be computationally costly to enumerate all of these words. instead, it is possible to sample only a subset of the words. using the notation introduced in equation, the gradient can be written as follows : 12. 8 ∂ p y c log ( | ) ∂θ = ∂log softmaxy ( ) a ∂θ ( 12. 13 ) = ∂ ∂θ log eay ieai ( 12. 14 ) = ∂ ∂θ ( a y −log i eai ) ( 12. 15 ) = ∂ay ∂θ − i p y i c ( = | ) ∂ai ∂θ ( 12. 16 ) where a is the vector of pre - softmax activations ( or scores ), with one element per word. the ﬁrst term is the positive phase term ( pushing ay up ) while the second term is the negative phase term ( pushing ai down for all i, with weight p ( i c | ). since the negative phase term is an expectation, we can estimate it with a monte carlo sample. however, that would require sampling from the model itself. sampling from the model requires computing p ( i c | ) for


p ( i c | ). since the negative phase term is an expectation, we can estimate it with a monte carlo sample. however, that would require sampling from the model itself. sampling from the model requires computing p ( i c | ) for all i in the vocabulary, which is precisely what we are trying to avoid. instead of sampling from the model, one can sample from another distribution, called the proposal distribution ( denoted q ), and use appropriate weights to correct for the bias introduced by sampling from the wrong distribution ( bengio and senecal 2003 bengio and senecal 2008, ;, ). this is an application of a more general technique called importance sampling, which will be described in more detail in section. unfortunately, even exact importance sampling is not [UNK] 17. 2 because it requires computing weights pi / qi, where pi = p ( i c | ), which can only be computed if all the scores ai are computed. the solution adopted for this application is called biased importance sampling, where the importance weights are normalized to sum to 1. when negative word ni is sampled, the associated gradient is weighted by wi = pni / qn i n j = 1 pnj / qnj. ( 12. 17 ) these weights are used


are normalized to sum to 1. when negative word ni is sampled, the associated gradient is weighted by wi = pni / qn i n j = 1 pnj / qnj. ( 12. 17 ) these weights are used to give the appropriate importance to the m negative samples from q used to form the estimated negative phase contribution to the 470


chapter 12. applications gradient : | | v i = 1 p i c ( | ) ∂ai ∂θ ≈1 m m i = 1 wi ∂ani ∂θ. ( 12. 18 ) a unigram or a bigram distribution works well as the proposal distribution q. it is easy to estimate the parameters of such a distribution from data. after estimating the parameters, it is also possible to sample from such a distribution very [UNK]. importance sampling is not only useful for speeding up models with large softmax outputs. more generally, it is useful for accelerating training with large sparse output layers, where the output is a sparse vector rather than a - of - 1 n choice. an example is a bag of words. a bag of words is a sparse vector v where vi indicates the presence or absence of word i from the vocabulary in the document. alternately, vi can indicate the number of times that word i appears. machine learning models that emit such sparse vectors can be expensive to train for a variety of reasons. early in learning, the model may not actually choose to make the output truly sparse. moreover, the loss function we use for training might most naturally be described in terms of comparing every element of the output to every element of the target. this means that


early in learning, the model may not actually choose to make the output truly sparse. moreover, the loss function we use for training might most naturally be described in terms of comparing every element of the output to every element of the target. this means that it is not always clear that there is a computational beneﬁt to using sparse outputs, because the model may choose to make the majority of the output non - zero and all of these non - zero values need to be compared to the corresponding training target, even if the training target is zero. dauphin 2011 et al. ( ) demonstrated that such models can be accelerated using importance sampling. the [UNK] algorithm minimizes the loss reconstruction for the “ positive words ” ( those that are non - zero in the target ) and an equal number of “ negative words. ” the negative words are chosen randomly, using a heuristic to sample words that are more likely to be mistaken. the bias introduced by this heuristic oversampling can then be corrected using importance weights. in all of these cases, the computational complexity of gradient estimation for the output layer is reduced to be proportional to the number of negative samples rather than proportional to the size of the output vector. 12. 4. 3. 4 noise - contrastive estimation and


all of these cases, the computational complexity of gradient estimation for the output layer is reduced to be proportional to the number of negative samples rather than proportional to the size of the output vector. 12. 4. 3. 4 noise - contrastive estimation and ranking loss other approaches based on sampling have been proposed to reduce the computa - tional cost of training neural language models with large vocabularies. an early example is the ranking loss proposed by collobert and weston 2008a ( ), which views the output of the neural language model for each word as a score and tries to make the score of the correct word ay be ranked high in comparison to the other 471


chapter 12. applications scores ai. the ranking loss proposed then is l = i max ( 0 1, −ay + ai ). ( 12. 19 ) the gradient is zero for the i - th term if the score of the observed word, ay, is greater than the score of the negative word ai by a margin of 1. one issue with this criterion is that it does not provide estimated conditional probabilities, which are useful in some applications, including speech recognition and text generation ( including conditional text generation tasks such as translation ). a more recently used training objective for neural language model is noise - contrastive estimation, which is introduced in section. this approach has 18. 6 been successfully applied to neural language models ( mnih and teh 2012 mnih, ; and kavukcuoglu 2013, ). 12. 4. 4 combining neural language models with - grams n a major advantage of n - gram models over neural networks is that n - gram models achieve high model capacity ( by storing the frequencies of very many tuples ) while requiring very little computation to process an example ( by looking up only a few tuples that match the current context ). if we use hash tables or trees to access the counts, the computation used for n -


frequencies of very many tuples ) while requiring very little computation to process an example ( by looking up only a few tuples that match the current context ). if we use hash tables or trees to access the counts, the computation used for n - grams is almost independent of capacity. in comparison, doubling a neural network ’ s number of parameters typically also roughly doubles its computation time. exceptions include models that avoid using all parameters on each pass. embedding layers index only a single embedding in each pass, so we can increase the vocabulary size without increasing the computation time per example. some other models, such as tiled convolutional networks, can add parameters while reducing the degree of parameter sharing in order to maintain the same amount of computation. however, typical neural network layers based on matrix multiplication use an amount of computation proportional to the number of parameters. one easy way to add capacity is thus to combine both approaches in an ensemble consisting of a neural language model and an n - gram language model ( bengio et al.,, ). as with any ensemble, this technique can reduce test error if 2001 2003 the ensemble members make independent mistakes. the ﬁeld of ensemble learning provides many ways of combining the ensemble members ’ predictions, including uniform weighting and


et al.,, ). as with any ensemble, this technique can reduce test error if 2001 2003 the ensemble members make independent mistakes. the ﬁeld of ensemble learning provides many ways of combining the ensemble members ’ predictions, including uniform weighting and weights chosen on a validation set. mikolov 2011a et al. ( ) extended the ensemble to include not just two models but a large array of models. it is also possible to pair a neural network with a maximum entropy model and train both jointly ( mikolov 2011b et al., ). this approach can be viewed as training 472


chapter 12. applications a neural network with an extra set of inputs that are connected directly to the output, and not connected to any other part of the model. the extra inputs are indicators for the presence of particular n - grams in the input context, so these variables are very high - dimensional and very sparse. the increase in model capacity is huge — the new portion of the architecture contains up to | | sv n parameters — but the amount of added computation needed to process an input is minimal because the extra inputs are very sparse. 12. 4. 5 neural machine translation machine translation is the task of reading a sentence in one natural language and emitting a sentence with the equivalent meaning in another language. machine translation systems often involve many components. at a high level, there is often one component that proposes many candidate translations. many of these translations will not be grammatical due to [UNK] between the languages. for example, many languages put adjectives after nouns, so when translated to english directly they yield phrases such as “ apple red. ” the proposal mechanism suggests many variants of the suggested translation, ideally including “ red apple. ” a second component of the translation system, a language model, evaluates the proposed translations, and can score “ red apple ” as better than “ apple red.


the proposal mechanism suggests many variants of the suggested translation, ideally including “ red apple. ” a second component of the translation system, a language model, evaluates the proposed translations, and can score “ red apple ” as better than “ apple red. ” the earliest use of neural networks for machine translation was to upgrade the language model of a translation system by using a neural language model ( schwenk et al., ; 2006 schwenk 2010, ). previously, most machine translation systems had used an n - gram model for this component. the n - gram based models used for machine translation include not just traditional back - [UNK] - gram models ( jelinek and mercer 1980 katz 1987 chen and goodman 1999, ;, ;, ) but also maximum entropy language models (, ), in which an [UNK] - softmax layer berger et al. 1996 predicts the next word given the presence of frequent - grams in the context. n traditional language models simply report the probability of a natural language sentence. because machine translation involves producing an output sentence given an input sentence, it makes sense to extend the natural language model to be conditional. as described in section, it is straightforward to extend a model 6. 2. 1. 1 that deﬁnes a marginal distribution


translation involves producing an output sentence given an input sentence, it makes sense to extend the natural language model to be conditional. as described in section, it is straightforward to extend a model 6. 2. 1. 1 that deﬁnes a marginal distribution over some variable to deﬁne a conditional distribution over that variable given a contextc, where c might be a single variable or a list of variables. ( ) beat the state - of - the - art in some statistical devlin et al. 2014 machine translation benchmarks by using an mlp to score a phrase t1, t2,..., tk in the target language given a phrase s1, s2,..., sn in the source language. the mlp estimates p ( t1, t2,..., tk | s1, s2,..., sn ). the estimate formed by this mlp replaces the estimate provided by conditional - gram models. n 473


chapter 12. applications decoder output object ( english sentence ) intermediate, semantic representation source object ( french sentence or image ) encoder figure 12. 5 : the encoder - decoder architecture to map back and forth between a surface representation ( such as a sequence of words or an image ) and a semantic representation. by using the output of an encoder of data from one modality ( such as the encoder mapping from french sentences to hidden representations capturing the meaning of sentences ) as the input to a decoder for another modality ( such as the decoder mapping from hidden representations capturing the meaning of sentences to english ), we can train systems that translate from one modality to another. this idea has been applied successfully not just to machine translation but also to caption generation from images. a drawback of the mlp - based approach is that it requires the sequences to be preprocessed to be of ﬁxed length. to make the translation more ﬂexible, we would like to use a model that can accommodate variable length inputs and variable length outputs. an rnn provides this ability. section describes several ways 10. 2. 4 of constructing an rnn that represents a conditional distribution over a sequence given some input, and section describes how


a model that can accommodate variable length inputs and variable length outputs. an rnn provides this ability. section describes several ways 10. 2. 4 of constructing an rnn that represents a conditional distribution over a sequence given some input, and section describes how to accomplish this conditioning 10. 4 when the input is a sequence. in all cases, one model ﬁrst reads the input sequence and emits a data structure that summarizes the input sequence. we call this summary the “ context ” c. the context c may be a list of vectors, or it may be a vector or tensor. the model that reads the input to produce c may be an rnn (, ; cho et al. 2014a sutskever 2014 jean 2014 et al., ; et al., ) or a convolutional network ( kalchbrenner and blunsom 2013, ). a second model, usually an rnn, then reads the context c and generates a sentence in the target language. this general idea of an encoder - decoder framework for machine translation is illustrated in ﬁgure. 12. 5 in order to generate an entire sentence conditioned on the source sentence, the model must have a way to represent the entire source sentence. earlier models


of an encoder - decoder framework for machine translation is illustrated in ﬁgure. 12. 5 in order to generate an entire sentence conditioned on the source sentence, the model must have a way to represent the entire source sentence. earlier models were only able to represent individual words or phrases. from a representation 474


chapter 12. applications learning point of view, it can be useful to learn a representation in which sentences that have the same meaning have similar representations regardless of whether they were written in the source language or the target language. this strategy was explored ﬁrst using a combination of convolutions and rnns ( kalchbrenner and blunsom 2013, ). later work introduced the use of an rnn for scoring proposed translations (, ) and for generating translated sentences ( cho et al. 2014a sutskever et al. et al., ). 2014 jean ( ) scaled these models to larger vocabularies. 2014 12. 4. 5. 1 using an attention mechanism and aligning pieces of data α ( t−1 ) α ( t−1 ) α ( ) t α ( ) t α ( + 1 ) t α ( + 1 ) t h ( t−1 ) h ( t−1 ) h ( ) t h ( ) t h ( + 1 ) t h ( + 1 ) t c × × × + figure 12. 6 : a modern attention mechanism, as introduced by ( ), is bahdanau et al. 2015 essentially a weighted average. a context vectorc is formed by taking a weighted average of feature vectors h


c × × × + figure 12. 6 : a modern attention mechanism, as introduced by ( ), is bahdanau et al. 2015 essentially a weighted average. a context vectorc is formed by taking a weighted average of feature vectors h ( ) t with weights α ( ) t. in some applications, the feature vectorsh are hidden units of a neural network, but they may also be raw input to the model. the weights α ( ) t are produced by the model itself. they are usually values in the interval [ 0, 1 ] and are intended to concentrate around just one h ( ) t so that the weighted average approximates reading that one speciﬁc time step precisely. the weightsα ( ) t are usually produced by applying a softmax function to relevance scores emitted by another portion of the model. the attention mechanism is more expensive computationally than directly indexing the desired h ( ) t, but direct indexing cannot be trained with gradient descent. the attention mechanism based on weighted averages is a smooth, [UNK] approximation that can be trained with existing optimization algorithms. using a ﬁxed - size representation to capture all the semantic details of a very long sentence of say 60 words is very [UNK]. it can be achieved by training a [UNK] large rn


a smooth, [UNK] approximation that can be trained with existing optimization algorithms. using a ﬁxed - size representation to capture all the semantic details of a very long sentence of say 60 words is very [UNK]. it can be achieved by training a [UNK] large rnn well enough and for long enough, as demonstrated by cho et al. ( ) and 2014a sutskever 2014 et al. ( ). however, a more [UNK] approach is to read the whole sentence or paragraph ( to get the context and the gist of what 475


chapter 12. applications is being expressed ), then produce the translated words one at a time, each time focusing on a [UNK] part of the input sentence in order to gather the semantic details that are required to produce the next output word. that is exactly the idea that ( ) ﬁrst introduced. the attention mechanism used bahdanau et al. 2015 to focus on speciﬁc parts of the input sequence at each time step is illustrated in ﬁgure. 12. 6 we can think of an attention - based system as having three components : 1. a process that “ reads ” raw data ( such as source words in a source sentence ), and converts them into distributed representations, with one feature vector associated with each word position. 2. a list of feature vectors storing the output of the reader. this can be understood as a “ ” containing a sequence of facts, which can be memory retrieved later, not necessarily in the same order, without having to visit all of them. 3. a process that “ ” the content of the memory to sequentially perform exploits a task, at each time step having the ability put attention on the content of one memory element ( or a few, with a [UNK] weight ). the third component generates the translated sentence. when words


the content of the memory to sequentially perform exploits a task, at each time step having the ability put attention on the content of one memory element ( or a few, with a [UNK] weight ). the third component generates the translated sentence. when words in a sentence written in one language are aligned with correspond - ing words in a translated sentence in another language, it becomes possible to relate the corresponding word embeddings. earlier work showed that one could learn a kind of translation matrix relating the word embeddings in one language with the word embeddings in another ( kocisky 2014 et al., ), yielding lower alignment error rates than traditional approaches based on the frequency counts in the phrase table. there is even earlier work on learning cross - lingual word vectors ( klementiev et al., 2012 ). many extensions to this approach are possible. for example, more [UNK] cross - lingual alignment (, ) allows training on larger datasets. gouws et al. 2014 12. 4. 6 historical perspective the idea of distributed representations for symbols was introduced by rumelhart et al. ( ) in one of the ﬁrst explorations of back - propagation, with symbols 1986a corresponding to the identity of family members


2014 12. 4. 6 historical perspective the idea of distributed representations for symbols was introduced by rumelhart et al. ( ) in one of the ﬁrst explorations of back - propagation, with symbols 1986a corresponding to the identity of family members and the neural network capturing the relationships between family members, with training examples forming triplets such as ( colin, mother, victoria ). the ﬁrst layer of the neural network learned a representation of each family member. for example, the features for colin 476


chapter 12. applications might represent which family tree colin was in, what branch of that tree he was in, what generation he was from, etc. one can think of the neural network as computing learned rules relating these attributes together in order to obtain the desired predictions. the model can then make predictions such as inferring who is the mother of colin. the idea of forming an embedding for a symbol was extended to the idea of an embedding for a word by deerwester 1990 et al. ( ). these embeddings were learned using the svd. later, embeddings would be learned by neural networks. the history of natural language processing is marked by transitions in the popularity of [UNK] ways of representing the input to the model. following this early work on symbols or words, some of the earliest applications of neural networks to nlp (, ; miikkulainen and dyer 1991 schmidhuber 1996, ) represented the input as a sequence of characters. bengio 2001 et al. ( ) returned the focus to modeling words and introduced neural language models, which produce interpretable word embeddings. these neural models have scaled up from deﬁning representations of a small set of symbols in the 1980s to millions of words


al. ( ) returned the focus to modeling words and introduced neural language models, which produce interpretable word embeddings. these neural models have scaled up from deﬁning representations of a small set of symbols in the 1980s to millions of words ( including proper nouns and misspellings ) in modern applications. this computational scaling [UNK] led to the invention of the techniques described above in section. 12. 4. 3 initially, the use of words as the fundamental units of language models yielded improved language modeling performance (, ). to this day, bengio et al. 2001 new techniques continually push both character - based models ( sutskever et al., 2011 ) and word - based models forward, with recent work (, ) even gillick et al. 2015 modeling individual bytes of unicode characters. the ideas behind neural language models have been extended into several natural language processing applications, such as parsing (,, ; henderson 2003 2004 collobert 2011, ), part - of - speech tagging, semantic role labeling, chunking, etc, sometimes using a single multi - task learning architecture ( collobert and weston, 2008a collobert 2011a ; et al., ) in which the word embeddings are shared across tasks. two


labeling, chunking, etc, sometimes using a single multi - task learning architecture ( collobert and weston, 2008a collobert 2011a ; et al., ) in which the word embeddings are shared across tasks. two - dimensional visualizations of embeddings became a popular tool for an - alyzing language models following the development of the t - sne dimensionality reduction algorithm ( van der maaten and hinton 2008, ) and its high - proﬁle appli - cation to visualization word embeddings by joseph turian in 2009. 477


chapter 12. applications 12. 5 other applications in this section we cover a few other types of applications of deep learning that are [UNK] from the standard object recognition, speech recognition and natural language processing tasks discussed above. part of this book will expand that iii scope even further to tasks that remain primarily research areas. 12. 5. 1 recommender systems one of the major families of applications of machine learning in the information technology sector is the ability to make recommendations of items to potential users or customers. two major types of applications can be distinguished : online advertising and item recommendations ( often these recommendations are still for the purpose of selling a product ). both rely on predicting the association between a user and an item, either to predict the probability of some action ( the user buying the product, or some proxy for this action ) or the expected gain ( which may depend on the value of the product ) if an ad is shown or a recommendation is made regarding that product to that user. the internet is currently ﬁnanced in great part by various forms of online advertising. there are major parts of the economy that rely on online shopping. companies including amazon and ebay use machine learning, including deep learning, for their product recommendations. sometimes, the items are not products that are actually for sale.


forms of online advertising. there are major parts of the economy that rely on online shopping. companies including amazon and ebay use machine learning, including deep learning, for their product recommendations. sometimes, the items are not products that are actually for sale. examples include selecting posts to display on social network news feeds, recommending movies to watch, recommending jokes, recommending advice from experts, matching players for video games, or matching people in dating services. often, this association problem is handled like a supervised learning problem : given some information about the item and about the user, predict the proxy of interest ( user clicks on ad, user enters a rating, user clicks on a “ like ” button, user buys product, user spends some amount of money on the product, user spends time visiting a page for the product, etc ). this often ends up being either a regression problem ( predicting some conditional expected value ) or a probabilistic classiﬁcation problem ( predicting the conditional probability of some discrete event ). the early work on recommender systems relied on minimal information as inputs for these predictions : the user id and the item id. in this context, the only way to generalize is to rely on the similarity between the patterns of values of the target variable for


early work on recommender systems relied on minimal information as inputs for these predictions : the user id and the item id. in this context, the only way to generalize is to rely on the similarity between the patterns of values of the target variable for [UNK] users or for [UNK] items. suppose that user 1 and user 2 both like items a, b and c. from this, we may infer that user 1 and 478


chapter 12. applications user 2 have similar tastes. if user 1 likes item d, then this should be a strong cue that user 2 will also like d. algorithms based on this principle come under the name of collaborative ﬁltering. both non - parametric approaches ( such as nearest - neighbor methods based on the estimated similarity between patterns of preferences ) and parametric methods are possible. parametric methods often rely on learning a distributed representation ( also called an embedding ) for each user and for each item. bilinear prediction of the target variable ( such as a rating ) is a simple parametric method that is highly successful and often found as a component of state - of - the - art systems. the prediction is obtained by the dot product between the user embedding and the item embedding ( possibly corrected by constants that depend only on either the user id or the item id ). let [UNK] be the matrix containing our predictions, a a matrix with user embeddings in its rows and b a matrix with item embeddings in its columns. let b and c be vectors that contain respectively a kind of bias for each user ( representing how grumpy or positive that user is in general ) and for each item ( representing its general


a matrix with item embeddings in its columns. let b and c be vectors that contain respectively a kind of bias for each user ( representing how grumpy or positive that user is in general ) and for each item ( representing its general popularity ). the bilinear prediction is thus obtained as follows : [UNK], i = bu + ci + j au, jbj, i. ( 12. 20 ) typically one wants to minimize the squared error between predicted ratings [UNK], i and actual ratings ru, i. user embeddings and item embeddings can then be conveniently visualized when they are ﬁrst reduced to a low dimension ( two or three ), or they can be used to compare users or items against each other, just like word embeddings. one way to obtain these embeddings is by performing a singular value decomposition of the matrix r of actual targets ( such as ratings ). this corresponds to factorizing r = udv ( or a normalized variant ) into the product of two factors, the lower rank matrices a = ud and b = v. one problem with the svd is that it treats the missing entries in an arbitrary way, as if they corresponded to a target value of 0.


variant ) into the product of two factors, the lower rank matrices a = ud and b = v. one problem with the svd is that it treats the missing entries in an arbitrary way, as if they corresponded to a target value of 0. instead we would like to avoid paying any cost for the predictions made on missing entries. fortunately, the sum of squared errors on the observed ratings can also be easily minimized by gradient - based optimization. the svd and the bilinear prediction of equation both 12. 20 performed very well in the competition for the netﬂix prize (, bennett and lanning 2007 ), aiming at predicting ratings for ﬁlms, based only on previous ratings by a large set of anonymous users. many machine learning experts participated in this competition, which took place between 2006 and 2009. it raised the level of research in recommender systems using advanced machine learning and yielded improvements in recommender systems. even though it did not win by itself, the simple bilinear prediction or svd was a component of the ensemble models 479


chapter 12. applications presented by most of the competitors, including the winners (, ; toscher et al. 2009 koren 2009, ). beyond these bilinear models with distributed representations, one of the ﬁrst uses of neural networks for collaborative ﬁltering is based on the rbm undirected probabilistic model ( salakhutdinov 2007 et al., ). rbms were an important element of the ensemble of methods that won the netﬂix competition ( toscher 2009 et al., ; koren 2009, ). more advanced variants on the idea of factorizing the ratings matrix have also been explored in the neural networks community ( salakhutdinov and mnih 2008, ). however, there is a basic limitation of collaborative ﬁltering systems : when a new item or a new user is introduced, its lack of rating history means that there is no way to evaluate its similarity with other items or users ( respectively ), or the degree of association between, say, that new user and existing items. this is called the problem of cold - start recommendations. a general way of solving the cold - start recommendation problem is to introduce extra information about the individual users and items. for example, this extra information could be


, that new user and existing items. this is called the problem of cold - start recommendations. a general way of solving the cold - start recommendation problem is to introduce extra information about the individual users and items. for example, this extra information could be user proﬁle information or features of each item. systems that use such information are called content - based recommender systems. the mapping from a rich set of user features or item features to an embedding can be learned through a deep learning architecture (, ; huang et al. 2013 elkahky 2015 et al., ). specialized deep learning architectures such as convolutional networks have also been applied to learn to extract features from rich content such as from musical audio tracks, for music recommendation ( van den oord 2013 et al., ). in that work, the convolutional net takes acoustic features as input and computes an embedding for the associated song. the dot product between this song embedding and the embedding for a user is then used to predict whether a user will listen to the song. 12. 5. 1. 1 exploration versus exploitation when making recommendations to users, an issue arises that goes beyond ordinary supervised learning and into the realm of reinforcement learning


##ding for a user is then used to predict whether a user will listen to the song. 12. 5. 1. 1 exploration versus exploitation when making recommendations to users, an issue arises that goes beyond ordinary supervised learning and into the realm of reinforcement learning. many recom - mendation problems are most accurately described theoretically as contextual bandits (, ;, ). the issue is that when we langford and zhang 2008 lu et al. 2010 use the recommendation system to collect data, we get a biased and incomplete view of the preferences of users : we only see the responses of users to the items they were recommended and not to the other items. in addition, in some cases we may not get any information on users for whom no recommendation has been made ( for example, with ad auctions, it may be that the price proposed for an 480


chapter 12. applications ad was below a minimum price threshold, or does not win the auction, so the ad is not shown at all ). more importantly, we get no information about what outcome would have resulted from recommending any of the other items. this would be like training a classiﬁer by picking one class [UNK] for each training example x ( typically the class with the highest probability according to the model ) and then only getting as feedback whether this was the correct class or not. clearly, each example conveys less information than in the supervised case where the true label y is directly accessible, so more examples are necessary. worse, if we are not careful, we could end up with a system that continues picking the wrong decisions even as more and more data is collected, because the correct decision initially had a very low probability : until the learner picks that correct decision, it does not learn about the correct decision. this is similar to the situation in reinforcement learning where only the reward for the selected action is observed. in general, reinforcement learning can involve a sequence of many actions and many rewards. the bandits scenario is a special case of reinforcement learning, in which the learner takes only a single action and receives a single reward. the bandit problem is easier in the


in general, reinforcement learning can involve a sequence of many actions and many rewards. the bandits scenario is a special case of reinforcement learning, in which the learner takes only a single action and receives a single reward. the bandit problem is easier in the sense that the learner knows which reward is associated with which action. in the general reinforcement learning scenario, a high reward or a low reward might have been caused by a recent action or by an action in the distant past. the term contextual bandits refers to the case where the action is taken in the context of some input variable that can inform the decision. for example, we at least know the user identity, and we want to pick an item. the mapping from context to action is also called a policy. the feedback loop between the learner and the data distribution ( which now depends on the actions of the learner ) is a central research issue in the reinforcement learning and bandits literature. reinforcement learning requires choosing a [UNK] exploration and exploitation. exploitation refers to taking actions that come from the current, best version of the learned policy — actions that we know will achieve a high reward. exploration refers to taking actions speciﬁcally in order to obtain more training data. if we know that given context x, action a gives


from the current, best version of the learned policy — actions that we know will achieve a high reward. exploration refers to taking actions speciﬁcally in order to obtain more training data. if we know that given context x, action a gives us a reward of 1, we do not know whether that is the best possible reward. we may want to exploit our current policy and continue taking action a in order to be relatively sure of obtaining a reward of 1. however, we may also want to explore by trying action a. we do not know what will happen if we try action a. we hope to get a reward of, but we 2 run the risk of getting a reward of. either way, we at least gain some knowledge. 0 exploration can be implemented in many ways, ranging from occasionally taking random actions intended to cover the entire space of possible actions, to model - based approaches that compute a choice of action based on its expected reward and the model ’ s amount of uncertainty about that reward. 481


chapter 12. applications many factors determine the extent to which we prefer exploration or exploitation. one of the most prominent factors is the time scale we are interested in. if the agent has only a short amount of time to accrue reward, then we prefer more exploitation. if the agent has a long time to accrue reward, then we begin with more exploration so that future actions can be planned more [UNK] with more knowledge. as time progresses and our learned policy improves, we move toward more exploitation. supervised learning has no [UNK] between exploration and exploitation because the supervision signal always speciﬁes which output is correct for each input. there is no need to try out [UNK] outputs to determine if one is better than the model ’ s current output — we always know that the label is the best output. another [UNK] arising in the context of reinforcement learning, besides the exploration - exploitation trade - [UNK], is the [UNK] of evaluating and comparing [UNK] policies. reinforcement learning involves interaction between the learner and the environment. this feedback loop means that it is not straightforward to evaluate the learner ’ s performance using a ﬁxed set of test set input values. the policy itself determines which inputs will be seen. ( ) present dudik et al. 2011 techniques for evaluating contextual bandits


that it is not straightforward to evaluate the learner ’ s performance using a ﬁxed set of test set input values. the policy itself determines which inputs will be seen. ( ) present dudik et al. 2011 techniques for evaluating contextual bandits. 12. 5. 2 knowledge representation, reasoning and question an - swering deep learning approaches have been very successful in language modeling, machine translation and natural language processing due to the use of embeddings for symbols (, ) and words ( rumelhart et al. 1986a deerwester 1990 bengio et al., ; et al., 2001 ). these embeddings represent semantic knowledge about individual words and concepts. a research frontier is to develop embeddings for phrases and for relations between words and facts. search engines already use machine learning for this purpose but much more remains to be done to improve these more advanced representations. 12. 5. 2. 1 knowledge, relations and question answering one interesting research direction is determining how distributed representations can be trained to capture the relations between two entities. these relations allow us to formalize facts about objects and how objects interact with each other. in mathematics, a binary relation is a set of ordered pairs of objects. pairs that are in the set are


be trained to capture the relations between two entities. these relations allow us to formalize facts about objects and how objects interact with each other. in mathematics, a binary relation is a set of ordered pairs of objects. pairs that are in the set are said to have the relation while those who are not in the set 482


chapter 12. applications do not. for example, we can deﬁne the relation “ is less than ” on the set of entities { 1, 2, 3 } by deﬁning the set of ordered pairs s = { ( 1, 2 ), ( 1, 3 ), ( 2, 3 ) }. once this relation is deﬁned, we can use it like a verb. because ( 1, 2 ) ∈s, we say that 1 is less than 2. because ( 2, 1 ) ∈s, we can not say that 2 is less than 1. of course, the entities that are related to one another need not be numbers. we could deﬁne a relation containing tuples like (, ). is _ a _ type _ of dog mammal in the context of ai, we think of a relation as a sentence in a syntactically simple and highly structured language. the relation plays the role of a verb, while two arguments to the relation play the role of its subject and object. these sentences take the form of a triplet of tokens ( subject verb object ),, ( 12. 21 ) with values ( entityi, relationj, entityk ). ( 12. 22 ) we can also deﬁ


and object. these sentences take the form of a triplet of tokens ( subject verb object ),, ( 12. 21 ) with values ( entityi, relationj, entityk ). ( 12. 22 ) we can also deﬁne an attribute, a concept analogous to a relation, but taking only one argument : ( entityi, attribute j ). ( 12. 23 ) for example, we could deﬁne the has _ fur attribute, and apply it to entities like dog. many applications require representing relations and reasoning about them. how should we best do this within the context of neural networks? machine learning models of course require training data. we can infer relations between entities from training datasets consisting of unstructured natural language. there are also structured databases that identify relations explicitly. a common structure for these databases is the relational database, which stores this same kind of information, albeit not formatted as three token sentences. when a database is intended to convey commonsense knowledge about everyday life or expert knowledge about an application area to an artiﬁcial intelligence system, we call the database a knowledge base. knowledge bases range from general ones like freebase, opencyc, wordnet, or wikibase, 1 etc. to more


expert knowledge about an application area to an artiﬁcial intelligence system, we call the database a knowledge base. knowledge bases range from general ones like freebase, opencyc, wordnet, or wikibase, 1 etc. to more specialized knowledge bases, like geneontology. 2 representations for entities and relations can be learned by considering each triplet in a knowledge base as a training example and maximizing a training objective that captures their joint distribution ( bordes et al., ). 2013a 1respectively available from these web sites : freebase. com, cyc. com / opencyc, wordnet. princeton. edu wikiba. se, 2geneontology. org 483


chapter 12. applications in addition to training data, we also need to deﬁne a model family to train. a common approach is to extend neural language models to model entities and relations. neural language models learn a vector that provides a distributed representation of each word. they also learn about interactions between words, such as which word is likely to come after a sequence of words, by learning functions of these vectors. we can extend this approach to entities and relations by learning an embedding vector for each relation. in fact, the parallel between modeling language and modeling knowledge encoded as relations is so close that researchers have trained representations of such entities by using both and knowledge bases natural language sentences (,, ; bordes et al. 2011 2012 wang 2014a et al., ) or combining data from multiple relational databases (, ). many bordes et al. 2013b possibilities exist for the particular parametrization associated with such a model. early work on learning about relations between entities (, paccanaro and hinton 2000 ) posited highly constrained parametric forms ( “ linear relational embeddings ” ), often using a [UNK] form of representation for the relation than for the entities. for example, paccanaro and hinton 2000 bordes 2011 ( ) and


posited highly constrained parametric forms ( “ linear relational embeddings ” ), often using a [UNK] form of representation for the relation than for the entities. for example, paccanaro and hinton 2000 bordes 2011 ( ) and et al. ( ) used vectors for entities and matrices for relations, with the idea that a relation acts like an operator on entities. alternatively, relations can be considered as any other entity ( bordes et al., ), allowing us to make statements about relations, but more ﬂexibility is 2012 put in the machinery that combines them in order to model their joint distribution. a practical short - term application of such models is link prediction : predict - ing missing arcs in the knowledge graph. this is a form of generalization to new facts, based on old facts. most of the knowledge bases that currently exist have been constructed through manual labor, which tends to leave many and probably the majority of true relations absent from the knowledge base. see wang et al. ( ), ( ) and ( ) for examples of such an 2014b lin et al. 2015 garcia - duran et al. 2015 application. evaluating the performance of a model on a link prediction task is [UNK] because we have only a dataset of positive examples (