chapter 9. convolutional networks convolutional layer input to layer convolution stage : a ne transform [UNK] detector stage : nonlinearity e. g., rectiﬁed linear pooling stage next layer input to layers convolution layer : a ne transform [UNK] detector layer : nonlinearity e. g., rectiﬁed linear pooling layer next layer complex layer terminology simple layer terminology figure 9. 7 : the components of a typical convolutional neural network layer. there are two commonly used sets of terminology for describing these layers. ( left ) in this terminology, the convolutional net is viewed as a small number of relatively complex layers, with each layer having many “ stages. ” in this terminology, there is a one - to - one mapping between kernel tensors and network layers. in this book we generally use this terminology. ( right ) in this terminology, the convolutional net is viewed as a larger number of simple layers ; every step of processing is regarded as a layer in its own right. this means that not every “ layer ” has parameters. 341


chapter 9. convolutional networks and chellappa 1988, ) operation reports the maximum output within a rectangular neighborhood. other popular pooling functions include the average of a rectangular neighborhood, the l2 norm of a rectangular neighborhood, or a weighted average based on the distance from the central pixel. in all cases, pooling helps to make the representation become approximately invariant to small translations of the input. invariance to translation means that if we translate the input by a small amount, the values of most of the pooled outputs do not change. see ﬁgure for an example of how this works. 9. 8 invariance to local translation can be a very useful property if we care more about whether some feature is present than exactly where it is. for example, when determining whether an image contains a face, we need not know the location of the eyes with pixel - perfect accuracy, we just need to know that there is an eye on the left side of the face and an eye on the right side of the face. in other contexts, it is more important to preserve the location of a feature. for example, if we want to ﬁnd a corner deﬁned by two edges meeting at a speciﬁc orientation, we need to preserve

/home/ricoiban/GEMMA/mnlp_chatsplaining/RAG/Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville (z-lib.org).pdf


of the face. in other contexts, it is more important to preserve the location of a feature. for example, if we want to ﬁnd a corner deﬁned by two edges meeting at a speciﬁc orientation, we need to preserve the location of the edges well enough to test whether they meet. the use of pooling can be viewed as adding an inﬁnitely strong prior that the function the layer learns must be invariant to small translations. when this assumption is correct, it can greatly improve the statistical [UNK] of the network. pooling over spatial regions produces invariance to translation, but if we pool over the outputs of separately parametrized convolutions, the features can learn which transformations to become invariant to ( see ﬁgure ). 9. 9 because pooling summarizes the responses over a whole neighborhood, it is possible to use fewer pooling units than detector units, by reporting summary statistics for pooling regions spaced k pixels apart rather than 1 pixel apart. see ﬁgure for an example. this improves the computational [UNK] of the 9. 10 network because the next layer has roughly k times fewer inputs to process. when the number of parameters in the next layer is a function of its input size ( such as when


##gure for an example. this improves the computational [UNK] of the 9. 10 network because the next layer has roughly k times fewer inputs to process. when the number of parameters in the next layer is a function of its input size ( such as when the next layer is fully connected and based on matrix multiplication ) this reduction in the input size can also result in improved statistical [UNK] and reduced memory requirements for storing the parameters. for many tasks, pooling is essential for handling inputs of varying size. for example, if we want to classify images of variable size, the input to the classiﬁcation layer must have a ﬁxed size. this is usually accomplished by varying the size of an [UNK] between pooling regions so that the classiﬁcation layer always receives the same number of summary statistics regardless of the input size. for example, the ﬁnal pooling layer of the network may be deﬁned to output four sets of summary statistics, one for each quadrant of an image, regardless of the image size. 342


chapter 9. convolutional networks 0. 1 1. 0. 2 1. 1. 1. 0. 1 0. 2............ 0. 3 0. 1 1. 1. 0. 3 1. 0. 2 1............. detector stage pooling stage pooling stage detector stage figure 9. 8 : max pooling introduces invariance. ( top ) a view of the middle of the output of a convolutional layer. the bottom row shows outputs of the nonlinearity. the top row shows the outputs of max pooling, with a stride of one pixel between pooling regions and a pooling region width of three pixels. a view of the same network, after ( bottom ) the input has been shifted to the right by one pixel. every value in the bottom row has changed, but only half of the values in the top row have changed, because the max pooling units are only sensitive to the maximum value in the neighborhood, not its exact location. 343


chapter 9. convolutional networks large response in pooling unit large response in pooling unit large response in detector unit 1 large response in detector unit 3 figure 9. 9 : example of learned invariances : a pooling unit that pools over multiple features that are learned with separate parameters can learn to be invariant to transformations of the input. here we show how a set of three learned ﬁlters and a max pooling unit can learn to become invariant to rotation. all three ﬁlters are intended to detect a hand - written 5. each ﬁlter attempts to match a slightly [UNK] orientation of the 5. when a 5 appears in the input, the corresponding ﬁlter will match it and cause a large activation in a detector unit. the max pooling unit then has a large activation regardless of which detector unit was activated. we show here how the network processes two [UNK] inputs, resulting in two [UNK] detector units being activated. the [UNK] on the pooling unit is roughly the same either way. this principle is leveraged by maxout networks ( goodfellow et al., 2013a ) and other convolutional networks. max pooling over spatial positions is naturally invariant to translation ; this multi - channel approach is only necessary for learning other


is leveraged by maxout networks ( goodfellow et al., 2013a ) and other convolutional networks. max pooling over spatial positions is naturally invariant to translation ; this multi - channel approach is only necessary for learning other transformations. 0. 1 1. 0. 2 1. 0. 2 0. 1 0. 1 0. 0 0. 1 figure 9. 10 : pooling with downsampling. here we use max - pooling with a pool width of three and a stride between pools of two. this reduces the representation size by a factor of two, which reduces the computational and statistical burden on the next layer. note that the rightmost pooling region has a smaller size, but must be included if we do not want to ignore some of the detector units. 344


chapter 9. convolutional networks some theoretical work gives guidance as to which kinds of pooling one should use in various situations (, ). it is also possible to dynamically boureau et al. 2010 pool features together, for example, by running a clustering algorithm on the locations of interesting features (, ). this approach yields a boureau et al. 2011 [UNK] set of pooling regions for each image. another approach is to learn a single pooling structure that is then applied to all images (, ). jia et al. 2012 pooling can complicate some kinds of neural network architectures that use top - down information, such as boltzmann machines and autoencoders. these issues will be discussed further when we present these types of networks in part. iii pooling in convolutional boltzmann machines is presented in section. the 20. 6 inverse - like operations on pooling units needed in some [UNK] networks will be covered in section. 20. 10. 6 some examples of complete convolutional network architectures for classiﬁcation using convolution and pooling are shown in ﬁgure. 9. 11 9. 4 convolution and pooling as an


10. 6 some examples of complete convolutional network architectures for classiﬁcation using convolution and pooling are shown in ﬁgure. 9. 11 9. 4 convolution and pooling as an inﬁnitely strong prior recall the concept of a prior probability distribution from section. this is 5. 2 a probability distribution over the parameters of a model that encodes our beliefs about what models are reasonable, before we have seen any data. priors can be considered weak or strong depending on how concentrated the probability density in the prior is. a weak prior is a prior distribution with high entropy, such as a gaussian distribution with high variance. such a prior allows the data to move the parameters more or less freely. a strong prior has very low entropy, such as a gaussian distribution with low variance. such a prior plays a more active role in determining where the parameters end up. an inﬁnitely strong prior places zero probability on some parameters and says that these parameter values are completely forbidden, regardless of how much support the data gives to those values. we can imagine a convolutional net as being similar to a fully connected net, but with an inﬁnitely strong


and says that these parameter values are completely forbidden, regardless of how much support the data gives to those values. we can imagine a convolutional net as being similar to a fully connected net, but with an inﬁnitely strong prior over its weights. this inﬁnitely strong prior says that the weights for one hidden unit must be identical to the weights of its neighbor, but shifted in space. the prior also says that the weights must be zero, except for in the small, spatially contiguous receptive ﬁeld assigned to that hidden unit. overall, we can think of the use of convolution as introducing an inﬁnitely strong prior probability distribution over the parameters of a layer. this prior 345


chapter 9. convolutional networks input image : 256x256x3 output of convolution + relu : 256x256x64 output of pooling with stride 4 : 64x64x64 output of convolution + relu : 64x64x64 output of pooling with stride 4 : 16x16x64 output of reshape to vector : 16, 384 units output of matrix multiply : 1, 000 units output of softmax : 1, 000 class probabilities input image : 256x256x3 output of convolution + relu : 256x256x64 output of pooling with stride 4 : 64x64x64 output of convolution + relu : 64x64x64 output of pooling to 3x3 grid : 3x3x64 output of reshape to vector : 576 units output of matrix multiply : 1, 000 units output of softmax : 1, 000 class probabilities input image : 256x256x3 output of convolution + relu : 256x256x64 output of pooling with stride 4 : 64x64x64 output of convolution +


000 class probabilities input image : 256x256x3 output of convolution + relu : 256x256x64 output of pooling with stride 4 : 64x64x64 output of convolution + relu : 64x64x64 output of convolution : 16x16x1, 000 output of average pooling : 1x1x1, 000 output of softmax : 1, 000 class probabilities output of pooling with stride 4 : 16x16x64 figure 9. 11 : examples of architectures for classiﬁcation with convolutional networks. the speciﬁc strides and depths used in this ﬁgure are not advisable for real use ; they are designed to be very shallow in order to ﬁt onto the page. real convolutional networks also often involve signiﬁcant amounts of branching, unlike the chain structures used here for simplicity. ( left ) a convolutional network that processes a ﬁxed image size. after alternating between convolution and pooling for a few layers, the tensor for the convolutional feature map is reshaped to ﬂatten out the spatial dimensions.


##tional network that processes a ﬁxed image size. after alternating between convolution and pooling for a few layers, the tensor for the convolutional feature map is reshaped to ﬂatten out the spatial dimensions. the rest of the network is an ordinary feedforward network classiﬁer, as described in chapter. 6 ( center ) a convolutional network that processes a variable - sized image, but still maintains a fully connected section. this network uses a pooling operation with variably - sized pools but a ﬁxed number of pools, in order to provide a ﬁxed - size vector of 576 units to the fully connected portion of the network. a convolutional network that does not ( right ) have any fully connected weight layer. instead, the last convolutional layer outputs one feature map per class. the model presumably learns a map of how likely each class is to occur at each spatial location. averaging a feature map down to a single value provides the argument to the softmax classiﬁer at the top. 346


chapter 9. convolutional networks says that the function the layer should learn contains only local interactions and is equivariant to translation. likewise, the use of pooling is an inﬁnitely strong prior that each unit should be invariant to small translations. of course, implementing a convolutional net as a fully connected net with an inﬁnitely strong prior would be extremely computationally wasteful. but thinking of a convolutional net as a fully connected net with an inﬁnitely strong prior can give us some insights into how convolutional nets work. one key insight is that convolution and pooling can cause underﬁtting. like any prior, convolution and pooling are only useful when the assumptions made by the prior are reasonably accurate. if a task relies on preserving precise spatial information, then using pooling on all features can increase the training error. some convolutional network architectures (, ) are designed to szegedy et al. 2014a use pooling on some channels but not on other channels, in order to get both highly invariant features and features that will not underﬁt when the translation invariance prior is incorrect


) are designed to szegedy et al. 2014a use pooling on some channels but not on other channels, in order to get both highly invariant features and features that will not underﬁt when the translation invariance prior is incorrect. when a task involves incorporating information from very distant locations in the input, then the prior imposed by convolution may be inappropriate. another key insight from this view is that we should only compare convolu - tional models to other convolutional models in benchmarks of statistical learning performance. models that do not use convolution would be able to learn even if we permuted all of the pixels in the image. for many image datasets, there are separate benchmarks for models that are permutation invariant and must discover the concept of topology via learning, and models that have the knowledge of spatial relationships hard - coded into them by their designer. 9. 5 variants of the basic convolution function when discussing convolution in the context of neural networks, we usually do not refer exactly to the standard discrete convolution operation as it is usually understood in the mathematical literature. the functions used in practice [UNK] slightly. here we describe these [UNK] in detail, and


##tion in the context of neural networks, we usually do not refer exactly to the standard discrete convolution operation as it is usually understood in the mathematical literature. the functions used in practice [UNK] slightly. here we describe these [UNK] in detail, and highlight some useful properties of the functions used in neural networks. first, when we refer to convolution in the context of neural networks, we usually actually mean an operation that consists of many applications of convolution in parallel. this is because convolution with a single kernel can only extract one kind of feature, albeit at many spatial locations. usually we want each layer of our network to extract many kinds of features, at many locations. 347


chapter 9. convolutional networks additionally, the input is usually not just a grid of real values. rather, it is a grid of vector - valued observations. for example, a color image has a red, green and blue intensity at each pixel. in a multilayer convolutional network, the input to the second layer is the output of the ﬁrst layer, which usually has the output of many [UNK] convolutions at each position. when working with images, we usually think of the input and output of the convolution as being 3 - d tensors, with one index into the [UNK] channels and two indices into the spatial coordinates of each channel. software implementations usually work in batch mode, so they will actually use 4 - d tensors, with the fourth axis indexing [UNK] examples in the batch, but we will omit the batch axis in our description here for simplicity. because convolutional networks usually use multi - channel convolution, the linear operations they are based on are not guaranteed to be commutative, even if kernel - ﬂipping is used. these multi - channel operations are only commutative if each operation has the same number of output channels as input channels. assume we have


they are based on are not guaranteed to be commutative, even if kernel - ﬂipping is used. these multi - channel operations are only commutative if each operation has the same number of output channels as input channels. assume we have a 4 - d kernel tensor k with element ki, j, k, l giving the connection strength between a unit in channel i of the output and a unit in channel j of the input, with an [UNK] of k rows and l columns between the output unit and the input unit. assume our input consists of observed data v with element vi, j, k giving the value of the input unit within channel i at row j and column k. assume our output consists of z with the same format as v. if z is produced by convolving k across without ﬂipping, then v k zi, j, k = l, m, n vl, j m, k n + −1 + −1ki, l, m, n ( 9. 7 ) where the summation over l, m and n is over all values for which the tensor indexing operations inside the summation is valid. in linear algebra notation, we index into arrays using a for the ﬁrst entry. this necess


7 ) where the summation over l, m and n is over all values for which the tensor indexing operations inside the summation is valid. in linear algebra notation, we index into arrays using a for the ﬁrst entry. this necessitates the 1 −1 in the above formula. programming languages such as c and python index starting from, rendering 0 the above expression even simpler. we may want to skip over some positions of the kernel in order to reduce the computational cost ( at the expense of not extracting our features as ﬁnely ). we can think of this as downsampling the output of the full convolution function. if we want to sample only every s pixels in each direction in the output, then we can deﬁne a downsampled convolution function such that c zi, j, k = ( ) c k v,, s i, j, k = l, m, n vl, j s m, k s n ( − × 1 ) + ( − × 1 ) + ki, l, m, n. ( 9. 8 ) we refer to s as the stride of this downsampled convolution. it is also possible 348


chapter 9. convolutional networks to deﬁne a separate stride for each direction of motion. see ﬁgure for an 9. 12 illustration. one essential feature of any convolutional network implementation is the ability to implicitly zero - pad the input v in order to make it wider. without this feature, the width of the representation shrinks by one pixel less than the kernel width at each layer. zero padding the input allows us to control the kernel width and the size of the output independently. without zero padding, we are forced to choose between shrinking the spatial extent of the network rapidly and using small kernels — both scenarios that signiﬁcantly limit the expressive power of the network. see ﬁgure for an example. 9. 13 three special cases of the zero - padding setting are worth mentioning. one is the extreme case in which no zero - padding is used whatsoever, and the convolution kernel is only allowed to visit positions where the entire kernel is contained entirely within the image. in matlab terminology, this is called valid convolution. in this case, all pixels in the output are a function of the same number of pixels in the input, so the behavior of an output pixel


is contained entirely within the image. in matlab terminology, this is called valid convolution. in this case, all pixels in the output are a function of the same number of pixels in the input, so the behavior of an output pixel is somewhat more regular. however, the size of the output shrinks at each layer. if the input image has width m and the kernel has width k, the output will be of width m k − + 1. the rate of this shrinkage can be dramatic if the kernels used are large. since the shrinkage is greater than 0, it limits the number of convolutional layers that can be included in the network. as layers are added, the spatial dimension of the network will eventually drop to 1 × 1, at which point additional layers cannot meaningfully be considered convolutional. another special case of the zero - padding setting is when just enough zero - padding is added to keep the size of the output equal to the size of the input. matlab calls this same convolution. in this case, the network can contain as many convolutional layers as the available hardware can support, since the operation of convolution does not modify the architectural possibilities


the input. matlab calls this same convolution. in this case, the network can contain as many convolutional layers as the available hardware can support, since the operation of convolution does not modify the architectural possibilities available to the next layer. however, the input pixels near the border inﬂuence fewer output pixels than the input pixels near the center. this can make the border pixels somewhat underrepresented in the model. this motivates the other extreme case, which matlab refers to as full convolution, in which enough zeroes are added for every pixel to be visited k times in each direction, resulting in an output image of width m + k −1. in this case, the output pixels near the border are a function of fewer pixels than the output pixels near the center. this can make it [UNK] to learn a single kernel that performs well at all positions in the convolutional feature map. usually the optimal amount of zero padding ( in terms of test set classiﬁcation accuracy ) lies somewhere between “ valid ” and “ same ” convolution. 349


chapter 9. convolutional networks x1 x1 x2 x2 x3 x3 s1s1 s2s2 x4 x4 x5 x5 s3s3 x1 x1 x2 x2 x3 x3 z2 z2 z1z1 z3z3 x4 x4 z4 z4 x5 x5 z5z5 s1s1 s2s2 s3s3 strided convolution downsampling convolution figure 9. 12 : convolution with a stride. in this example, we use a stride of two. ( top ) convolution with a stride length of two implemented in a single operation. ( bot - tom ) convolution with a stride greater than one pixel is mathematically equivalent to convolution with unit stride followed by downsampling. obviously, the two - step approach involving downsampling is computationally wasteful, because it computes many values that are then discarded. 350


chapter 9. convolutional networks........................... figure 9. 13 : the [UNK] of zero padding on network size : consider a convolutional network with a kernel of width six at every layer. in this example, we do not use any pooling, so only the convolution operation itself shrinks the network size. ( top ) in this convolutional network, we do not use any implicit zero padding. this causes the representation to shrink by ﬁve pixels at each layer. starting from an input of sixteen pixels, we are only able to have three convolutional layers, and the last layer does not ever move the kernel, so arguably only two of the layers are truly convolutional. the rate of shrinking can be mitigated by using smaller kernels, but smaller kernels are less expressive and some shrinking is inevitable in this kind of architecture. by adding ﬁve implicit zeroes ( bottom ) to each layer, we prevent the representation from shrinking with depth. this allows us to make an arbitrarily deep convolutional network. 351


chapter 9. convolutional networks in some cases, we do not actually want to use convolution, but rather locally connected layers (,, ). in this case, the adjacency matrix in the lecun 1986 1989 graph of our mlp is the same, but every connection has its own weight, speciﬁed by a 6 - d tensor w. the indices into w are respectively : i, the output channel, j, the output row, k, the output column, l, the input channel, m, the row [UNK] within the input, and n, the column [UNK] within the input. the linear part of a locally connected layer is then given by zi, j, k = l, m, n [ vl, j m, k n + −1 + −1wi, j, k, l, m, n ]. ( 9. 9 ) this is sometimes also called unshared convolution, because it is a similar oper - ation to discrete convolution with a small kernel, but without sharing parameters across locations. figure compares local connections, convolution, and full 9. 14 connections. locally connected layers are useful when we know that each


oper - ation to discrete convolution with a small kernel, but without sharing parameters across locations. figure compares local connections, convolution, and full 9. 14 connections. locally connected layers are useful when we know that each feature should be a function of a small part of space, but there is no reason to think that the same feature should occur across all of space. for example, if we want to tell if an image is a picture of a face, we only need to look for the mouth in the bottom half of the image. it can also be useful to make versions of convolution or locally connected layers in which the connectivity is further restricted, for example to constrain each output channel i to be a function of only a subset of the input channels l. a common way to do this is to make the ﬁrst m output channels connect to only the ﬁrst n input channels, the second m output channels connect to only the second n input channels, and so on. see ﬁgure for an example. modeling interactions 9. 15 between few channels allows the network to have fewer parameters in order to reduce memory consumption and increase statistical [UNK], and also reduces the amount of computation needed to perform forward and back - propagation.


see ﬁgure for an example. modeling interactions 9. 15 between few channels allows the network to have fewer parameters in order to reduce memory consumption and increase statistical [UNK], and also reduces the amount of computation needed to perform forward and back - propagation. it accomplishes these goals without reducing the number of hidden units. tiled convolution (, ;, ) [UNK] a com - gregor and lecun 2010a le et al. 2010 promise between a convolutional layer and a locally connected layer. rather than learning a separate set of weights at spatial location, we learn a set of kernels every that we rotate through as we move through space. this means that immediately neighboring locations will have [UNK] ﬁlters, like in a locally connected layer, but the memory requirements for storing the parameters will increase only by a factor of the size of this set of kernels, rather than the size of the entire output feature map. see ﬁgure for a comparison of locally connected layers, tiled 9. 16 convolution, and standard convolution. 352


chapter 9. convolutional networks x1 x1 x2 x2 x3 x3 s2s2 s1s1 s3 s3 x4 x4 s4s4 x5 x5 s5s5 x1 x1 x2 x2 s1s1 s3 s3 x5 x5 s5s5 x1 x1 x2 x2 x3 x3 s2s2 s1s1 s3 s3 x4 x4 s4s4 x5 x5 s5s5 a b a b a b a b a a b c d e f g h i x4 x4 x3 x3 s4s4 s2s2 figure 9. 14 : comparison of local connections, convolution, and full connections. ( top ) a locally connected layer with a patch size of two pixels. each edge is labeled with a unique letter to show that each edge is associated with its own weight parameter. ( center ) a convolutional layer with a kernel width of two pixels. this model has exactly the same connectivity as the locally connected layer. the [UNK] lies not in which units interact with each other, but in how the parameters are shared. the locally connected


convolutional layer with a kernel width of two pixels. this model has exactly the same connectivity as the locally connected layer. the [UNK] lies not in which units interact with each other, but in how the parameters are shared. the locally connected layer has no parameter sharing. the convolutional layer uses the same two weights repeatedly across the entire input, as indicated by the repetition of the letters labeling each edge. ( bottom ) a fully connected layer resembles a locally connected layer in the sense that each edge has its own parameter ( there are too many to label explicitly with letters in this diagram ). however, it does not have the restricted connectivity of the locally connected layer. 353


chapter 9. convolutional networks input tensor output tensor spatial coordinates channel coordinates figure 9. 15 : a convolutional network with the ﬁrst two output channels connected to only the ﬁrst two input channels, and the second two output channels connected to only the second two input channels. 354


chapter 9. convolutional networks x1 x1 x2 x2 x3 x3 s2s2 s1s1 s3 s3 x4 x4 s4s4 x5 x5 s5s5 x1 x1 x2 x2 x3 x3 s2s2 s1s1 s3 s3 x4 x4 s4s4 x5 x5 s5s5 a b a b a b a b a a b c d e f g h i x1 x1 x2 x2 x3 x3 s2s2 s1s1 s3 s3 x4 x4 s4s4 x5 x5 s5s5 a b c d a b c d a figure 9. 16 : a comparison of locally connected layers, tiled convolution, and standard convolution. all three have the same sets of connections between units, when the same size of kernel is used. this diagram illustrates the use of a kernel that is two pixels wide. the [UNK] between the methods lies in how they share parameters. ( top ) a locally connected layer has no sharing at all. we indicate that each connection has its own weight by labeling each connection with a unique


of a kernel that is two pixels wide. the [UNK] between the methods lies in how they share parameters. ( top ) a locally connected layer has no sharing at all. we indicate that each connection has its own weight by labeling each connection with a unique letter. tiled convolution has a set of ( center ) t [UNK] kernels. here we illustrate the case of t = 2. one of these kernels has edges labeled “ a ” and “ b, ” while the other has edges labeled “ c ” and “ d. ” each time we move one pixel to the right in the output, we move on to using a [UNK] kernel. this means that, like the locally connected layer, neighboring units in the output have [UNK] parameters. unlike the locally connected layer, after we have gone through allt available kernels, we cycle back to the ﬁrst kernel. if two output units are separated by a multiple of t steps, then they share parameters. traditional convolution is equivalent to tiled ( bottom ) convolution with t = 1. there is only one kernel and it is applied everywhere, as indicated in the diagram by using the kernel with weights labeled “ a ” and “ b ” everywhere. 355


chapter 9. convolutional networks to deﬁne tiled convolution algebraically, let k be a 6 - d tensor, where two of the dimensions correspond to [UNK] locations in the output map. rather than having a separate index for each location in the output map, output locations cycle through a set of t [UNK] choices of kernel stack in each direction. if t is equal to the output width, this is the same as a locally connected layer. zi, j, k = l, m, n vl, j m, k n + −1 + −1ki, l, m, n, j t, k t % + 1 % + 1, ( 9. 10 ) where is the modulo operation, with % t % t = 0 (, t + 1 ) % t = 1, etc. it is straightforward to generalize this equation to use a [UNK] tiling range for each dimension. both locally connected layers and tiled convolutional layers have an interesting interaction with max - pooling : the detector units of these layers are driven by [UNK] ﬁlters. if these ﬁlters learn to detect [UNK] transformed versions of the same underlying features, then the max - pooled units become invariant to the learned


interaction with max - pooling : the detector units of these layers are driven by [UNK] ﬁlters. if these ﬁlters learn to detect [UNK] transformed versions of the same underlying features, then the max - pooled units become invariant to the learned transformation ( see ﬁgure ). convolutional layers are hard - coded to be 9. 9 invariant speciﬁcally to translation. other operations besides convolution are usually necessary to implement a convolutional network. to perform learning, one must be able to compute the gradient with respect to the kernel, given the gradient with respect to the outputs. in some simple cases, this operation can be performed using the convolution operation, but many cases of interest, including the case of stride greater than 1, do not have this property. recall that convolution is a linear operation and can thus be described as a matrix multiplication ( if we ﬁrst reshape the input tensor into a ﬂat vector ). the matrix involved is a function of the convolution kernel. the matrix is sparse and each element of the kernel is copied to several elements of the matrix. this view helps us to derive some of the other operations needed to implement a convolu


matrix involved is a function of the convolution kernel. the matrix is sparse and each element of the kernel is copied to several elements of the matrix. this view helps us to derive some of the other operations needed to implement a convolutional network. multiplication by the transpose of the matrix deﬁned by convolution is one such operation. this is the operation needed to back - propagate error derivatives through a convolutional layer, so it is needed to train convolutional networks that have more than one hidden layer. this same operation is also needed if we wish to reconstruct the visible units from the hidden units (, ). simard et al. 1992 reconstructing the visible units is an operation commonly used in the models described in part of this book, such as autoencoders, rbms, and sparse coding. iii transpose convolution is necessary to construct convolutional versions of those models. like the kernel gradient operation, this input gradient operation can be 356


chapter 9. convolutional networks implemented using a convolution in some cases, but in the general case requires a third operation to be implemented. care must be taken to coordinate this transpose operation with the forward propagation. the size of the output that the transpose operation should return depends on the zero padding policy and stride of the forward propagation operation, as well as the size of the forward propagation ’ s output map. in some cases, multiple sizes of input to forward propagation can result in the same size of output map, so the transpose operation must be explicitly told what the size of the original input was. these three operations — convolution, backprop from output to weights, and backprop from output to inputs — are [UNK] to compute all of the gradients needed to train any depth of feedforward convolutional network, as well as to train convolutional networks with reconstruction functions based on the transpose of convolution. see ( ) for a full derivation of the equations in the goodfellow 2010 fully general multi - dimensional, multi - example case. to give a sense of how these equations work, we present the two dimensional, single example version here. suppose we want to


) for a full derivation of the equations in the goodfellow 2010 fully general multi - dimensional, multi - example case. to give a sense of how these equations work, we present the two dimensional, single example version here. suppose we want to train a convolutional network that incorporates strided convolution of kernel stack k applied to multi - channel image v with stride s as deﬁned by c ( k v,, s ) as in equation. suppose we want to minimize some loss 9. 8 function j ( v k, ). during forward propagation, we will need to use c itself to output z, which is then propagated through the rest of the network and used to compute the cost function j. during back - propagation, we will receive a tensor g such that gi, j, k = ∂ ∂zi, j, k j,. ( v k ) to train the network, we need to compute the derivatives with respect to the weights in the kernel. to do so, we can use a function g,, s ( g v ) i, j, k, l = ∂ ∂ki, j, k, l j, ( v k ) = m, n gi, m, nvj,


so, we can use a function g,, s ( g v ) i, j, k, l = ∂ ∂ki, j, k, l j, ( v k ) = m, n gi, m, nvj, m s k, n s l ( − × 1 ) + ( − × 1 ) +. ( 9. 11 ) if this layer is not the bottom layer of the network, we will need to compute the gradient with respect to v in order to back - propagate the error farther down. to do so, we can use a function h,, s ( k g ) i, j, k = ∂ ∂vi, j, k j, ( v k ) ( 9. 12 ) = l, m s. t. ( 1 ) + = l− ×s m j n, p s. t. ( 1 ) + = n− ×s p k q kq, i, m, pgq, l, n. ( 9. 13 ) autoencoder networks, described in chapter, are feedforward networks 14 trained to copy their input to their output. a simple example is the pca algorithm, 357


chapter 9. convolutional networks that copies its input x to an approximate reconstruction r using the function w wx. it is common for more general autoencoders to use multiplication by the transpose of the weight matrix just as pca does. to make such models convolutional, we can use the function h to perform the transpose of the convolution operation. suppose we have hidden units h in the same format as z and we deﬁne a reconstruction r k h = ( h,, s. ) ( 9. 14 ) in order to train the autoencoder, we will receive the gradient with respect to r as a tensor e. to train the decoder, we need to obtain the gradient with respect to k. this is given by g ( h e,, s ). to train the encoder, we need to obtain the gradient with respect to h. this is given by c ( k e,, s ). it is also possible to [UNK] through g using c and h, but these operations are not needed for the back - propagation algorithm on any standard network architectures. generally, we do not use only a linear operation in order to transform from the inputs to the outputs in a


to [UNK] through g using c and h, but these operations are not needed for the back - propagation algorithm on any standard network architectures. generally, we do not use only a linear operation in order to transform from the inputs to the outputs in a convolutional layer. we generally also add some bias term to each output before applying the nonlinearity. this raises the question of how to share parameters among the biases. for locally connected layers it is natural to give each unit its own bias, and for tiled convolution, it is natural to share the biases with the same tiling pattern as the kernels. for convolutional layers, it is typical to have one bias per channel of the output and share it across all locations within each convolution map. however, if the input is of known, ﬁxed size, it is also possible to learn a separate bias at each location of the output map. separating the biases may slightly reduce the statistical [UNK] of the model, but also allows the model to correct for [UNK] in the image statistics at [UNK] locations. for example, when using implicit zero padding, detector units at the edge of the image receive less total input and may need larger biases. 9. 6


, but also allows the model to correct for [UNK] in the image statistics at [UNK] locations. for example, when using implicit zero padding, detector units at the edge of the image receive less total input and may need larger biases. 9. 6 structured outputs convolutional networks can be used to output a high - dimensional, structured object, rather than just predicting a class label for a classiﬁcation task or a real value for a regression task. typically this object is just a tensor, emitted by a standard convolutional layer. for example, the model might emit a tensor s, where si, j, k is the probability that pixel ( j, k ) of the input to the network belongs to class i. this allows the model to label every pixel in an image and draw precise masks that follow the outlines of individual objects. one issue that often comes up is that the output plane can be smaller than the 358


chapter 9. convolutional networks [UNK] ( 1 ) [UNK] ( 1 ) [UNK] ( 2 ) [UNK] ( 2 ) [UNK] ( 3 ) [UNK] ( 3 ) h ( 1 ) h ( 1 ) h ( 2 ) h ( 2 ) h ( 3 ) h ( 3 ) x u u u v v v w w figure 9. 17 : an example of a recurrent convolutional network for pixel labeling. the input is an image tensor, with axes corresponding to image rows, image columns, and x channels ( red, green, blue ). the goal is to output a tensor of labels [UNK], with a probability distribution over labels for each pixel. this tensor has axes corresponding to image rows, image columns, and the [UNK] classes. rather than outputting [UNK] in a single shot, the recurrent network iteratively reﬁnes its estimate [UNK] by using a previous estimate of [UNK] as input for creating a new estimate. the same parameters are used for each updated estimate, and the estimate can be reﬁned as many times as we wish. the tensor of convolution kernels u is used on each step to compute the hidden representation given the input image. the kernel tensor v is used to produce an estimate of the labels given the


reﬁned as many times as we wish. the tensor of convolution kernels u is used on each step to compute the hidden representation given the input image. the kernel tensor v is used to produce an estimate of the labels given the hidden values. on all but the ﬁrst step, the kernels w are convolved over [UNK] to provide input to the hidden layer. on the ﬁrst time step, this term is replaced by zero. because the same parameters are used on each step, this is an example of a recurrent network, as described in chapter. 10 input plane, as shown in ﬁgure. in the kinds of architectures typically used for 9. 13 classiﬁcation of a single object in an image, the greatest reduction in the spatial dimensions of the network comes from using pooling layers with large stride. in order to produce an output map of similar size as the input, one can avoid pooling altogether (, ). another strategy is to simply emit a lower - resolution jain et al. 2007 grid of labels (,, ). finally, in principle, one could pinheiro and collobert 2014 2015 use a pooling operator with unit stride. one strategy for pixel - wise labeling of images is


- resolution jain et al. 2007 grid of labels (,, ). finally, in principle, one could pinheiro and collobert 2014 2015 use a pooling operator with unit stride. one strategy for pixel - wise labeling of images is to produce an initial guess of the image labels, then reﬁne this initial guess using the interactions between neighboring pixels. repeating this reﬁnement step several times corresponds to using the same convolutions at each stage, sharing weights between the last layers of the deep net (, ). this makes the sequence of computations performed jain et al. 2007 by the successive convolutional layers with weights shared across layers a particular kind of recurrent network (,, ). figure shows pinheiro and collobert 2014 2015 9. 17 the architecture of such a recurrent convolutional network. 359


chapter 9. convolutional networks once a prediction for each pixel is made, various methods can be used to further process these predictions in order to obtain a segmentation of the image into regions (, ; briggman et al. 2009 turaga 2010 farabet 2013 et al., ; et al., ). the general idea is to assume that large groups of contiguous pixels tend to be associated with the same label. graphical models can describe the probabilistic relationships between neighboring pixels. alternatively, the convolutional network can be trained to maximize an approximation of the graphical model training objective (, ;, ). ning et al. 2005 thompson et al. 2014 9. 7 data types the data used with a convolutional network usually consists of several channels, each channel being the observation of a [UNK] quantity at some point in space or time. see table for examples of data types with [UNK] dimensionalities 9. 1 and number of channels. for an example of convolutional networks applied to video, see chen et al. ( ). 2010 so far we have discussed only the case where every example in the train and test data has the same spatial dimensions. one advantage to convolutional networks is


##lutional networks applied to video, see chen et al. ( ). 2010 so far we have discussed only the case where every example in the train and test data has the same spatial dimensions. one advantage to convolutional networks is that they can also process inputs with varying spatial extents. these kinds of input simply cannot be represented by traditional, matrix multiplication - based neural networks. this provides a compelling reason to use convolutional networks even when computational cost and overﬁtting are not signiﬁcant issues. for example, consider a collection of images, where each image has a [UNK] width and height. it is unclear how to model such inputs with a weight matrix of ﬁxed size. convolution is straightforward to apply ; the kernel is simply applied a [UNK] number of times depending on the size of the input, and the output of the convolution operation scales accordingly. convolution may be viewed as matrix multiplication ; the same convolution kernel induces a [UNK] size of doubly block circulant matrix for each size of input. sometimes the output of the network is allowed to have variable size as well as the input, for example if we want to assign a class label to each pixel


a [UNK] size of doubly block circulant matrix for each size of input. sometimes the output of the network is allowed to have variable size as well as the input, for example if we want to assign a class label to each pixel of the input. in this case, no further design work is necessary. in other cases, the network must produce some ﬁxed - size output, for example if we want to assign a single class label to the entire image. in this case we must make some additional design steps, like inserting a pooling layer whose pooling regions scale in size proportional to the size of the input, in order to maintain a ﬁxed number of pooled outputs. some examples of this kind of strategy are shown in ﬁgure. 9. 11 360


chapter 9. convolutional networks single channel multi - channel 1 - d audio waveform : the axis we convolve over corresponds to time. we discretize time and measure the amplitude of the waveform once per time step. skeleton animation data : anima - tions of 3 - d computer - rendered characters are generated by alter - ing the pose of a “ skeleton ” over time. at each point in time, the pose of the character is described by a speciﬁcation of the angles of each of the joints in the charac - ter ’ s skeleton. each channel in the data we feed to the convolu - tional model represents the angle about one axis of one joint. 2 - d audio data that has been prepro - cessed with a fourier transform : we can transform the audio wave - form into a 2d tensor with dif - ferent rows corresponding to dif - ferent frequencies and [UNK] columns corresponding to [UNK] - ent points in time. using convolu - tion in the time makes the model equivariant to shifts in time. us - ing convolution across the fre - quency axis makes the model equivariant to frequency, so that


convolu - tion in the time makes the model equivariant to shifts in time. us - ing convolution across the fre - quency axis makes the model equivariant to frequency, so that the same melody played in a dif - ferent octave produces the same representation but at a [UNK] height in the network ’ s output. color image data : one channel contains the red pixels, one the green pixels, and one the blue pixels. the convolution kernel moves over both the horizontal and vertical axes of the image, conferring translation equivari - ance in both directions. 3 - d volumetric data : a common source of this kind of data is med - ical imaging technology, such as ct scans. color video data : one axis corre - sponds to time, one to the height of the video frame, and one to the width of the video frame. table 9. 1 : examples of [UNK] formats of data that can be used with convolutional networks. 361


chapter 9. convolutional networks note that the use of convolution for processing variable sized inputs only makes sense for inputs that have variable size because they contain varying amounts of observation of the same kind of thing — [UNK] lengths of recordings over time, [UNK] widths of observations over space, etc. convolution does not make sense if the input has variable size because it can optionally include [UNK] kinds of observations. for example, if we are processing college applications, and our features consist of both grades and standardized test scores, but not every applicant took the standardized test, then it does not make sense to convolve the same weights over both the features corresponding to the grades and the features corresponding to the test scores. 9. 8 [UNK] convolution algorithms modern convolutional network applications often involve networks containing more than one million units. powerful implementations exploiting parallel computation resources, as discussed in section, are essential. however, in many cases it 12. 1 is also possible to speed up convolution by selecting an appropriate convolution algorithm. convolution is equivalent to converting both the input and the kernel to the frequency domain using a fourier transform, performing point - wise multiplication of the two signals, and


speed up convolution by selecting an appropriate convolution algorithm. convolution is equivalent to converting both the input and the kernel to the frequency domain using a fourier transform, performing point - wise multiplication of the two signals, and converting back to the time domain using an inverse fourier transform. for some problem sizes, this can be faster than the naive implementation of discrete convolution. when a d - dimensional kernel can be expressed as the outer product of d vectors, one vector per dimension, the kernel is called separable. when the kernel is separable, naive convolution is [UNK]. it is equivalent to compose d one - dimensional convolutions with each of these vectors. the composed approach is signiﬁcantly faster than performing one d - dimensional convolution with their outer product. the kernel also takes fewer parameters to represent as vectors. if the kernel is w elements wide in each dimension, then naive multidimensional convolution requires o ( wd ) runtime and parameter storage space, while separable convolution requires o ( w d × ) runtime and parameter storage space. of course, not every convolution can be represented in this way. devising faster ways


##d ) runtime and parameter storage space, while separable convolution requires o ( w d × ) runtime and parameter storage space. of course, not every convolution can be represented in this way. devising faster ways of performing convolution or approximate convolution without harming the accuracy of the model is an active area of research. even tech - niques that improve the [UNK] of only forward propagation are useful because in the commercial setting, it is typical to devote more resources to deployment of a network than to its training. 362


chapter 9. convolutional networks 9. 9 random or unsupervised features typically, the most expensive part of convolutional network training is learning the features. the output layer is usually relatively inexpensive due to the small number of features provided as input to this layer after passing through several layers of pooling. when performing supervised training with gradient descent, every gradient step requires a complete run of forward propagation and backward propagation through the entire network. one way to reduce the cost of convolutional network training is to use features that are not trained in a supervised fashion. there are three basic strategies for obtaining convolution kernels without supervised training. one is to simply initialize them randomly. another is to design them by hand, for example by setting each kernel to detect edges at a certain orientation or scale. finally, one can learn the kernels with an unsupervised criterion. for example, ( ) apply coates et al. 2011 k - means clustering to small image patches, then use each learned centroid as a convolution kernel. part iii describes many more unsupervised learning approaches. learning the features with an unsupervised criterion allows them to be determined separately from the


small image patches, then use each learned centroid as a convolution kernel. part iii describes many more unsupervised learning approaches. learning the features with an unsupervised criterion allows them to be determined separately from the classiﬁer layer at the top of the architecture. one can then extract the features for the entire training set just once, essentially constructing a new training set for the last layer. learning the last layer is then typically a convex optimization problem, assuming the last layer is something like logistic regression or an svm. random ﬁlters often work surprisingly well in convolutional networks ( jarrett et al. et al. et al., ; 2009 saxe, ; 2011 pinto, ; 2011 cox and pinto 2011 saxe, ). et al. ( ) showed that layers consisting of convolution following by pooling naturally 2011 become frequency selective and translation invariant when assigned random weights. they argue that this provides an inexpensive way to choose the architecture of a convolutional network : ﬁrst evaluate the performance of several convolutional network architectures by training only the last layer, then take the best of these architectures and train the entire architecture using a more expensive approach. an intermediate approach is


##lutional network : ﬁrst evaluate the performance of several convolutional network architectures by training only the last layer, then take the best of these architectures and train the entire architecture using a more expensive approach. an intermediate approach is to learn the features, but using methods that do not require full forward and back - propagation at every gradient step. as with multilayer perceptrons, we use greedy layer - wise pretraining, to train the ﬁrst layer in isolation, then extract all features from the ﬁrst layer only once, then train the second layer in isolation given those features, and so on. chapter has described 8 how to perform supervised greedy layer - wise pretraining, and part extends this iii to greedy layer - wise pretraining using an unsupervised criterion at each layer. the canonical example of greedy layer - wise pretraining of a convolutional model is the convolutional deep belief network (, ). convolutional networks [UNK] lee et al. 2009 363


chapter 9. convolutional networks us the opportunity to take the pretraining strategy one step further than is possible with multilayer perceptrons. instead of training an entire convolutional layer at a time, we can train a model of a small patch, as ( ) do with coates et al. 2011 k - means. we can then use the parameters from this patch - based model to deﬁne the kernels of a convolutional layer. this means that it is possible to use unsupervised learning to train a convolutional network without ever using convolution during the training process. using this approach, we can train very large models and incur a high computational cost only at inference time (, ;, ranzato et al. 2007b jarrett et al. 2009 kavukcuoglu 2010 coates 2013 ; et al., ; et al., ). this approach was popular from roughly 2007 – 2013, when labeled datasets were small and computational power was more limited. today, most convolutional networks are trained in a purely supervised fashion, using full forward and back - propagation through the entire network on each training iteration. as with other


when labeled datasets were small and computational power was more limited. today, most convolutional networks are trained in a purely supervised fashion, using full forward and back - propagation through the entire network on each training iteration. as with other approaches to unsupervised pretraining, it remains [UNK] to tease apart the cause of some of the beneﬁts seen with this approach. unsupervised pretraining may [UNK] some regularization relative to supervised training, or it may simply allow us to train much larger architectures due to the reduced computational cost of the learning rule. 9. 10 the neuroscientiﬁc basis for convolutional net - works convolutional networks are perhaps the greatest success story of biologically inspired artiﬁcial intelligence. though convolutional networks have been guided by many other ﬁelds, some of the key design principles of neural networks were drawn from neuroscience. the history of convolutional networks begins with neuroscientiﬁc experiments long before the relevant computational models were developed. neurophysiologists david hubel and torsten wiesel collaborated for several years to determine many of the most basic facts about how the mammalian vision


begins with neuroscientiﬁc experiments long before the relevant computational models were developed. neurophysiologists david hubel and torsten wiesel collaborated for several years to determine many of the most basic facts about how the mammalian vision system works ( hubel and wiesel 1959 1962 1968,,, ). their accomplishments were eventually recognized with a nobel prize. their ﬁndings that have had the greatest inﬂuence on contemporary deep learning models were based on recording the activity of individual neurons in cats. they observed how neurons in the cat ’ s brain responded to images projected in precise locations on a screen in front of the cat. their great discovery was that neurons in the early visual system responded most strongly to very speciﬁc patterns of light, such as precisely oriented bars, but responded hardly at all to other patterns. 364


chapter 9. convolutional networks their work helped to characterize many aspects of brain function that are beyond the scope of this book. from the point of view of deep learning, we can focus on a simpliﬁed, cartoon view of brain function. in this simpliﬁed view, we focus on a part of the brain called v1, also known as the primary visual cortex. v1 is the ﬁrst area of the brain that begins to perform signiﬁcantly advanced processing of visual input. in this cartoon view, images are formed by light arriving in the eye and stimulating the retina, the light - sensitive tissue in the back of the eye. the neurons in the retina perform some simple preprocessing of the image but do not substantially alter the way it is represented. the image then passes through the optic nerve and a brain region called the lateral geniculate nucleus. the main role, as far as we are concerned here, of both of these anatomical regions is primarily just to carry the signal from the eye to v1, which is located at the back of the head. a convolutional network layer is designed to capture three properties of v1 : 1. v1 is arranged in a spatial


primarily just to carry the signal from the eye to v1, which is located at the back of the head. a convolutional network layer is designed to capture three properties of v1 : 1. v1 is arranged in a spatial map. it actually has a two - dimensional structure mirroring the structure of the image in the retina. for example, light arriving at the lower half of the retina [UNK] only the corresponding half of v1. convolutional networks capture this property by having their features deﬁned in terms of two dimensional maps. 2. v1 contains many simple cells. a simple cell ’ s activity can to some extent be characterized by a linear function of the image in a small, spatially localized receptive ﬁeld. the detector units of a convolutional network are designed to emulate these properties of simple cells. 3. v1 also contains many complex cells. these cells respond to features that are similar to those detected by simple cells, but complex cells are invariant to small shifts in the position of the feature. this inspires the pooling units of convolutional networks. complex cells are also invariant to some changes in lighting that cannot be captured simply by pooling over spatial locations.


cells are invariant to small shifts in the position of the feature. this inspires the pooling units of convolutional networks. complex cells are also invariant to some changes in lighting that cannot be captured simply by pooling over spatial locations. these invariances have inspired some of the cross - channel pooling strategies in convolutional networks, such as maxout units (, ). goodfellow et al. 2013a though we know the most about v1, it is generally believed that the same basic principles apply to other areas of the visual system. in our cartoon view of the visual system, the basic strategy of detection followed by pooling is repeatedly applied as we move deeper into the brain. as we pass through multiple anatomical layers of the brain, we eventually ﬁnd cells that respond to some speciﬁc concept and are invariant to many transformations of the input. these cells have been 365


chapter 9. convolutional networks nicknamed “ grandmother cells ” — the idea is that a person could have a neuron that activates when seeing an image of their grandmother, regardless of whether she appears in the left or right side of the image, whether the image is a close - up of her face or zoomed out shot of her entire body, whether she is brightly lit, or in shadow, etc. these grandmother cells have been shown to actually exist in the human brain, in a region called the medial temporal lobe (, ). researchers quiroga et al. 2005 tested whether individual neurons would respond to photos of famous individuals. they found what has come to be called the “ halle berry neuron ” : an individual neuron that is activated by the concept of halle berry. this neuron ﬁres when a person sees a photo of halle berry, a drawing of halle berry, or even text containing the words “ halle berry. ” of course, this has nothing to do with halle berry herself ; other neurons responded to the presence of bill clinton, jennifer aniston, etc. these medial temporal lobe neurons are somewhat more general than modern convolutional networks, which would not automatically generalize to identifying a person or object when reading its


other neurons responded to the presence of bill clinton, jennifer aniston, etc. these medial temporal lobe neurons are somewhat more general than modern convolutional networks, which would not automatically generalize to identifying a person or object when reading its name. the closest analog to a convolutional network ’ s last layer of features is a brain area called the inferotemporal cortex ( it ). when viewing an object, information ﬂows from the retina, through the lgn, to v1, then onward to v2, then v4, then it. this happens within the ﬁrst 100ms of glimpsing an object. if a person is allowed to continue looking at the object for more time, then information will begin to ﬂow backwards as the brain uses top - down feedback to update the activations in the lower level brain areas. however, if we interrupt the person ’ s gaze, and observe only the ﬁring rates that result from the ﬁrst 100ms of mostly feedforward activation, then it proves to be very similar to a convolutional network. convolutional networks can predict it ﬁring rates, and also perform very similarly to ( time limited ) humans on object recognition


of mostly feedforward activation, then it proves to be very similar to a convolutional network. convolutional networks can predict it ﬁring rates, and also perform very similarly to ( time limited ) humans on object recognition tasks (, ). dicarlo 2013 that being said, there are many [UNK] between convolutional networks and the mammalian vision system. some of these [UNK] are well known to computational neuroscientists, but outside the scope of this book. some of these [UNK] are not yet known, because many basic questions about how the mammalian vision system works remain unanswered. as a brief list : • the human eye is mostly very low resolution, except for a tiny patch called the fovea. the fovea only observes an area about the size of a thumbnail held at arms length. though we feel as if we can see an entire scene in high resolution, this is an illusion created by the subconscious part of our brain, as it stitches together several glimpses of small areas. most convolutional networks actually receive large full resolution photographs as input. the human brain makes 366


chapter 9. convolutional networks several eye movements called saccades to glimpse the most visually salient or task - relevant parts of a scene. incorporating similar attention mechanisms into deep learning models is an active research direction. in the context of deep learning, attention mechanisms have been most successful for natural language processing, as described in section. several visual models 12. 4. 5. 1 with foveation mechanisms have been developed but so far have not become the dominant approach ( larochelle and hinton 2010 denil 2012, ; et al., ). • the human visual system is integrated with many other senses, such as hearing, and factors like our moods and thoughts. convolutional networks so far are purely visual. • the human visual system does much more than just recognize objects. it is able to understand entire scenes including many objects and relationships between objects, and processes rich 3 - d geometric information needed for our bodies to interface with the world. convolutional networks have been applied to some of these problems but these applications are in their infancy. • even simple brain areas like v1 are heavily impacted by feedback from higher levels. feedback has been explored extensively in neural network models but has not yet been shown to [UNK] a compelling


applied to some of these problems but these applications are in their infancy. • even simple brain areas like v1 are heavily impacted by feedback from higher levels. feedback has been explored extensively in neural network models but has not yet been shown to [UNK] a compelling improvement. • while feedforward it ﬁring rates capture much of the same information as convolutional network features, it is not clear how similar the intermediate computations are. the brain probably uses very [UNK] activation and pooling functions. an individual neuron ’ s activation probably is not well - characterized by a single linear ﬁlter response. a recent model of v1 involves multiple quadratic ﬁlters for each neuron (, ). indeed our rust et al. 2005 cartoon picture of “ simple cells ” and “ complex cells ” might create a non - existent distinction ; simple cells and complex cells might both be the same kind of cell but with their “ parameters ” enabling a continuum of behaviors ranging from what we call “ simple ” to what we call “ complex. ” it is also worth mentioning that neuroscience has told us relatively little about how to train convolutional networks. model structures with parameter sharing across multiple spatial locations date back to early connectionist models of vision (, ), but these


complex. ” it is also worth mentioning that neuroscience has told us relatively little about how to train convolutional networks. model structures with parameter sharing across multiple spatial locations date back to early connectionist models of vision (, ), but these models did not use the modern marr and poggio 1976 back - propagation algorithm and gradient descent. for example, the neocognitron ( fukushima 1980, ) incorporated most of the model architecture design elements of the modern convolutional network but relied on a layer - wise unsupervised clustering algorithm. 367


chapter 9. convolutional networks lang and hinton 1988 ( ) introduced the use of back - propagation to train time - delay neural networks ( tdnns ). to use contemporary terminology, tdnns are one - dimensional convolutional networks applied to time series. back - propagation applied to these models was not inspired by any neuroscientiﬁc observa - tion and is considered by some to be biologically implausible. following the success of back - propagation - based training of tdnns, (, ) developed lecun et al. 1989 the modern convolutional network by applying the same training algorithm to 2 - d convolution applied to images. so far we have described how simple cells are roughly linear and selective for certain features, complex cells are more nonlinear and become invariant to some transformations of these simple cell features, and stacks of layers that alternate between selectivity and invariance can yield grandmother cells for very speciﬁc phenomena. we have not yet described precisely what these individual cells detect. in a deep, nonlinear network, it can be [UNK] to understand the function of individual cells. simple cells in the ﬁrst layer are easier to analyze, because their responses are driven


phenomena. we have not yet described precisely what these individual cells detect. in a deep, nonlinear network, it can be [UNK] to understand the function of individual cells. simple cells in the ﬁrst layer are easier to analyze, because their responses are driven by a linear function. in an artiﬁcial neural network, we can just display an image of the convolution kernel to see what the corresponding channel of a convolutional layer responds to. in a biological neural network, we do not have access to the weights themselves. instead, we put an electrode in the neuron itself, display several samples of white noise images in front of the animal ’ s retina, and record how each of these samples causes the neuron to activate. we can then ﬁt a linear model to these responses in order to obtain an approximation of the neuron ’ s weights. this approach is known as reverse correlation ( ringach and shapley 2004, ). reverse correlation shows us that most v1 cells have weights that are described by gabor functions. the gabor function describes the weight at a 2 - d point in the image. we can think of an image as being a function of 2 - d coordinates, i ( x, y ). likewise


weights that are described by gabor functions. the gabor function describes the weight at a 2 - d point in the image. we can think of an image as being a function of 2 - d coordinates, i ( x, y ). likewise, we can think of a simple cell as sampling the image at a set of locations, deﬁned by a set of x coordinates x and a set of y coordinates, y, and applying weights that are also a function of the location, w ( x, y ). from this point of view, the response of a simple cell to an image is given by s i ( ) = x∈x y∈y w x, y i x, y. ( ) ( ) ( 9. 15 ) speciﬁcally, takes the form of a gabor function : w x, y ( ) w x, y α, β ( ; x, βy, f, φ, x0, y0, τ α ) = exp −βxx2 −βyy2 cos ( fx + ) φ, ( 9. 16 ) where x = ( x x − 0 ) cos ( ) + ( τ y y − 0 ) sin ( ) τ ( 9. 17 )


##xx2 −βyy2 cos ( fx + ) φ, ( 9. 16 ) where x = ( x x − 0 ) cos ( ) + ( τ y y − 0 ) sin ( ) τ ( 9. 17 ) 368


chapter 9. convolutional networks and y = ( −x x − 0 ) sin ( ) + ( τ y y − 0 ) cos ( ) τ. ( 9. 18 ) here, α, βx, βy, f, φ, x0, y0, and τ are parameters that control the properties of the gabor function. figure shows some examples of gabor functions with 9. 18 [UNK] settings of these parameters. the parameters x0, y0, and τ deﬁne a coordinate system. we translate and rotate x and y to form xand y. speciﬁcally, the simple cell will respond to image features centered at the point ( x0, y 0 ), and it will respond to changes in brightness as we move along a line rotated radians from the horizontal. τ viewed as a function of xand y, the function w then responds to changes in brightness as we move along the xaxis. it has two important factors : one is a gaussian function and the other is a cosine function. the gaussian factor α exp −βxx2 −βyy2 can be seen as a gating term that ensures the simple cell will only respond to


is a gaussian function and the other is a cosine function. the gaussian factor α exp −βxx2 −βyy2 can be seen as a gating term that ensures the simple cell will only respond to values near where x and yare both zero, in other words, near the center of the cell ’ s receptive ﬁeld. the scaling factor α adjusts the total magnitude of the simple cell ’ s response, while βx and β y control how quickly its receptive ﬁeld falls [UNK]. the cosine factor cos ( fx + φ ) controls how the simple cell responds to changing brightness along the xaxis. the parameter f controls the frequency of the cosine and controls its phase [UNK]. φ altogether, this cartoon view of simple cells means that a simple cell responds to a speciﬁc spatial frequency of brightness in a speciﬁc direction at a speciﬁc location. simple cells are most excited when the wave of brightness in the image has the same phase as the weights. this occurs when the image is bright where the weights are positive and dark where the weights are negative. simple cells are most inhibited when the wave of brightness is fully out of phase with the weights —


the image has the same phase as the weights. this occurs when the image is bright where the weights are positive and dark where the weights are negative. simple cells are most inhibited when the wave of brightness is fully out of phase with the weights — when the image is dark where the weights are positive and bright where the weights are negative. the cartoon view of a complex cell is that it computes the l2 norm of the 2 - d vector containing two simple cells ’ responses : c ( i ) = s0 ( ) i 2 + s1 ( ) i 2. an important special case occurs when s1 has all of the same parameters as s0 except for φ, and φ is set such that s1 is one quarter cycle out of phase with s0. in this case, s0 and s1 form a quadrature pair. a complex cell deﬁned in this way responds when the gaussian reweighted image i ( x, y ) exp ( −βxx2 −βyy 2 ) contains a high amplitude sinusoidal wave with frequency f in direction τ near ( x0, y0 ), regardless of the phase [UNK] of this wave. in other words, the complex cell is invariant to small translations of the


##yy 2 ) contains a high amplitude sinusoidal wave with frequency f in direction τ near ( x0, y0 ), regardless of the phase [UNK] of this wave. in other words, the complex cell is invariant to small translations of the image in direction τ, or to negating the image 369


chapter 9. convolutional networks figure 9. 18 : gabor functions with a variety of parameter settings. white indicates large positive weight, black indicates large negative weight, and the background gray corresponds to zero weight. ( left ) gabor functions with [UNK] values of the parameters that control the coordinate system : x 0, y0, and τ. each gabor function in this grid is assigned a value of x0 and y0 proportional to its position in its grid, and τ is chosen so that each gabor ﬁlter is sensitive to the direction radiating out from the center of the grid. for the other two plots, x0, y 0, and τ are ﬁxed to zero. gabor functions with ( center ) [UNK] gaussian scale parameters βx and βy. gabor functions are arranged in increasing width ( decreasing βx ) as we move left to right through the grid, and increasing height ( decreasing βy ) as we move top to bottom. for the other two plots, the β values are ﬁxed to 1. 5× the image width. gabor functions with [UNK] sinusoid parameters ( right ) f and φ. as we move top to bottom, f increases, and as we move left to right, φ increases


β values are ﬁxed to 1. 5× the image width. gabor functions with [UNK] sinusoid parameters ( right ) f and φ. as we move top to bottom, f increases, and as we move left to right, φ increases. for the other two plots, is ﬁxed to 0 and is ﬁxed to 5 the image width. φ f × ( replacing black with white and vice versa ). some of the most striking correspondences between neuroscience and machine learning come from visually comparing the features learned by machine learning models with those employed by v1. ( ) showed that olshausen and field 1996 a simple unsupervised learning algorithm, sparse coding, learns features with receptive ﬁelds similar to those of simple cells. since then, we have found that an extremely wide variety of statistical learning algorithms learn features with gabor - like functions when applied to natural images. this includes most deep learning algorithms, which learn these features in their ﬁrst layer. figure 9. 19 shows some examples. because so many [UNK] learning algorithms learn edge detectors, it is [UNK] to conclude that any speciﬁc learning algorithm is the “ right ” model of the brain just based on the features that it learns ( though it can certainly be a bad


examples. because so many [UNK] learning algorithms learn edge detectors, it is [UNK] to conclude that any speciﬁc learning algorithm is the “ right ” model of the brain just based on the features that it learns ( though it can certainly be a bad sign if an algorithm does learn some sort of edge detector not when applied to natural images ). these features are an important part of the statistical structure of natural images and can be recovered by many [UNK] approaches to statistical modeling. see hyvarinen 2009 et al. ( ) for a review of the ﬁeld of natural image statistics. 370


chapter 9. convolutional networks figure 9. 19 : many machine learning algorithms learn features that detect edges or speciﬁc colors of edges when applied to natural images. these feature detectors are reminiscent of the gabor functions known to be present in primary visual cortex. ( left ) weights learned by an unsupervised learning algorithm ( spike and slab sparse coding ) applied to small image patches. ( right ) convolution kernels learned by the ﬁrst layer of a fully supervised convolutional maxout network. neighboring pairs of ﬁlters drive the same maxout unit. 9. 11 convolutional networks and the history of deep learning convolutional networks have played an important role in the history of deep learning. they are a key example of a successful application of insights obtained by studying the brain to machine learning applications. they were also some of the ﬁrst deep models to perform well, long before arbitrary deep models were considered viable. convolutional networks were also some of the ﬁrst neural networks to solve important commercial applications and remain at the forefront of commercial applications of deep learning today. for example, in the 1990s, the neural network research group at at & t developed a convolution


networks were also some of the ﬁrst neural networks to solve important commercial applications and remain at the forefront of commercial applications of deep learning today. for example, in the 1990s, the neural network research group at at & t developed a convolutional network for reading checks (, ). by the end of the 1990s, this system deployed lecun et al. 1998b by nec was reading over 10 % of all the checks in the us. later, several ocr and handwriting recognition systems based on convolutional nets were deployed by microsoft (, ). see chapter for more details on such applications simard et al. 2003 12 and more modern applications of convolutional networks. see ( ) lecun et al. 2010 for a more in - depth history of convolutional networks up to 2010. convolutional networks were also used to win many contests. the current intensity of commercial interest in deep learning began when krizhevsky et al. ( ) won the imagenet object recognition challenge, but convolutional networks 2012 371


chapter 9. convolutional networks had been used to win other machine learning and computer vision contests with less impact for years earlier. convolutional nets were some of the ﬁrst working deep networks trained with back - propagation. it is not entirely clear why convolutional networks succeeded when general back - propagation networks were considered to have failed. it may simply be that convolutional networks were more computationally [UNK] than fully connected networks, so it was easier to run multiple experiments with them and tune their implementation and hyperparameters. larger networks also seem to be easier to train. with modern hardware, large fully connected networks appear to perform reasonably on many tasks, even when using datasets that were available and activation functions that were popular during the times when fully connected networks were believed not to work well. it may be that the primary barriers to the success of neural networks were psychological ( practitioners did not expect neural networks to work, so they did not make a serious [UNK] to use neural networks ). whatever the case, it is fortunate that convolutional networks performed well decades ago. in many ways, they carried the torch for the rest of deep learning and paved the way to the acceptance of neural networks in general. con


networks ). whatever the case, it is fortunate that convolutional networks performed well decades ago. in many ways, they carried the torch for the rest of deep learning and paved the way to the acceptance of neural networks in general. convolutional networks provide a way to specialize neural networks to work with data that has a clear grid - structured topology and to scale such models to very large size. this approach has been the most successful on a two - dimensional, image topology. to process one - dimensional, sequential data, we turn next to another powerful specialization of the neural networks framework : recurrent neural networks. 372


chapter 10 sequence modeling : recurrent and recursive nets recurrent neural networks or rnns (, ) are a family of rumelhart et al. 1986a neural networks for processing sequential data. much as a convolutional network is a neural network that is specialized for processing a grid of values x such as an image, a recurrent neural network is a neural network that is specialized for processing a sequence of values x ( 1 ),..., x ( ) τ. just as convolutional networks can readily scale to images with large width and height, and some convolutional networks can process images of variable size, recurrent networks can scale to much longer sequences than would be practical for networks without sequence - based specialization. most recurrent networks can also process sequences of variable length. to go from multi - layer networks to recurrent networks, we need to take advan - tage of one of the early ideas found in machine learning and statistical models of the 1980s : sharing parameters across [UNK] parts of a model. parameter sharing makes it possible to extend and apply the model to examples of [UNK] forms ( [UNK] lengths, here ) and generalize across them. if we had separate parameters for each value of the time index,


sharing parameters across [UNK] parts of a model. parameter sharing makes it possible to extend and apply the model to examples of [UNK] forms ( [UNK] lengths, here ) and generalize across them. if we had separate parameters for each value of the time index, we could not generalize to sequence lengths not seen during training, nor share statistical strength across [UNK] sequence lengths and across [UNK] positions in time. such sharing is particularly important when a speciﬁc piece of information can occur at multiple positions within the sequence. for example, consider the two sentences “ i went to nepal in 2009 ” and “ in 2009, i went to nepal. ” if we ask a machine learning model to read each sentence and extract the year in which the narrator went to nepal, we would like it to recognize the year 2009 as the relevant piece of information, whether it appears in the sixth 373


chapter 10. sequence modeling : recurrent and recursive nets word or the second word of the sentence. suppose that we trained a feedforward network that processes sentences of ﬁxed length. a traditional fully connected feedforward network would have separate parameters for each input feature, so it would need to learn all of the rules of the language separately at each position in the sentence. by comparison, a recurrent neural network shares the same weights across several time steps. a related idea is the use of convolution across a 1 - d temporal sequence. this convolutional approach is the basis for time - delay neural networks ( lang and hinton 1988 waibel 1989 lang 1990, ; et al., ; et al., ). the convolution operation allows a network to share parameters across time, but is shallow. the output of convolution is a sequence where each member of the output is a function of a small number of neighboring members of the input. the idea of parameter sharing manifests in the application of the same convolution kernel at each time step. recurrent networks share parameters in a [UNK] way. each member of the output is a function of the previous members of the output. each member of the output


sharing manifests in the application of the same convolution kernel at each time step. recurrent networks share parameters in a [UNK] way. each member of the output is a function of the previous members of the output. each member of the output is produced using the same update rule applied to the previous outputs. this recurrent formulation results in the sharing of parameters through a very deep computational graph. for the simplicity of exposition, we refer to rnns as operating on a sequence that contains vectors x ( ) t with the time step index t ranging from to 1 τ. in practice, recurrent networks usually operate on minibatches of such sequences, with a [UNK] sequence length τ for each member of the minibatch. we have omitted the minibatch indices to simplify notation. moreover, the time step index need not literally refer to the passage of time in the real world. sometimes it refers only to the position in the sequence. rnns may also be applied in two dimensions across spatial data such as images, and even when applied to data involving time, the network may have connections that go backwards in time, provided that the entire sequence is observed before it is provided to the network. this chapter extends the idea of a computational graph to include cycles. these cycles


and even when applied to data involving time, the network may have connections that go backwards in time, provided that the entire sequence is observed before it is provided to the network. this chapter extends the idea of a computational graph to include cycles. these cycles represent the inﬂuence of the present value of a variable on its own value at a future time step. such computational graphs allow us to deﬁne recurrent neural networks. we then describe many [UNK] ways to construct, train, and use recurrent neural networks. for more information on recurrent neural networks than is available in this chapter, we refer the reader to the textbook of graves 2012 ( ). 374


chapter 10. sequence modeling : recurrent and recursive nets 10. 1 unfolding computational graphs a computational graph is a way to formalize the structure of a set of computations, such as those involved in mapping inputs and parameters to outputs and loss. please refer to section for a general introduction. in this section we explain 6. 5. 1 the idea of unfolding a recursive or recurrent computation into a computational graph that has a repetitive structure, typically corresponding to a chain of events. unfolding this graph results in the sharing of parameters across a deep network structure. for example, consider the classical form of a dynamical system : s ( ) t = ( f s ( 1 ) t− ; ) θ, ( 10. 1 ) where s ( ) t is called the state of the system. equation is recurrent because the deﬁnition of 10. 1 s at time t refers back to the same deﬁnition at time. t −1 for a ﬁnite number of time steps τ, the graph can be unfolded by applying the deﬁnition τ −1 times. for example, if we unfold equation for 10. 1 τ = 3 time steps, we obtain s ( 3 ) = ( f s (


time steps τ, the graph can be unfolded by applying the deﬁnition τ −1 times. for example, if we unfold equation for 10. 1 τ = 3 time steps, we obtain s ( 3 ) = ( f s ( 2 ) ; ) θ ( 10. 2 ) = ( ( f f s ( 1 ) ; ) ; ) θ θ ( 10. 3 ) unfolding the equation by repeatedly applying the deﬁnition in this way has yielded an expression that does not involve recurrence. such an expression can now be represented by a traditional directed acyclic computational graph. the unfolded computational graph of equation and equation is illustrated in 10. 1 10. 3 ﬁgure. 10. 1 s ( t−1 ) s ( t−1 ) s ( ) t s ( ) t s ( + 1 ) t s ( + 1 ) t f s ( )... s ( )... s ( )... s ( )... f f f figure 10. 1 : the classical dynamical system described by equation, illustrated as an 10. 1 unfolded computational graph. each node represents the state at some timet and the function f maps the state at t to the state at t + 1


f figure 10. 1 : the classical dynamical system described by equation, illustrated as an 10. 1 unfolded computational graph. each node represents the state at some timet and the function f maps the state at t to the state at t + 1. the same parameters ( the same value of used to parametrize ) are used for all time steps. θ f as another example, let us consider a dynamical system driven by an external signal x ( ) t, s ( ) t = ( f s ( 1 ) t−, x ( ) t ; ) θ, ( 10. 4 ) 375


chapter 10. sequence modeling : recurrent and recursive nets where we see that the state now contains information about the whole past sequence. recurrent neural networks can be built in many [UNK] ways. much as almost any function can be considered a feedforward neural network, essentially any function involving recurrence can be considered a recurrent neural network. many recurrent neural networks use equation or a similar equation to 10. 5 deﬁne the values of their hidden units. to indicate that the state is the hidden units of the network, we now rewrite equation using the variable 10. 4 h to represent the state : h ( ) t = ( f h ( 1 ) t−, x ( ) t ; ) θ, ( 10. 5 ) illustrated in ﬁgure, typical rnns will add extra architectural features such 10. 2 as output layers that read information out of the state to make predictions. h when the recurrent network is trained to perform a task that requires predicting the future from the past, the network typically learns to use h ( ) t as a kind of lossy summary of the task - relevant aspects of the past sequence of inputs up to t. this summary is in general necessarily lossy, since it maps an arbitrary length sequence (


the network typically learns to use h ( ) t as a kind of lossy summary of the task - relevant aspects of the past sequence of inputs up to t. this summary is in general necessarily lossy, since it maps an arbitrary length sequence ( x ( ) t, x ( 1 ) t−, x ( 2 ) t−,..., x ( 2 ), x ( 1 ) ) to a ﬁxed length vector h ( ) t. depending on the training criterion, this summary might selectively keep some aspects of the past sequence with more precision than other aspects. for example, if the rnn is used in statistical language modeling, typically to predict the next word given previous words, it may not be necessary to store all of the information in the input sequence up to time t, but rather only enough information to predict the rest of the sentence. the most demanding situation is when we ask h ( ) t to be rich enough to allow one to approximately recover the input sequence, as in autoencoder frameworks ( chapter ). 14 f h x h ( t−1 ) h ( t−1 ) h ( ) t h ( ) t h ( + 1 ) t h ( + 1 ) t x ( t−1 ) x ( t


##coder frameworks ( chapter ). 14 f h x h ( t−1 ) h ( t−1 ) h ( ) t h ( ) t h ( + 1 ) t h ( + 1 ) t x ( t−1 ) x ( t−1 ) x ( ) t x ( ) t x ( + 1 ) t x ( + 1 ) t h ( )... h ( )... h ( )... h ( )... f unfold f f f figure 10. 2 : a recurrent network with no outputs. this recurrent network just processes information from the input x by incorporating it into the state h that is passed forward through time. ( left ) circuit diagram. the black square indicates a delay of a single time step. the same network seen as an unfolded computational graph, where each ( right ) node is now associated with one particular time instance. equation can be drawn in two [UNK] ways. one way to draw the rnn 10. 5 is with a diagram containing one node for every component that might exist in a 376


chapter 10. sequence modeling : recurrent and recursive nets physical implementation of the model, such as a biological neural network. in this view, the network deﬁnes a circuit that operates in real time, with physical parts whose current state can inﬂuence their future state, as in the left of ﬁgure. 10. 2 throughout this chapter, we use a black square in a circuit diagram to indicate that an interaction takes place with a delay of a single time step, from the state at time t to the state at time t + 1. the other way to draw the rnn is as an unfolded computational graph, in which each component is represented by many [UNK] variables, with one variable per time step, representing the state of the component at that point in time. each variable for each time step is drawn as a separate node of the computational graph, as in the right of ﬁgure. what we 10. 2 call unfolding is the operation that maps a circuit as in the left side of the ﬁgure to a computational graph with repeated pieces as in the right side. the unfolded graph now has a size that depends on the sequence length. we can represent the unfolded recurrence after steps with a function t g ( ) t :


the ﬁgure to a computational graph with repeated pieces as in the right side. the unfolded graph now has a size that depends on the sequence length. we can represent the unfolded recurrence after steps with a function t g ( ) t : h ( ) t = g ( ) t ( x ( ) t, x ( 1 ) t−, x ( 2 ) t−,..., x ( 2 ), x ( 1 ) ) ( 10. 6 ) = ( f h ( 1 ) t−, x ( ) t ; ) θ ( 10. 7 ) the function g ( ) t takes the whole past sequence ( x ( ) t, x ( 1 ) t−, x ( 2 ) t−,..., x ( 2 ), x ( 1 ) ) as input and produces the current state, but the unfolded recurrent structure allows us to factorize g ( ) t into repeated application of a function f. the unfolding process thus introduces two major advantages : 1. regardless of the sequence length, the learned model always has the same input size, because it is speciﬁed in terms of transition from one state to another state, rather than speciﬁed in terms of a variable - length history


1. regardless of the sequence length, the learned model always has the same input size, because it is speciﬁed in terms of transition from one state to another state, rather than speciﬁed in terms of a variable - length history of states. 2. it is possible to use the transition function same f with the same parameters at every time step. these two factors make it possible to learn a single model f that operates on all time steps and all sequence lengths, rather than needing to learn a separate model g ( ) t for all possible time steps. learning a single, shared model allows generalization to sequence lengths that did not appear in the training set, and allows the model to be estimated with far fewer training examples than would be required without parameter sharing. both the recurrent graph and the unrolled graph have their uses. the recurrent graph is succinct. the unfolded graph provides an explicit description of which computations to perform. the unfolded graph also helps to illustrate the idea of 377


chapter 10. sequence modeling : recurrent and recursive nets information ﬂow forward in time ( computing outputs and losses ) and backward in time ( computing gradients ) by explicitly showing the path along which this information ﬂows. 10. 2 recurrent neural networks armed with the graph unrolling and parameter sharing ideas of section, we 10. 1 can design a wide variety of recurrent neural networks. u v w o ( t−1 ) o ( t−1 ) h o y l x o ( ) t o ( ) t o ( + 1 ) t o ( + 1 ) t l ( t−1 ) l ( t−1 ) l ( ) t l ( ) t l ( + 1 ) t l ( + 1 ) t y ( t−1 ) y ( t−1 ) y ( ) t y ( ) t y ( + 1 ) t y ( + 1 ) t h ( t−1 ) h ( t−1 ) h ( ) t h ( ) t h ( + 1 ) t h ( + 1 ) t x ( t−1 ) x ( t−1 ) x ( ) t x ( ) t x ( + 1 ) t x ( + 1 ) t w w w w h ( )... h ( )...


+ 1 ) t x ( t−1 ) x ( t−1 ) x ( ) t x ( ) t x ( + 1 ) t x ( + 1 ) t w w w w h ( )... h ( )... h ( )... h ( )... v v v u u u unfold figure 10. 3 : the computational graph to compute the training loss of a recurrent network that maps an input sequence of x values to a corresponding sequence of output o values. a loss l measures how far each o is from the corresponding training targety. when using softmax outputs, we assume o is the unnormalized log probabilities. the lossl internally computes [UNK] = softmax ( o ) and compares this to the target y. the rnn has input to hidden connections parametrized by a weight matrix u, hidden - to - hidden recurrent connections parametrized by a weight matrix w, and hidden - to - output connections parametrized by a weight matrix v. equation deﬁnes forward propagation in this model. 10. 8 ( left ) the rnn and its loss drawn with recurrent connections. ( right ) the same seen as an time - unfolded computational graph, where each


by a weight matrix v. equation deﬁnes forward propagation in this model. 10. 8 ( left ) the rnn and its loss drawn with recurrent connections. ( right ) the same seen as an time - unfolded computational graph, where each node is now associated with one particular time instance. some examples of important design patterns for recurrent neural networks include the following : 378


chapter 10. sequence modeling : recurrent and recursive nets • recurrent networks that produce an output at each time step and have recurrent connections between hidden units, illustrated in ﬁgure. 10. 3 • recurrent networks that produce an output at each time step and have recurrent connections only from the output at one time step to the hidden units at the next time step, illustrated in ﬁgure 10. 4 • recurrent networks with recurrent connections between hidden units, that read an entire sequence and then produce a single output, illustrated in ﬁgure. 10. 5 ﬁgure is a reasonably representative example that we return to throughout 10. 3 most of the chapter. the recurrent neural network of ﬁgure and equation is universal in the 10. 3 10. 8 sense that any function computable by a turing machine can be computed by such a recurrent network of a ﬁnite size. the output can be read from the rnn after a number of time steps that is asymptotically linear in the number of time steps used by the turing machine and asymptotically linear in the length of the input ( siegelmann and sontag 1991 siegelmann 1995 siegelmann and sontag 1995, ;, ;,