##slavsky, t. ( 1975 ). facing up to arrangements : face - count formulas for partitions of space by hyperplanes. number no. 154 in memoirs of the american mathematical society. american mathematical society. 550 zeiler, m. d. and fergus, r. ( 2014 ). visualizing and understanding convolutional networks. in. eccv ’ 14 6 zeiler, m. d., ranzato, m., monga, r., mao, m., yang, k., le, q., nguyen, p., senior, a., vanhoucke, v., dean, j., and hinton, g. e. ( 2013 ). on rectiﬁed linear units for speech processing. in. icassp 2013 460 zhou, b., khosla, a., lapedriza, a., oliva, a., and torralba, a. ( 2015 ). object detectors emerge in deep scene cnns. iclr ’ 2015, arxiv : 1412. 6856. 551 zhou, j. and troyanskaya, o. g. ( 2014 ). deep supervised and convo


). object detectors emerge in deep scene cnns. iclr ’ 2015, arxiv : 1412. 6856. 551 zhou, j. and troyanskaya, o. g. ( 2014 ). deep supervised and convolutional generative stochastic network for protein secondary structure prediction. in. icml ’ 2014 715 zhou, y. and chellappa, r. ( 1988 ). computation of optical ﬂow using a neural network. in neural networks, 1988., ieee international conference on, pages 71 – 78. ieee. 339 zohrer, m. and pernkopf, f. ( 2014 ). general stochastic networks for classiﬁcation. in nips ’ 2014. 716 777

/home/ricoiban/GEMMA/mnlp_chatsplaining/RAG/Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville (z-lib.org).pdf


index 0 - 1 loss,, 102 274 absolute value rectiﬁcation, 191 accuracy, 420 activation function, 169 active constraint, 94 adagrad, 305 adaline, see adaptive linear element adam,, 307 422 adaptive linear element,,, 15 23 26 adversarial example, 265 adversarial training,,, 266 268 526 [UNK], 109 ais, see annealed importance sampling almost everywhere, 70 almost sure convergence, 128 ancestral sampling,, 576 591 ann, see artiﬁcial neural network annealed importance sampling,,, 621 662 711 approximate bayesian computation, 710 approximate inference, 579 artiﬁcial intelligence, 1 artiﬁcial neural network, see neural net - work asr, see automatic speech recognition asymptotically unbiased, 123 audio,,, 101 357 455 autoencoder,,, 4 353 498 automatic speech recognition, 455 back - propagation, 201 back - propagation through time, 381 backprop, see back - propagation bag of words, 467 bagging, 252 batch normalization,, 264 422 bayes error, 116 bayes ’ rule, 69 bayesian hyperparameter optimization, 433 bayesian


time, 381 backprop, see back - propagation bag of words, 467 bagging, 252 batch normalization,, 264 422 bayes error, 116 bayes ’ rule, 69 bayesian hyperparameter optimization, 433 bayesian network, see directed graphical model bayesian probability, 54 bayesian statistics, 134 belief network, see directed graphical model bernoulli distribution, 61 bfgs, 314 bias,, 123 227 bias parameter, 109 biased importance sampling, 589 bigram, 458 binary relation, 478 block gibbs sampling, 595 boltzmann distribution, 566 boltzmann machine,, 566 648 bptt, see back - propagation through time broadcasting, 33 burn - in, 593 cae, see contractive autoencoder calculus of variations, 178 categorical distribution, see multinoulli dis - tribution cd, see contrastive divergence centering trick ( dbm ), 667 central limit theorem, 63 chain rule ( calculus ), 203 chain rule of probability, 58 778


index chess, 2 chord, 575 chordal graph, 575 class - based language models, 460 classical dynamical system, 372 classiﬁcation, 99 clique potential, see factor ( graphical model ) cnn, see convolutional neural network collaborative filtering, 474 collider, see explaining away color images, 357 complex cell, 362 computational graph, 202 computer vision, 449 concept drift, 533 condition number, 277 conditional computation, see dynamic struc - ture conditional independence,, xiii 59 conditional probability, 58 conditional rbm, 679 connectionism,, 17 440 connectionist temporal classiﬁcation, 457 consistency,, 128 509 constrained optimization,, 92 235 content - based addressing, 416 content - based recommender systems, 475 context - speciﬁc independence, 569 contextual bandits, 476 continuation methods, 324 contractive autoencoder, 516 contrast, 451 contrastive divergence,,, 289 606 666 convex optimization, 140 convolution,, 327 677 convolutional network, 16 convolutional neural network,, 250 327,, 422 456 coordinate descent,, 319 665 correlation, 60 cost function


##6 convex optimization, 140 convolution,, 327 677 convolutional network, 16 convolutional neural network,, 250 327,, 422 456 coordinate descent,, 319 665 correlation, 60 cost function, see objective function covariance,, xiii 60 covariance matrix, 61 coverage, 421 critical temperature, 599 cross - correlation, 329 cross - entropy,, 74 131 cross - validation, 121 ctc, see connectionist temporal classiﬁca - tion curriculum learning, 326 curse of dimensionality, 153 cyc, 2 d - separation, 568 dae, see denoising autoencoder data generating distribution,, 110 130 data generating process, 110 data parallelism, 444 dataset, 103 dataset augmentation,, 268 454 dbm, see deep boltzmann machine dcgan,,, 547 548 695 decision tree,, 144 544 decoder, 4 deep belief network,,,,,, 26 525 626 651 654 678 686, deep blue, 2 deep boltzmann machine,,,,, 23 26 525 626 647 651 657 666 678,,,, deep


,,, 26 525 626 651 654 678 686, deep blue, 2 deep boltzmann machine,,,,, 23 26 525 626 647 651 657 666 678,,,, deep feedforward network,, 166 422 deep learning,, 2 5 denoising autoencoder,, 506 683 denoising score matching, 615 density estimation, 102 derivative,, xiii 82 design matrix, 105 detector layer, 336 determinant, xii diagonal matrix, 40 [UNK] entropy,, 73 641 dirac delta function, 64 directed graphical model,,,, 76 503 559 685 directional derivative, 84 discriminative ﬁne - tuning, see supervised ﬁne - tuning discriminative rbm, 680 distributed representation,,, 17 149 542 domain adaptation, 532 779


index dot product,, 33 139 double backprop, 268 doubly block circulant matrix, 330 dream sleep,, 605 647 dropconnect, 263 dropout,,,,,, 255 422 427 428 666 683 dynamic structure, 445 e - step, 629 early stopping,,,,, 244 246 270 271 422 ebm, see energy - based model echo state network,,, 23 26 401 [UNK] capacity, 113 eigendecomposition, 41 eigenvalue, 41 eigenvector, 41 elbo, see evidence lower bound element - wise product, see hadamard prod - uct, see hadamard product em, see expectation maximization embedding, 512 empirical distribution, 65 empirical risk, 274 empirical risk minimization, 274 encoder, 4 energy function, 565 energy - based model,,,, 565 591 648 657 ensemble methods, 252 epoch, 244 equality constraint, 93 equivariance, 335 error function, see objective function esn, see echo state network euclidean norm, 38 euler - lagrange equation, 641 evidence lower bound,, 628 655 example, 98 expectation,


constraint, 93 equivariance, 335 error function, see objective function esn, see echo state network euclidean norm, 38 euler - lagrange equation, 641 evidence lower bound,, 628 655 example, 98 expectation, 59 expectation maximization, 629 expected value, see expectation explaining away,,, 570 626 639 exploitation, 477 exploration, 477 exponential distribution, 64 f - score, 420 factor ( graphical model ), 563 factor analysis, 486 factor graph, 575 factors of variation, 4 feature, 98 feature selection, 234 feedforward neural network, 166 fine - tuning, 321 finite [UNK], 436 forget gate, 304 forward propagation, 201 fourier transform,, 357 359 fovea, 363 fpcd, 610 free energy,, 567 674 freebase, 479 frequentist probability, 54 frequentist statistics, 134 frobenius norm, 45 fully - visible bayes network, 699 functional derivatives, 640 fvbn, see fully - visible bayes network gabor function, 365 gans, see generative adversarial networks gated recurrent unit, 422 gaussian distribution, see normal distribu - tion gaussian kernel


##n, see fully - visible bayes network gabor function, 365 gans, see generative adversarial networks gated recurrent unit, 422 gaussian distribution, see normal distribu - tion gaussian kernel, 140 gaussian mixture,, 66 187 gcn, see global contrast normalization geneontology, 479 generalization, 109 generalized lagrange function, see general - ized lagrangian generalized lagrangian, 93 generative adversarial networks,, 683 693 generative moment matching networks, 696 generator network, 687 gibbs distribution, 564 gibbs sampling,, 577 595 global contrast normalization, 451 gpu, see graphics processing unit gradient, 83 780


index gradient clipping,, 287 411 gradient descent,, 82 84 graph, xii graphical model, see structured probabilis - tic model graphics processing unit, 441 greedy algorithm, 321 greedy layer - wise unsupervised pretraining, 524 greedy supervised pretraining, 321 grid search, 429 hadamard product,, xii 33 hard, tanh 195 harmonium, see restricted boltzmann ma - chine harmony theory, 567 helmholtz free energy, see evidence lower bound hessian, 221 hessian matrix,, xiii 86 heteroscedastic, 186 hidden layer,, 6 166 hill climbing, 85 hyperparameter optimization, 429 hyperparameters,, 119 427 hypothesis space,, 111 117 i. i. d. assumptions,,, 110 121 265 identity matrix, 35 ilsvrc, see imagenet large scale visual recognition challenge imagenet large scale visual recognition challenge, 22 immorality, 573 importance sampling,,, 588 620 691 importance weighted autoencoder, 691 independence,, xiii 59 independent and identically distributed, see i. i. d. assumptions independent component analysis, 487 independent subspace analysis, 489 inequality constraint, 93 inference,


620 691 importance weighted autoencoder, 691 independence,, xiii 59 independent and identically distributed, see i. i. d. assumptions independent component analysis, 487 independent subspace analysis, 489 inequality constraint, 93 inference,,,,,,,, 558 579 626 628 630 633 643 646 information retrieval, 520 initialization, 298 integral, xiii invariance, 339 isotropic, 64 jacobian matrix,,, xiii 71 85 joint probability, 56 k - means,, 361 542 k - nearest neighbors,, 141 544 karush - kuhn - tucker conditions,, 94 235 karush – kuhn – tucker, 93 kernel ( convolution ),, 328 329 kernel machine, 544 kernel trick, 139 kkt, see karush – kuhn – tucker kkt conditions, see karush - kuhn - tucker conditions kl divergence, see kullback - leibler diver - gence knowledge base,, 2 479 krylov methods, 222 kullback - leibler divergence,, xiii 73 label smoothing, 241 lagrange multipliers,, 93 641 lagrangian, see generalized


- gence knowledge base,, 2 479 krylov methods, 222 kullback - leibler divergence,, xiii 73 label smoothing, 241 lagrange multipliers,, 93 641 lagrangian, see generalized lagrangian lapgan, 695 laplace distribution,, 64 492 latent variable, 66 layer ( neural network ), 166 lcn, see local contrast normalization leaky relu, 191 leaky units, 404 learning rate, 84 line search,,, 84 85 92 linear combination, 36 linear dependence, 37 linear factor models, 485 linear regression,,, 106 109 138 link prediction, 480 lipschitz constant, 91 lipschitz continuous, 91 liquid state machine, 401 781


index local conditional probability distribution, 560 local contrast normalization, 452 logistic regression,,, 3 138 139 logistic sigmoid,, 7 66 long short - term memory,,,, 18 24 304 407, 422 loop, 575 loopy belief propagation, 581 loss function, see objective function lp norm, 38 lstm, see long short - term memory m - step, 629 machine learning, 2 machine translation, 100 main diagonal, 32 manifold, 159 manifold hypothesis, 160 manifold learning, 160 manifold tangent classiﬁer, 268 map approximation,, 137 501 marginal probability, 57 markov chain, 591 markov chain monte carlo, 591 markov network, see undirected model markov random ﬁeld, see undirected model matrix,,, xi xii 31 matrix inverse, 35 matrix product, 33 max norm, 39 max pooling, 336 maximum likelihood, 130 maxout,, 191 422 mcmc, see markov chain monte carlo mean ﬁeld,,, 633 634 666 mean squared error, 107 measure theory, 70 measure zero, 70 memory network,, 413 415 method of steepest descent, see gradient descent minibatch, 277 missing inputs, 99 mixing ( markov


,, 633 634 666 mean squared error, 107 measure theory, 70 measure zero, 70 memory network,, 413 415 method of steepest descent, see gradient descent minibatch, 277 missing inputs, 99 mixing ( markov chain ), 597 mixture density networks, 187 mixture distribution, 65 mixture model,, 187 506 mixture of experts,, 446 544 mlp, see multilayer perception mnist,,, 20 21 666 model averaging, 252 model compression, 444 model identiﬁability, 282 model parallelism, 444 moment matching, 696 moore - penrose pseudoinverse,, 44 237 moralized graph, 573 mp - dbm, see multi - prediction dbm mrf ( markov random field ), see undi - rected model mse, see mean squared error multi - modal learning, 535 multi - prediction dbm, 668 multi - task learning,, 242 533 multilayer perception, 5 multilayer perceptron, 26 multinomial distribution, 61 multinoulli distribution, 61 n - gram, 458 nade, 702 naive bayes, 3 nat, 72 natural image, 555 natural language processing, 457


##layer perceptron, 26 multinomial distribution, 61 multinoulli distribution, 61 n - gram, 458 nade, 702 naive bayes, 3 nat, 72 natural image, 555 natural language processing, 457 nearest neighbor regression, 114 negative deﬁnite, 88 negative phase,,, 466 602 604 neocognitron,,,, 16 23 26 364 nesterov momentum, 298 netﬂix grand prize,, 255 475 neural language model,, 460 472 neural network, 13 neural turing machine, 415 neuroscience, 15 newton ’ s method,, 88 309 nlm, see neural language model nlp, see natural language processing no free lunch theorem, 115 782


index noise - contrastive estimation, 616 non - parametric model, 113 norm,, xiv 38 normal distribution,,, 62 63 124 normal equations,,,, 108 108 111 232 normalized initialization, 301 numerical [UNK], see ﬁnite [UNK] - ences object detection, 449 object recognition, 449 objective function, 81 omp -, k see orthogonal matching pursuit one - shot learning, 534 operation, 202 optimization,, 79 81 orthodox statistics, see frequentist statistics orthogonal matching pursuit,, 26 252 orthogonal matrix, 41 orthogonality, 40 output layer, 166 parallel distributed processing, 17 parameter initialization,, 298 403 parameter sharing,,,,, 249 332 370 372 386 parameter tying, see parameter sharing parametric model, 113 parametric relu, 191 partial derivative, 83 partition function,,, 564 601 663 pca, see principal components analysis pcd, see stochastic maximum likelihood perceptron,, 15 26 persistent contrastive divergence, see stochas - tic maximum likelihood perturbation analysis, see reparametrization trick point estimator, 121 policy, 476 pooling,, 327 677 positive deﬁnite, 88 positive phase,,


##gence, see stochas - tic maximum likelihood perturbation analysis, see reparametrization trick point estimator, 121 policy, 476 pooling,, 327 677 positive deﬁnite, 88 positive phase,,,,, 466 602 604 650 662 precision, 420 precision ( of a normal distribution ),, 62 64 predictive sparse decomposition, 519 preprocessing, 450 pretraining,, 320 524 primary visual cortex, 362 principal components analysis,,,, 47 145 146 486 626, prior probability distribution, 134 probabilistic max pooling, 677 probabilistic pca,,, 486 487 627 probability density function, 57 probability distribution, 55 probability mass function, 55 probability mass function estimation, 102 product of experts, 566 product rule of probability, see chain rule of probability psd, see predictive sparse decomposition pseudolikelihood, 611 quadrature pair, 366 quasi - newton methods, 314 radial basis function, 195 random search, 431 random variable, 55 ratio matching, 614 rbf, 195 rbm, see restricted boltzmann machine recall, 420 receptive ﬁeld, 334 recommender systems


quasi - newton methods, 314 radial basis function, 195 random search, 431 random variable, 55 ratio matching, 614 rbf, 195 rbm, see restricted boltzmann machine recall, 420 receptive ﬁeld, 334 recommender systems, 474 rectiﬁed linear unit,,,, 170 191 422 503 recurrent network, 26 recurrent neural network, 375 regression, 99 regularization,,,,, 119 119 176 226 427 regularizer, 118 reinforce, 683 reinforcement learning,,,, 24 105 476 683 relational database, 479 relations, 478 reparametrization trick, 682 representation learning, 3 representational capacity, 113 restricted boltzmann machine,,, 353 456 475 583 626 650 651 666 670,,,,,,, 783


index 672 674 677,, ridge regression, see weight decay risk, 273 rnn - rbm, 679 saddle points, 283 sample mean, 124 scalar,,, xi xii 30 score matching,, 509 613 second derivative, 85 second derivative test, 88 self - information, 72 semantic hashing, 521 semi - supervised learning, 241 separable convolution, 359 separation ( probabilistic modeling ), 568 set, xii sgd, see stochastic gradient descent shannon entropy,, xiii 73 shortlist, 462 sigmoid,, xiv see logistic sigmoid sigmoid belief network, 26 simple cell, 362 singular value, see singular value decompo - sition singular value decomposition,,, 43 146 475 singular vector, see singular value decom - position slow feature analysis, 489 sml, see stochastic maximum likelihood softmax,,, 182 415 446 softplus,,, xiv 67 195 spam detection, 3 sparse coding,,,,, 319 353 492 626 686 sparse initialization,, 302 403 sparse representation,,,,, 145 224 251 501 552 spearmint, 433 spectral radius,


spam detection, 3 sparse coding,,,,, 319 353 492 626 686 sparse initialization,, 302 403 sparse representation,,,,, 145 224 251 501 552 spearmint, 433 spectral radius, 401 speech recognition, see automatic speech recognition sphering, see whitening spike and slab restricted boltzmann ma - chine, 674 spn, see sum - product network square matrix, 37 ssrbm, see spike and slab restricted boltz - mann machine standard deviation, 60 standard error, 126 standard error of the mean,, 126 276 statistic, 121 statistical learning theory, 109 steepest descent, see gradient descent stochastic back - propagation, see reparametriza - tion trick stochastic gradient descent,,,, 15 149 277 292, 666 stochastic maximum likelihood,, 608 666 stochastic pooling, 263 structure learning, 578 structured output,, 100 679 structured probabilistic model,, 76 554 sum rule of probability, 57 sum - product network, 549 supervised ﬁne - tuning,, 525 656 supervised learning, 104 support vector machine, 139 surrogate loss function, 274 svd, see singular value decomposition symmetric


, 76 554 sum rule of probability, 57 sum - product network, 549 supervised ﬁne - tuning,, 525 656 supervised learning, 104 support vector machine, 139 surrogate loss function, 274 svd, see singular value decomposition symmetric matrix,, 40 42 tangent distance, 267 tangent plane, 511 tangent prop, 267 tdnn, see time - delay neural network teacher forcing,, 379 380 tempering, 599 template matching, 140 tensor,,, xi xii 32 test set, 109 tikhonov regularization, see weight decay tiled convolution, 349 time - delay neural network,, 364 371 toeplitz matrix, 330 topographic ica, 489 trace operator, 45 training error, 109 transcription, 100 transfer learning, 532 784


index transpose,, xii 32 triangle inequality, 38 triangulated graph, see chordal graph trigram, 458 unbiased, 123 undirected graphical model,, 76 503 undirected model, 562 uniform distribution, 56 unigram, 458 unit norm, 40 unit vector, 40 universal approximation theorem, 196 universal approximator, 549 unnormalized probability distribution, 563 unsupervised learning,, 104 144 unsupervised pretraining,, 456 524 v - structure, see explaining away v1, 362 vae, see variational autoencoder vapnik - chervonenkis dimension, 113 variance,,, xiii 60 227 variational autoencoder,, 683 690 variational derivatives, see functional deriva - tives variational free energy, see evidence lower bound vc dimension, see vapnik - chervonenkis di - mension vector,,, xi xii 31 virtual adversarial examples, 266 visible layer, 6 volumetric data, 357 wake - sleep,, 646 655 weight decay,,,, 117 176 229 428 weight space symmetry, 282 weights,, 15 106 whitening, 452


##versarial examples, 266 visible layer, 6 volumetric data, 357 wake - sleep,, 646 655 weight decay,,,, 117 176 229 428 weight space symmetry, 282 weights,, 15 106 whitening, 452 wikibase, 479 wikibase, 479 word embedding, 460 word - sense disambiguation, 480 wordnet, 479 zero - data learning, see zero - shot learning zero - shot learning, 534 785