Deep Learning with TensorFlow and Keras: Build and deploy supervised, unsupervised, deep, and reinforcement learning models, 3rd Edition

enter site Preface Who this book is for What this book covers Get in touch References Neural Network Foundations with TF What is TensorFlow (TF)? What is Keras? Introduction to neural networks Perceptron Our first example of TensorFlow code Multi-layer perceptron: our first example of a network Problems in training the perceptron and solution Activation function: sigmoid Activation function: tanh Activation function: ReLU Two additional activation functions: ELU and Leaky ReLU Activation functions In short: what are neural networks after all? A real example: recognizing handwritten digits One hot-encoding (OHE) Defining a simple neural net in TensorFlow Running a simple TensorFlow net and establishing a baseline Improving the simple net in TensorFlow with hidden layers Further improving the simple net in TensorFlow with dropout Testing different optimizers in TensorFlow Increasing the number of epochs Controlling the optimizer learning rate Increasing the number of internal hidden neurons Increasing the size of batch computation Summarizing experiments run to recognizing handwritten digits Regularization Adopting regularization to avoid overfitting Understanding batch normalization Playing with Google Colab: CPUs, GPUs, and TPUs Sentiment analysis Hyperparameter tuning and AutoML Predicting output A practical overview of backpropagation What have we learned so far? Toward a deep learning approach Summary References Regression and Classification What is regression? Prediction using linear regression Simple linear regression Multiple linear regression Multivariate linear regression Neural networks for linear regression Simple linear regression using TensorFlow Keras Multiple and multivariate linear regression using the TensorFlow Keras API Classification tasks and decision boundaries Logistic regression Logistic regression on the MNIST dataset Summary References Convolutional Neural Networks Deep convolutional neural networks Local receptive fields Shared weights and bias A mathematical example ConvNets in TensorFlow Pooling layers Max pooling Average pooling ConvNets summary An example of DCNN: LeNet LeNet code in TF Understanding the power of deep learning Recognizing CIFAR-10 images with deep learning Improving the CIFAR-10 performance with a deeper network Improving the CIFAR-10 performance with data augmentation Predicting with CIFAR-10 Very deep convolutional networks for large-scale image recognition Recognizing cats with a VGG16 network Utilizing the tf.Keras built-in VGG16 net module Recycling pre-built deep learning models for extracting features Deep Inception V3 for transfer learning Other CNN architectures AlexNet Residual networks HighwayNets and DenseNets Xception Style transfer Content distance Style distance Summary References Word Embeddings Word embedding ‒ origins and fundamentals Distributed representations Static embeddings Word2Vec GloVe Creating your own embeddings using Gensim Exploring the embedding space with Gensim Using word embeddings for spam detection Getting the data Making the data ready for use Building the embedding matrix Defining the spam classifier Training and evaluating the model Running the spam detector Neural embeddings – not just for words Item2Vec node2vec Character and subword embeddings Dynamic embeddings Sentence and paragraph embeddings Language model-based embeddings Using BERT as a feature extractor Summary References Recurrent Neural Networks The basic RNN cell Backpropagation through time (BPTT) Vanishing and exploding gradients RNN cell variants Long short-term memory (LSTM) Gated recurrent unit (GRU) Peephole LSTM RNN variants Bidirectional RNNs Stateful RNNs RNN topologies Example ‒ One-to-many – Learning to generate text Example ‒ Many-to-one – Sentiment analysis Example ‒ Many-to-many – POS tagging Encoder-decoder architecture – seq2seq Example ‒ seq2seq without attention for machine translation Attention mechanism Example ‒ seq2seq with attention for machine translation Summary References Transformers Architecture Key intuitions Positional encoding Attention Self-attention Multi-head (self-)attention How to compute attention Encoder-decoder architecture Residual and normalization layers An overview of the transformer architecture Training Transformers’ architectures Categories of transformers Decoder or autoregressive Encoder or autoencoding Seq2seq Multimodal Retrieval Attention Full versus sparse LSH attention Local attention Pretraining Encoder pretraining Decoder pretraining Encoder-decoder pretraining A taxonomy for pretraining tasks An overview of popular and well-known models BERT GPT-2 GPT-3 Reformer BigBird Transformer-XL XLNet RoBERTa ALBERT StructBERT T5 and MUM ELECTRA DeBERTa The Evolved Transformer and MEENA LaMDA Switch Transformer RETRO Pathways and PaLM Implementation Transformer reference implementation: An example of translation Hugging Face Generating text Autoselecting a model and autotokenization Named entity recognition Summarization Fine-tuning TFHub Evaluation Quality GLUE SuperGLUE SQuAD RACE NLP-progress Size Larger doesn’t always mean better Cost of serving Optimization Quantization Weight pruning Distillation Common pitfalls: dos and don’ts Dos Don’ts The future of transformers Summary Unsupervised Learning Principal component analysis PCA on the MNIST dataset TensorFlow Embedding API K-means clustering K-means in TensorFlow Variations in k-means Self-organizing maps Colour mapping using a SOM Restricted Boltzmann machines Reconstructing images using an RBM Deep belief networks Summary References Autoencoders Introduction to autoencoders Vanilla autoencoders TensorFlow Keras layers ‒ defining custom layers Reconstructing handwritten digits using an autoencoder Sparse autoencoder Denoising autoencoders Clearing images using a denoising autoencoder Stacked autoencoder Convolutional autoencoder for removing noise from images A TensorFlow Keras autoencoder example ‒ sentence vectors Variational autoencoders Summary References Generative Models What is a GAN? MNIST using GAN in TensorFlow Deep convolutional GAN (DCGAN) DCGAN for MNIST digits Some interesting GAN architectures SRGAN CycleGAN InfoGAN Cool applications of GANs CycleGAN in TensorFlow Flow-based models for data generation Diffusion models for data generation Summary References Self-Supervised Learning Previous work Self-supervised learning Self-prediction Autoregressive generation PixelRNN Image GPT (IPT) GPT-3 XLNet WaveNet WaveRNN Masked generation BERT Stacked denoising autoencoder Context autoencoder Colorization Innate relationship prediction Relative position Solving jigsaw puzzles Rotation Hybrid self-prediction VQ-VAE Jukebox DALL-E VQ-GAN Contrastive learning Training objectives Contrastive loss Triplet loss N-pair loss Lifted structural loss NCE loss InfoNCE loss Soft nearest neighbors loss Instance transformation SimCLR Barlow Twins BYOL Feature clustering DeepCluster SwAV InterCLR Multiview coding AMDIM CMC Multimodal models CLIP CodeSearchNet Data2Vec Pretext tasks Summary References Reinforcement Learning An introduction to RL RL lingo Deep reinforcement learning algorithms How does the agent choose its actions, especially when untrained? How does the agent maintain a balance between exploration and exploitation? How to deal with the highly correlated input state space How to deal with the problem of moving targets Reinforcement success in recent years Simulation environments for RL An introduction to OpenAI Gym Random agent playing Breakout Wrappers in Gym Deep Q-networks DQN for CartPole DQN to play a game of Atari DQN variants Double DQN Dueling DQN Rainbow Deep deterministic policy gradient Summary References Probabilistic TensorFlow TensorFlow Probability TensorFlow Probability distributions Using TFP distributions Coin Flip Example Normal distribution Bayesian networks Handling uncertainty in predictions using TensorFlow Probability Aleatory uncertainty Epistemic uncertainty Creating a synthetic dataset Building a regression model using TensorFlow Probabilistic neural networks for aleatory uncertainty Accounting for the epistemic uncertainty Summary References An Introduction to AutoML What is AutoML? Achieving AutoML Automatic data preparation Automatic feature engineering Automatic model generation AutoKeras Google Cloud AutoML and Vertex AI Using the Google Cloud AutoML Tables solution Using the Google Cloud AutoML Text solution Using the Google Cloud AutoML Video solution Cost Summary References The Math Behind Deep Learning History Some mathematical tools Vectors Derivatives and gradients everywhere Gradient descent Chain rule A few differentiation rules Matrix operations Activation functions Derivative of the sigmoid Derivative of tanh Derivative of ReLU Backpropagation Forward step Backstep Case 1: From hidden layer to output layer Case 2: From hidden layer to hidden layer Cross entropy and its derivative Batch gradient descent, stochastic gradient descent, and mini-batch Batch gradient descent Stochastic gradient descent Mini-batch gradient descent Thinking about backpropagation and ConvNets Thinking about backpropagation and RNNs A note on TensorFlow and automatic differentiation Summary References Tensor Processing Unit C/G/T processing units CPUs and GPUs TPUs Four generations of TPUs, plus Edge TPU First generation TPU Second generation TPU Third generation TPU Fourth generation TPUs Edge TPU TPU performance How to use TPUs with Colab Checking whether TPUs are available Keras MNIST TPU end-to-end training Using pretrained TPU models Summary References Other Useful Deep Learning Libraries Hugging Face OpenAI OpenAI GPT-3 API OpenAI DALL-E 2 OpenAI Codex PyTorch ONNX H2O.ai H2O AutoML AutoML using H2O H2O model explainability Partial dependence plots Variable importance heatmap Model correlation Summary Graph Neural Networks Graph basics Graph machine learning Graph convolutions – the intuition behind GNNs Common graph layers Graph convolution network Graph attention network GraphSAGE (sample and aggregate) Graph isomorphism network Common graph applications Node classification Graph classification Link prediction Graph customizations Custom layers and message passing Custom graph dataset Single graphs in datasets Set of multiple graphs in datasets Future directions Heterogeneous graphs Temporal Graphs Summary References Machine Learning Best Practices The need for best practices Data best practices Feature selection Features and data Augmenting textual data Model best practices Baseline models Pretrained models, model APIs, and AutoML Model evaluation and validation Model improvements Summary References TensorFlow 2 Ecosystem TensorFlow Hub Using pretrained models for inference TensorFlow Datasets Load a TFDS dataset Building data pipelines using TFDS TensorFlow Lite Quantization FlatBuffers Mobile converter Mobile optimized interpreter Supported platforms Architecture Using TensorFlow Lite A generic example of an application Using GPUs and accelerators An example of an application Pretrained models in TensorFlow Lite Image classification Object detection Pose estimation Smart reply Segmentation Style transfer Text classification Large language models A note about using mobile GPUs An overview of federated learning at the edge TensorFlow FL APIs TensorFlow.js Vanilla TensorFlow.js Converting models Pretrained models Node.js Summary References Advanced Convolutional Neural Networks Composing CNNs for complex tasks Classification and localization Semantic segmentation Object detection Instance segmentation Application zoos with tf.Keras and TensorFlow Hub Keras Applications TensorFlow Hub Answering questions about images (visual Q&A) Creating a DeepDream network Inspecting what a network has learned Video Classifying videos with pretrained nets in six different ways Text documents Using a CNN for sentiment analysis Audio and music Dilated ConvNets, WaveNet, and NSynth A summary of convolution operations Basic CNNs Dilated convolution Transposed convolution Separable convolution Depthwise convolution Depthwise separable convolution Capsule networks What is the problem with CNNs? What is new with capsule networks? Summary References Other Books You May Enjoy Index