What this book covers
Chapter 1, Neural Network Foundations with TF, is where we learn the basics of TensorFlow, an open-source library developed by Google for machine learning and deep learning. In addition, we introduce the basics of neural networks and deep learning, two areas of machine learning that had incredible growth during the last few years. The idea behind this chapter is to provide all the tools needed to do basic but fully hands-on deep learning.
Chapter 2, Regression and Classification, focuses on the fundamental tasks in ML techniques: regression and classification. We will learn how to use TensorFlow to build simple, multiple, and multivariate regression models. We will use logistic regression to solve a multi-class classification problem.
Chapter 3, Convolutional Neural Networks, covers how to use deep learning ConvNets for recognizing MNIST handwritten characters with high accuracy. We use the CIFAR 10 dataset to build a deep learning classifier with 10 categories, and the ImageNet dataset to build an accurate classifier with 1,000 categories. In addition, we investigate how to use large deep learning networks such as VGG16 and very deep networks such as InceptionV3. We will conclude with a discussion on transfer learning
Chapter 4, Word Embeddings, is where we describe the origins of and theory behind distributed representations and word embeddings and chart the progress of word embeddings from static word-based embeddings more dynamic and expressive embeddings based on sentences and paragraphs. We also explore how the idea of word embeddings can be extended to include non-word sequences as well, such as nodes in a graph or user sessions in a web application. The chapter also contains multiple examples of using word embeddings of various kinds.
Chapter 5, Recurrent Neural Networks, describes an important architectural subclass of neural networks that are optimized for handling sequence data such as natural language or time series. We describe the important architectures in this genre, such as LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) and show how they can be extended to handle bidirectional states and states across batches. We also provide examples of using RNNs with various topologies for specific tasks, such as generating text, sentiment analysis, and part-of-speech tagging. We also describe the popular seq2seq architecture, which uses a pair of RNNs in an encoder-decoder pipeline to solve a variety of NLP tasks.
Chapter 6, Transformers, covers transformers, a deep learning architecture that has revolutionized the traditional natural language processing field. We start by reviewing the key intuitions behind the architecture and various categories of transformers, together with a deep dive into the most popular models. Then, we focus on implementations both based on the vanilla architecture and on popular libraries, such as Hugging Face and TensorFlow Hub. After that, we briefly discuss evaluation, optimization, and some of the best practices commonly adopted when using transformers. The last section is devoted to reviewing how transformers can be used to perform computer vision tasks, a totally different domain from NLP. That requires a careful definition of the attention mechanism. In the end, attention is all you need! And at the core of attention, there is nothing more than the cosine similarity between vectors.
Chapter 7, Unsupervised Learning, delves into unsupervised learning models. It will cover techniques required for clustering and dimensionality reduction like PCA, k-means, and self-organized maps. It will go into the details of Boltzmann machines and their implementation using TensorFlow. The concepts covered will be extended to build Restricted Boltzmann Machines (RBMs).
Chapter 8, Autoencoders, describes autoencoders, a class of neural networks that attempt to recreate the input as its target. It will cover different varieties of autoencoders like sparse autoencoders, convolutional autoencoders, and denoising autoencoders. The chapter will train a denoising autoencoder to remove noise from input images. It will demonstrate how autoencoders can be used to create MNIST digits. It will also cover the steps involved in building an LSTM autoencoder to generate sentence vectors. Finally, we will learn how to build a variational autoencoder to generate images.
Chapter 9, Generative Models, focuses on Generative Adversarial Networks (GANs). We start with the first proposed GAN model and use it to forge MNIST characters. The chapter shows you how to use deep convolutional GANs to create celebrity images. The chapter discusses the various GAN architectures like SRGAN, InfoGAN, and CycleGAN. The chapter covers a range of cool GAN applications. Finally, the chapter concludes with a TensorFlow implementation of CycleGAN to convert winter-summer images.
Chapter 10, Self-Supervised Learning, provides an overview of various strategies used for self-supervised learning in computer vision, audio, and natural language processing. It covers self-prediction through strategies such as autoregressive generation, masked generation, relationship prediction, and hybrids of these approaches. It also covers contrastive learning, a popular technique for self-supervised learning, and its application to various pretext tasks in various application domains.
Chapter 11, Reinforcement Learning, focuses on reinforcement learning, covering the Q-learning algorithm and the Bellman equation. The chapter covers discounted rewards, exploration and exploitation, and discount factors. It explains policy-based and model-based reinforcement learning. We will build a Deep Q-learning Network (DQN) to play an Atari game. And finally, we learn how to train agents using the policy gradient algorithm.
Chapter 12, Probabilistic TensorFlow, introduces TensorFlow Probability, the library built over TensorFlow to perform probabilistic reasoning and statistical analysis. The chapter demonstrates how to use TensorFlow Probability to generate synthetic data. We will build Bayesian networks and perform inference. The chapter also introduces the concept of uncertainty, aleatory and epistemic, and how to calculate the uncertainty of your trained models.
Chapter 13, An Introduction to AutoML, introduces AutoML, whose goal is to enable domain experts who are unfamiliar with machine learning technologies to use ML techniques easily. We will go through a practical exercise using Google Cloud Platform and do quite a bit of hands-on work after briefly discussing the fundamentals. The chapter covers automatic data preparation, automatic feature engineering, and automatic model generation. Then, we introduce AutoKeras and Google Cloud AutoML with its multiple solutions for table, vision, text, translation, and video processing.
Chapter 14, The Math Behind Deep Learning, covers the math behind deep learning. This topic is quite advanced and not necessarily required for practitioners. However, it is recommended reading to understand what is going on “under the hood” when we play with neural networks. We start with a historical introduction, and then we will review the high school concept of derivatives and gradients and introduce the gradient descent and backpropagation algorithms commonly used to optimize deep learning networks.
Chapter 15, Tensor Processing Unit, discusses TPUs. TPUs are very special ASIC chips developed at Google for executing neural network mathematical operations in an ultra-fast manner. The core of the computation is a systolic multiplier that computes multiple dot products (row * column) in parallel, thus accelerating the computation of basic deep learning operations. Think of a TPU as a special-purpose co-processor for deep learning that is focused on matrix or tensor operations. We will review the four generations of TPUs so far, plus an additional Edge TPU for IoT.
Chapter 16, Other Useful Deep Learning Libraries, introduces other deep learning frameworks. We will explore Hugging Face, OpenAI’s GPT3, and DALL-E 2. The chapter introduces another very popular deep learning framework, PyTorch. We also cover H2O.ai and its AutoML module. The chapter also briefly discusses the ONNX open-source format for deep learning models.
Chapter 17, Graph Neural Networks, introduces graphs and graph machine learning, with particular emphasis on graph neural networks and the popular Deep Graph Library (DGL). We describe the theory behind various commonly used graph layers used in GNNs (and available in DGL) and provide examples of GNNs used for node classification, link prediction, and graph classification. We also show how to work with your own graph dataset and customize graph layers to create novel GNN architectures. We then cover more cutting-edge advances in the field of Graph ML, such as heterogeneous graphs and temporal graphs.
Chapter 18, Machine Learning Best Practices, focuses on strategies and practices to follow to get the best model in training and production. The chapter discusses the best practices from two different perspectives: the best practices for the data and the best practices with respect to models.
Chapter 19, TensorFlow 2 Ecosystem, lays out the different components of the TensorFlow ecosystem. We introduce TensorFlow Hub, a repository for pretrained deep learning models. The chapter talks about TensorFlow Datasets – a collection of ready-to-use datasets. We will also talk about TensorFlow Lite and TensorFlow JS – the framework for mobile and embedded systems and the web. Finally, the chapter talks about federated learning, a decentralized machine learning framework.
Chapter 20, Advanced Convolutional Neural Networks, shows more advanced uses for convolutional neural networks (CNNs). We will explore how CNNs can be applied within the areas of computer vision, video, textual documents, audio, and music. We’ll conclude with a section summarizing convolution operations.
Download the example code files
The code bundle for the book is hosted on GitHub at https://packt.link/dltf. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!
Download the color images
We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://static.packt-cdn.com/downloads/9781803232911_ColorImages.pdf.
Conventions used
There are a number of text conventions used throughout this book.
CodeInText: Indicates code words in the text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. For example: “Each neuron can be initialized with specific weights via the 'kernel_initializer' parameter.”
A block of code is set as follows:
# Build the model.
model = tf.keras.models.Sequential()
model.add(keras.layers.Dense(NB_CLASSES,
input_shape=(RESHAPED,),
name='dense_layer',
activation='softmax'))
When we wish to draw your attention to a particular part of a code block, the relevant lines or items are highlighted:
# Build the model.
model = tf.keras.models.Sequential()
model.add(keras.layers.Dense(NB_CLASSES,
input_shape=(RESHAPED,),
name='dense_layer',
activation='softmax'))
Any command-line input or output is written as follows:
pip install gym
Bold: Indicates a new term, an important word, or words that you see on the screen. For instance, words in menus or dialog boxes appear in the text like this. For example: “A Deep Convolutional Neural Network (DCNN) consists of many neural network layers.”
Warnings or important notes appear like this.
Tips and tricks appear like this.
