You will require to have basic knowledge of Python 3, TensorFlow, Keras and OpenCV.

The code files of this chapter can be found on GitHub:
https://github.com/PacktPublishing/Intelligent-Projects-using-Python/tree/master/Chapter02

Check out the following video to see the code in action:
http://bit.ly/2t6LLyB

In a traditional machine learning paradigm (see Figure 2.1), every use case or task is modeled independently, based on the data at hand. In transfer learning, we use the knowledge gained from a particular task (in the form of architecture and model parameters) to solve a different (but related) task, as illustrated in the following diagram:

Training an artificial neural network from scratch is a difficult...

In this chapter, using transfer learning, we are going to build a model to detect diabetic retinopathy in the human eye. Diabetic retinopathy is generally found in diabetic patients, where high blood sugar levels cause damage to the blood vessels in the retina. The following image shows a normal retina on the left, and one with diabetic retinopathy on the right:

In healthcare, diabetic retinopathy detection is generally a manual process that involves a trained physician examining color fundus retina images. This introduces a delay in the process of diagnosis, often leading to delayed treatment. As a part of our project, we are going to build a robust artificial intelligence system that can take the color fundus images of the retina and classify the...

The dataset for the building the Diabetic Retinopathy detection application is obtained from Kaggle and can be downloaded from following the link: https://www.kaggle.com/c/ classroom-diabetic-retinopathy-detection-competition/data.

Both the training and the holdout test datasets are present within the train_dataset.zip file, which is available at the preceding link.

We will use the labeled training data to build the model through cross-validation. We will evaluate the model on the holdout dataset.

Since we are dealing with class prediction, accuracy will be a useful validation metric. Accuracy is defined as follows:

Here, c is the number of correctly classified samples, and N is the total number of evaluated samples.

We will also use the quadratic weighted kappa statistics to determine the quality of the model, and to have a benchmark as to how...

The data for this use case has five classes, pertaining to no diabetic retinopathy, mild diabetic retinopathy, moderate diabetic retinopathy, severe diabetic retinopathy, and proliferative diabetic retinopathy. Hence, we can treat this as a categorical classification problem. For our categorical classification problem, the output labels need to be one-hot encoded, as shown here:

• No diabetic retinopathy: [1 0 0 0 0]^T
• Mild diabetic retinopathy: [0 1 0 0 0]^T
• Moderate diabetic retinopathy: [0 0 1 0 0]^T
• Severe diabetic retinopathy: [0 0 0 1 0]^T
• Proliferative diabetic retinopathy: [0 0 0 0 1]^T

Softmax would be the best activation function for presenting the probability of the different classes in the output layer, while the sum of the categorical cross-entropy loss of each of the data points would be the best loss to optimize. For a single data point...

Class imbalance is a major problem when it comes to classification. The following diagram depicts the class densities of the five severity classes:

As we can see from the preceding chart, nearly 73% of the training data belongs to Class 0, which stands for no diabetic retinopathy condition. So if we happen to label all data points as Class 0, then we would have 73% percent accuracy. This is not desirable in patient heath conditions. We would rather have a test say a patient has a certain heath condition when it doesn't (false positive) than have a test that misses detecting a certain heath condition when it does (false negative). A 73% accuracy may mean nothing if the model learns to classify all points as belonging to Class 0.

Detecting the higher severity classes are more important...

The images for the different classes will be stored in different folders, so it will be easy to label their classes. We will read the images using Opencv functions, and will resize them to different dimensions, such as 224 x 224 x 3. We'll subtract the mean pixel intensity channel-wise from each of the images, based on the ImageNet dataset. This means subtraction will bring the diabetic retinopathy images to the same intensity range as that of the processed ImageNet images, on which the pre-trained models are trained. Once each image has been prepossessed, they will be stored in a numpy array. The image preprocessing functions can be defined as follows:

def get_im_cv2(path,dim=224):
     img = cv2.imread(path)
     resized = cv2.resize(img, (dim,dim), cv2.INTER_LINEAR)
     return resized
 
 def pre_process(img):
     img[:,:,0] =     img[:,:,0] - 103...

We will use the keras ImageDataGenerator to generate additional data, using affine transformation on the image pixel coordinates. The transformations that we will primarily use are rotation, translation, and scaling. If the pixel spatial coordinate is defined by x = [x₁x₂]^T ∈ R², then the new coordinate of the pixel can be given by the following:

Here, M = R^2x2 is the affine transformation matrix, and b = [b₁ b₂]^T ∈ R² is a translation vector.

The term b₁ specifies the translation along one of the spatial directions, while b₂ provides the translation along the other spatial dimension.

These transformations are required, because neural networks are not, in general, translational invariant, rotational invariant, or scale invariant. Pooling operations do provide some translational invariance, but it is generally...

We will now experiment with the pre-trained ResNet50, InceptionV3, and VGG16 networks, and find out which one gives the best results. Each of the pre-trained models' weights are based on ImageNet. I have provided the links to the original papers for the ResNet, InceptionV3, and VGG16 architectures, for reference. Readers are advised to go over these papers, to get an in-depth understanding of these architectures and the subtle differences between them.

The VGG paper link is as follows:

• Title: Very Deep Convolutional Networks for Large-Scale Image Recognition
• Link: https://arxiv.org/abs/1409.1556

The ResNet paper link is as follows:

• Title: Deep Residual Learning for Image Recognition
• Link: https://arxiv.org/abs/1512.03385

The InceptionV3 paper link is as follows:

• Title: Rethinking the Inception Architecture for Computer Vision
• Link: https://arxiv...

The Adam optimizer (adaptive moment estimator) is used in training that implements an advanced version of stochastic gradient descent. The Adam optimizer takes care of the curvature in the cost function, and at the same time, it uses momentum to ensure steady progress toward a good local minima. For the problem at hand, since we are using transfer learning and want to use as many of the previously learned features from the pre-trained network as possible, we will use a small initial learning rate of 0.00001. This will ensure that the network doesn't lose the useful features learned by the pre-trained networks, and fine-tunes to an optimal point less aggressively, based on the new data for the problem at hand. The Adam optimizer can be defined as follows:

adam = optimizers.Adam(lr=0.00001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay...

Since the training dataset is small, we will perform five-fold cross-validation, to get a better sense of the model's ability to generalize to new data. We will also use all five of the models built in the different folds of cross-validation in training, for inference. The probability of a test data point belonging to a class label would be the average probability prediction of all five models, which is represented as follows:

Since the aim is to predict the actual classes and not the probability, we would select the class that has the maximum probability. This methodology works when we are working with a classification-based network and cost function. If we are treating the problem as a regression problem, then there are a few alterations to the process, which we will discuss later on.

It is always a good practice to save the model when the validation score chosen for evaluation improves. For our project, we will be tracking the validation log loss, and will save the model as the validation score improves over the different epochs. This way, after the training, we will save the model weights that provided the best validation score, and not the final model weights from when we stopped the training. The training will continue until the maximum number of epochs defined for the training is reached, or until the validation log loss hasn't reduced for 10 epochs in a row. We will also reduce the learning rate when the validation log loss doesn't improve for 3 epochs. The following code block can be used to perform the learning rate reduction and checkpoint operation:

reduce_lr = keras.callbacks.ReduceLROnPlateau...

The following Python code block shows an end-to-end implementation of the training process. It consists of all of the functional blocks that were discussed in the preceding sections. Let's start by calling all of the Python packages that are required, as follows:

import numpy as np
np.random.seed(1000)

import os
import glob
import cv2
import datetime
import pandas as pd
import time
import warnings
warnings.filterwarnings("ignore")
from sklearn.model_selection import KFold
from sklearn.metrics import cohen_kappa_score
from keras.models import Sequential,Model
from keras.layers.core import Dense, Dropout, Flatten
from keras.layers.convolutional import Convolution2D, MaxPooling2D, ZeroPadding2D
from keras.layers import GlobalMaxPooling2D,GlobalAveragePooling2D
from keras.optimizers import SGD
from keras.callbacks import EarlyStopping
from...

The categorical classification is performed by using all three of the neural network architectures: VGG16, ResNet50, and InceptionV3. The best results were obtained using the InceptionV3 version of the transfer learning network for this diabetic retinopathy use case. In case of categorical classification we are just converting the class with the maximum predicted class probability as the predicted severity label. However since the classes in the problem has an ordinal sense one of the ways in which we can utilize the softmax probabilities is to take the expectation of the class severity with respect to the softmax probabilities and come up with an expected score as follows:

We can rank order the scores and determine three thresholds to determine which class the image belongs to. These thresholds can be chosen by training a secondary...

The following code can be used to carry out inference on the unlabeled test data:

import keras
import numpy as np
import pandas as pd
import cv2
import os
import time
from sklearn.externals import joblib
import argparse

# Read the Image and resize to the suitable dimension size
def get_im_cv2(path,dim=224):
    img = cv2.imread(path)
    resized = cv2.resize(img, (dim,dim), cv2.INTER_LINEAR)
    return resized

# Pre Process the Images based on the ImageNet pre-trained model Image transformation
def pre_process(img):
    img[:,:,0] = img[:,:,0] - 103.939
    img[:,:,1] = img[:,:,0] - 116.779
    img[:,:,2] = img[:,:,0] - 123.68
    return img

   
# Function to build test input data
def read_data_test(path,dim):
    test_X = [] 
    test_files = []
    file_list = os.listdir(path) 
    for f in file_list:
        img = get_im_cv2(path + '/' + f)
        img = pre_process...

One of the things that we discussed in the Formulating the loss function section, was the fact that the class labels are not independent categorical classes, but do have an ordinal sense with the increasing severity of the diabetic retinopathy condition. Hence, it would be worthwhile to perform regression through the defined transfer learning networks, instead of classification, and see how the results turned out. The only thing that we would need to change would be the output unit, from a softmax to a linear unit. We will, in fact, change it to be a ReLU, since we want to avoid negative scores. The following code block shows the InceptionV3 version of the regression network:

def inception_pseudo(dim=224,freeze_layers=30,full_freeze='N'):
    model = InceptionV3(weights='imagenet',include_top=False...

Keras has a good batch generator named keras.utils.sequence() that helps you customize batch creation with great flexibility. In fact, with keras.utils.sequence() one can design the whole epoch pipeline. We are going to use this utility in this regression problem to get accustomed to this utility. For the transfer learning problem we can design a generator class using keras.utils.sequence() as follows:

class DataGenerator(keras.utils.Sequence):
    'Generates data for Keras'
    def __init__(self,files,labels,batch_size=32,n_classes=5,dim=(224,224,3),shuffle=True):
        'Initialization'
        self.labels = labels
        self.files = files
        self.batch_size = batch_size
        self.n_classes = n_classes
        self.dim = dim
        self.shuffle = shuffle
        self.on_epoch_end()

    def __len__(self):
  ...

In this chapter, we went over the practical aspects of transfer learning, to solve a real-world problem in the healthcare sector. The readers are expected to further build upon these concepts by trying to customize these examples wherever possible.

The accuracy and the kappa score that we achieved through both the classification and the regression-based neural networks are good enough for production implementation. In Chapter 3, Neural Machine Translation, we will work on implementing intelligent machine translation systems, which is a much more advanced topic than what was presented in this chapter. I look forward to your participation.