Auto differentiation

Auto differentiation is also known as algorithmic differentiation, which is an automatic way of numerically computing the derivatives of a function. It is helpful for computing gradients, Jacobians, and Hessians for use in applications such as numerical optimization. Backpropagation algorithm is an implementation of the reverse mode of automatic differentiation for calculating the gradient.

In the following example, using the mnist dataset, we calculate the loss using one of the loss functions. The question is: how do we fit the model to the data?

We can use tf.train.Optimizer and create an optimizer. tf.train.Optimizer.minimize(loss, var_list) adds an optimization operation to the computational graph and automatic differentiation computes gradients without user input:

import TensorFlow  as tf

# get mnist dataset
from TensorFlow .examples.tutorials.mnist import input_data
data = input_data.read_data_sets("MNIST_data/", one_hot=True)

# x represents image with 784 values...