Linear Regression in TensorFlow

Linear Regression is a machine learning algorithm that is based on supervised learning. It performs a regression function. The regression models a target predictive value based on the independent variable. It is mostly used to detect the relation between variables and forecasts.

Linear regression is a linear model; for example, a model that assumes a linear relationship between an input variable (x) and a single output variable (y). In particular, y can be calculated by a linear combination of input variables (x).

Linear regression is a prevalent statistical method that allows us to learn a function or relation from a set of continuous data. For example, we are given some data point of x and the corresponding, and we need to know the relationship between them, which is called the hypothesis.

In the case of linear regression, the hypothesis is a straight line, that is,

h (x) = wx + b

Where w is a vector called weight, and b is a scalar called Bias. Weight and bias are called parameters of the model.

We need to estimate the value of w and b from the set of data such that the resultant hypothesis produces at least cost 'j,' which has been defined by the below cost function.

Where m is the data points in the particular dataset.

This cost function is called the Mean Squared Error.

For optimization of parameters for which the value of j is minimal, we will use a commonly used optimizer algorithm, called gradient descent. The following is pseudocode for gradient descent:

Repeat until Convergence {
    w = w - ? * ?J/?w
    b = b - ? * ?J/?b
}

Where ? is a hyperparameter which called the learning rate.

Implementation of Linear Regression

We will start to import the necessary libraries in Tensorflow. We will use Numpy with Tensorflow for computation and Matplotlib for plotting purposes.

First, we have to import packages:

import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf

To make the random numbers predicted, we have to define fixed seeds for both Tensorflow and Numpy.

tf.set_random_seed(101)
np.random.seed(101)

Now, we have to generate some random data for training the Linear Regression Model.

# Generating random linear data 
# There will be 50 data points which are ranging from 0 to 50.
x = np.linspace(0, 50, 50) 
y = np.linspace(0, 50, 50) 
  
# Adding noise to the random linear data 
x += np.random.uniform(-4, 4, 50) 
y += np.random.uniform(-4, 4, 50) 
n= len(x) #Number of data points

Let us visualize the training data.

# Plot of Training Data

plt.scatter(x, y) 
plt.xlabel('x') 
plt.xlabel('y') 
plt.title("Training Data") 
plt.show() 

Output

Now, we will start building our model by defining placeholders x and y, so that we feed the training examples x and y into the optimizer while the training process.

X= tf.placeholder("float")
Y= tf.placeholder("float")

Now, we can declare two trainable TensorFlow variables for the bias and Weights initializing them randomly using the method:

np.random.randn().
W= tf.Variable(np.random.randn(), name="W")
B= tf.Variable(np.random,randn(), name="b")

Now we define the hyperparameter of the model, the learning rate and the number of Epochs.

learning_rate= 0 .01
training_epochs= 1000

Now, we will build Hypothesis, Cost Function and Optimizer. We will not manually implement the Gradient Decent Optimizer because it is built inside TensorFlow. After that, we will initialize the variables in the method.

# Hypothesis of the function
y_pred = tf.add(tf.multiply(X, W), b) 
# Mean Square Error function
      cost = tf.reduce_sum(tf.pow(y_pred-Y, 2)) / (2 * n)  
# Gradient Descent Optimizer function
optimizer = tf.train.GradientDescentOptimizer (learning_rate).minimize(cost)
# Global Variables Initializer 
init = tf.global_variables_initializer( )

Now we start the training process inside the TensorFlow Session.

# Starting the Tensorflow Session 
with tf.Session() as sess: 
      
   # Initializing the Variables 
    sess.run(init) 
      
   # Iterating through all the epochs 
    for epoch in range(training_epochs): 
          
    # Feeding each data point into the optimizer according to the Feed Dictionary.
  for (_x, _y) in zip(x, y):  
  sess.run(optimizer, feed_dict = {X : _x, Y : _y})  
 
# Here, we are displaying the result after every 50 epoch  
        if (epoch + 1) % 50 ==0: 
 # Calculating the cost at every epoch. 
 c = sess.run(cost, feed_dict = {X : x, Y : y})  
print("Epoch", (epoch + 1), ": cost =", c, "W =", sess.run(W), "b=", sess.run(b))
 # Store the necessary value which has used outside the Session 
    training_cost = sess.run (cost, feed_dict ={X: x, Y: y})  
    weight = sess.run(W) 
    bias = sess.run(b) 

Output is given below:

Epoch: 50  cost = 5.8868037 W = 0.9951241 b = 1.2381057
Epoch: 100 cost = 5.7912708 W = 0.9981236 b = 1.0914398
Epoch: 150 cost = 5.7119676 W = 1.0008028 b = 0.96044315
Epoch: 200 cost = 5.6459414 W = 1.0031956 b = 0.8434396
Epoch: 250 cost = 5.590798 W = 1.0053328 b = 0.7389358
Epoch: 300 cost = 5.544609 W = 1.007242 b = 0.6455922
Epoch: 350 cost = 5.5057884 W = 1.008947 b = 0.56223
Epoch: 400 cost = 5.473068 W = 1.01047 b = 0.46775345
Epoch: 450 cost = 5.453845 W = 1.0118302 b = 0.42124168
Epoch: 500 cost = 5.421907 W = 1.0130452 b = 0.36183489
Epoch: 550 cost = 5.4019218 W = 1.0141305 b = 0.30877414
Epoch: 600 cost = 5.3848578 W = 1.0150996  b = 0.26138115
Epoch: 650 cost = 5.370247 W = 1.0159653  b = 0.21905092
Epoch: 700 cost = 5.3576995 W = 1.0167387  b = 0.18124212
Epoch: 750 cost = 5.3468934 W = 1.0174294  b = 0.14747245
Epoch: 800 cost = 5.3375574 W = 1.0180461  b = 0.11730932
Epoch: 850 cost = 5.3294765 W = 1.0185971  b = 0.090368526
Epoch: 900 cost = 5.322459 W = 1.0190894  b = 0.0663058
Epoch: 950 cost = 5.3163588 W = 1.0195289  b = 0.044813324
Epoch: 1000 cost = 5.3110332 W = 1.0199218  b = 0.02561669

Now, see the result.

# Calculate the predictions
predictions = weight * x + bias 
print ("Training cost =", training_cost, "Weight =", weight, "bias =", bias, '\n') 

Output

Training cost= 5.3110332 Weight= 1.0199214 bias=0.02561663

Note that in this case, both weight and bias are scalars in order. This is because we have examined only one dependent variable in our training data. If there are m dependent variables in our training dataset, the weight will be a one-dimensional vector while Bias will be a scalar.

Finally, we will plotting our result:

# Plotting the Results below
plt.plot(x, y, 'ro', label ='original data')
plt.plot(x, predictions, label ='Fited line')
plt.title('Linear Regression Result') 
plt.legend() 
plt.show() 

Output