Non-linear Boundary in Deep Neural Network

In the perceptron model, we used a linear model to classify two regions of data. Realistic data is much more complex and not always classify by a straight line. For this purpose, we need a non-linear boundary to separate our data. Perceptron model is work on the most basic form of a neural network, but for realistic data classification, we used Deep Neural Network.

When our model is unable to represent a set of data, we use a non-linear model instead of it. The non-linear model is used in the following situation

In the above image, there is a curve which perfectly classifies our data but how we can obtain this curve. For this, we combine two perceptrons into a third one. It's quite typical to understand so that for better understanding, we take two linear models and combine them to form a single non-linear model.

It is clear from the above pictures that both models are unable to classify our data. Following are some steps which are used to form a non-linear model from two linear models:

Step 1:

Let start by combining each linear model to form a non-linear model. If we have two linear models, then by combining them the resulted model will look like

The output model is a linear combination of the two other models.

Step 2:

Now, what we have to do, we will treat both the linear model as an input node which contains some linear equation. We denote our first model as x1 and second as x2.

Step 3:

In our next step, we will multiply our model with some weight such as w1 and w2, and we also consider bias so that we will also treat bias value as a node.

Step 4:

Now, everything is added to obtain a linear combination. For this purpose, we will apply the sigmoid activation function, which gives us the expected curve.

Step 5:

We will mathematically multiply all the nodes with its weight value such as w1=0.4, w2=1, and b=0.5 and apply the sigmoid then the resulting curve will be as follows:

Step 6:

In the second linear model x2, if we will take the weight value 3, the resulting model will give us an unexpected curve, and it looks like a

From step 5 and step 6 it is clear that the model which is created with weight value 1.5 and 1, classify our data best rather than the model which is created with weight value 1.5 and 3. Weight defined the non-linear boundary of the non-linear model.

The process of combining two linear models to form a non-linear model is not so simple. It is quite essential to understand the structure of the Neural Network to implement the non-linear boundary of deep neural network.