Question

1. Explain the following concepts:

i. What are Linearly Separable Problems? How to make a neural network to solve non-linearly separable problems?

ii. Single perceptron vs Multilayer neural network with backpropagation

iii. Multilayer neural network with backpropagation vs multilayer neural network with deep learning.

iv. List known deep learning algorithms and their features.

Answer 1

`Hey,

Note: Brother if you have any queries related the answer please do comment. I would be very happy to resolve all your queries.

1)

Suppose you want to write an algorithm that decides, based on two parameters, size and price, if an house will sell in the same year it was put on sale or not. So you have 2 inputs, size and price, and one output, will sell or will not sell. Now, when you receive your training sets, it could happen that the output is not accumulated to make our prediction easy (Can you tell me, based on the first graph if X will be an N or S? How about the second graph):

        ^
        |  N S   N
       s|  S X    N
       i|  N     N S
       z|  S  N  S  N
       e|  N S  S N
        +----------->
          price


        ^
        |  S S   N
       s|  X S    N
       i|  S     N N
       z|  S  N  N  N
       e|    N N N
        +----------->
          price

Where:

S-sold,
N-not sold

As you can see in the first graph, you can't really separate the two possible outputs (sold/not sold) by a straight line, no matter how you try there will always be both S and N on the both sides of the line, which means that your algorithm will have a lot of possible lines but no ultimate, correct line to split the 2 outputs (and of course to predict new ones, which is the goal from the very beginning). That's why linearly separable (the second graph) data sets are much easier to predict.

You definitely can’t solve the NLS problem with any of the architectures above. So, the solution we came up with involves a re-envisioning of the problem, which is realized in a change to the architecture. Instead of the traditional perceptron architecture with dimensions-as-inputs and categories-as-outputs, we designed an autoassociator-based classifier:

I’ve color-coded the weights so that it doesn’t look like a tangled mess. Instead of having one output unit per category, the network has one output unit per category, per dimension. The output units are split into two, category-specific, channels, which is why we call it a “Divergent Autoassociator”.

You could also imagine that network as two separate, regular autoassociators, which is formally the same (it just doesn’t look divergent):

The network’s goal is to “reconstruct” everything it sees in the correct category. So if D1=0D1=0 and D2=1D2=1 (i.e., input = [0,1]), and the network is told that belongs to category A, then the goal is for A1=0A1=0 and A2=1A2=1. The units on the opposite-category channel (B1B1 and B2B2) are left alone.

Based on that learning objective, the network needs to learn weights that accurately reconstruct each category’s items on the correct channel. This produces a network which learns how dimensions are correlated within each category, rather than a network which learns how dimensions predict categories.

Note: Brother According to HomeworkLib's policy we are only allowed to answer first part if there are many. So, I request you to post other part as separate posts

Kindly revert for any queries

Thanks.