Question

Explain why conditional entropy used in ID3 learning algorithm is a good heuristics for selecting an attributes.

Answer 1

To understand why conditional entropy used in the ID3 learning algorithm is a good heuristics for selecting attributes. first, let us understand what ID3 learning algorithm is:

ID3 Algorithm: It is the core algorithm to build a decision tree Model. This algorithm performs a greedy search through each possible node/branch in the decision tree. The main concepts used by ID3 to build a decision tree model are:

1. Entropy
2. Information Gain

To build a decision tree from a root node ID3 divides the data into various subsets containing instances of similar values. as say you have a messy room with all your toys, clothes, and books spread all over your room. The most obvious thing you will do to clean the mess is to separate the toys, clothes, and books from the mess and place them into their specified places. Like the same ID3 algorithm clean up the data by partitioning the data in the groups containing instances of similar values.

To find similarity between data ID3 uses the mathematical concept of entropy which gives the similarity values of data.

$Entropy(E)= -p \log_2(p) - q \log_2(q)$

• If the data is completely homogeneous, E=0
• If the data is equally divided into similar groups, E=1

To complete the goal of data cleaning with respect to the target value. ID3 uses the concept of Information Gain which is the measure of the amount of decrease in Entropy value after splitting the data w.r.t the attribute. The attribute showing the maximum Information Gain value is used to split the data.

As the concept of grouping the data according to their similarity w.r.t the target value. To reach the homogeneous subset of data. The concept of Entropy is important and considered as a good heuristic to select an attribute.