Question

Calculate the entropy of each of the following datasets, where + represents a positive instance and -, a negative instance.
D1: + + + + + + + +
D2: + + + + - - - -
D3: + + - - - - - +
D4: + + + - + + + +

Answer 1

This is the formula for entropy where pi is the probability of ith class/instance in total c classes/instance

a)  D1: + + + + + + + +

8+, 0-

p1 = p(+) = 8/8 =1

p2 = p(-) = 0/8 = 0

Entropy= -1*log(1) - 0*log(0) = 0+0=0

b)  D2: + + + + - - - -

4+, 4-

p1 = p(+) = 4/8 =0.5

p2 = p(-) = 4/8 = 0.5

Entropy= -0.5*log(0.5) - 0.5*log(0.5) = -log(0.5) = 1

c)  D3: + + - - - - - +

3+, 5-

p1 = p(+) = 3/8 =0.375

p2 = p(-) = 5/8 = 0.625

Entropy= -0.375*log(0.375) - 0.625*log(0.625) = 0.53+0.42 = 0.95

d)  D4: + + + - + + + +

7+, 1-

p1 = p(+) = 7/8 =0.875

p2 = p(-) = 1/8 = 0.125

Entropy= -0.875*log(0.875) - 0.125*log(0.125) = 0.168+0.375 = 0.543

Answer 2

To calculate the entropy of a dataset, we first need to find the proportion of positive and negative instances in the dataset, denoted by p+ and p-. Then, we can use the formula for entropy:

Entropy = -p+ * log2(p+) - p- * log2(p-)

1. D1: Number of positive instances = 7 Number of negative instances = 0 Total instances = 7 p+ = 7/7 = 1 p- = 0/7 = 0 Entropy = -1 * log2(1) - 0 * log2(0) = -1 * 0 - 0 * undefined = 0

2. D2: Number of positive instances = 4 Number of negative instances = 4 Total instances = 8 p+ = 4/8 = 0.5 p- = 4/8 = 0.5 Entropy = -0.5 * log2(0.5) - 0.5 * log2(0.5) = 1

3. D3: Number of positive instances = 2 Number of negative instances = 5 Total instances = 7 p+ = 2/7 p- = 5/7 Entropy = -2/7 * log2(2/7) - 5/7 * log2(5/7) = 0.863

4. D4: Number of positive instances = 7 Number of negative instances = 1 Total instances = 8 p+ = 7/8 p- = 1/8 Entropy = -7/8 * log2(7/8) - 1/8 * log2(1/8) = 0.543

Therefore, the entropy of D1 is 0, the entropy of D2 is 1, the entropy of D3 is 0.863, and the entropy of D4 is 0.543.