Question

Derive the Backpropagation algorithm (only for the output nodes of a multi-layer perceptron) if the activation function is: phi(v) = e^-(v/sigma)^2 where sigma is a parameter to be learned for each neuron? Simplify the equations as much as possible.

Answer 1

Answer:

For the derivation, lets assume certain conventions:

1. Subscript k will represent an output node. Subscript j will represent a hidden node.

2. Weight from a hidden node to an output node will be represented as $w_{hj}^{k}$ .

3. Weight change rule for hidden to output layer weights will be given by:

$w_{hj}^{k+1}=w_{hj}^{k} + \Delta w_{hj}^{k}$

where $\Delta w_{hj}^{k}$ is the weight change.

4. Activation function or signal will be represented as $\varphi (\nu)$ . In present case, $\varphi (\nu) = e^{-(\frac{\nu}{\sigma})^{2}}$ .

5. $y_{j}^{k}$ will be activation value at any output node.

Derivation of backpropagation algorithm for output layer:

Computation of neoronal signals:

A. For output layer, neuronal signal and activation value at any output node can be calculated as:

1. $y_{j}^{k} = \sum_{h=0}^{q}w_{hj}^{k}\varphi (\nu_{h}^{k}), j = 1,............,p$

2. $\varphi (\nu_{h}^{k}) = e^{-(\frac{\nu_{h}^{k}}{\sigma })^{2}}, j = 1,............,p$

where $w_{0j}^{k}$ will be the biases of output neurons.

Computation of error gradients:

B. Weight gradients of hidden weight gradients:

It can be calculated using chain rule of calculus as shown below:

3. $\frac{\partial \varepsilon _{k}}{\partial w_{hj}^{k}} = \frac{\partial \varepsilon _{k}}{\partial \varphi(y_{j}^{k})}\frac{\partial \varphi(y_{j}^{k})}{\partial y_{j}^{k}}\frac{\partial y_{j}^{k}}{\partial w_{hj}^{k}}$

Individual derivatives in the above equation can be calculated as:

i) $\frac{\partial \varepsilon _{k}}{\partial \varphi(y_{j}^{k})} = -(d_{j}^{k} - \varphi(y_{j}^{k})) = -e_{j}^{k}$

where $d_{j}^{k}$ is the desired output for given node.

ii) $\frac{\partial \varphi(y_{j}^{k})}{\partial y_{j}^{k}} = \varphi(y_{j}^{k})(1-\varphi(y_{j}^{k}))$

iii) $\frac{\partial y_{j}^{k}}{\partial w_{hj}^{k}} = \varphi(z_{h}^{j})$

where $z_{h}^{j}$ is activaion at concerned hidden unit.

Now on replacing values from i), ii) and iii) in 3, we get:

$\frac{\partial \varepsilon _{k}}{\partial w_{hj}^{k}} = -e_{j}^{k}\varphi(y_{j}^{k})(1-\varphi(y_{j}^{k}))\varphi (z_{h}^{k})$

Weight updates for output layer:          

C. For hidden to output layer weights:

$w_{hj}^{k+1}=w_{hj}^{k} + \Delta w_{hj}^{k}$

           $=w_{hj}^{k} + \sigma \left ( -\frac{\partial \varepsilon _{k}}{\partial w_{hj}^{k}} \right )$

           $= w_{hj}^{k} + \sigma e_{j}^{k}\varphi(y_{j}^{k})(1-\varphi(y_{j}^{k}))\varphi (z_{h}^{k})$