Regularization

1. Dropout

   nn.dropout(A, keep_prob=0.8)

   Dropout helps regularize the network by randomly shutting down the units. The intuition is that, any single neuron cannot rely on single feature or input, because, it any time it can go off, and that’s why it will be reluctunt to give more weight to any single input or feature.

   Specifically here I have installed Inverted Dropout.

   With Dropout the feedforward operation becames.

   \[ \begin{align}\begin{aligned}D \sim Bernouli(keepprob)\\Z = W.X + b\\A = g(Z)\\A = \frac{A*D}{keep_prob}\end{aligned}\end{align} \]