Cost Function

My intuition is that the single most import thing in the neural net trainig is the cost function. You will say, why?. Beacause if you know that we calculate the gradients of cost function with respect to each parameters. And that’s why we have different cost functions for different usecase. for example Binary cross entropy is for binary classification(0,1), Cross Entropy for multiclass classification.

The idea is to design a costfunction in such a way it

  1. Binary Cross Entropy

    nn.BCE(Y, yh)

    This function computes the Binary Cross Entropy cost function.

    \[J() = - Y log(\hat{Y}) + (1-Y) log(1-\hat{Y})\]

  2. Cross Entropy

    nn.CE(s, y)

    Cross entropy loss function for softmax

    \[ \begin{align}\begin{aligned}L(Y,S) = \sum_{i=1}^{C} Y_i.log(S_i)\\J(W,b) = \sum_{j=1}^{m}L(Y^j, S^j)\end{aligned}\end{align} \]

    Here \(L\) is loss function, calculates how well our model is performing on single training example for all classes \(C\). And \(J\) is cost function that calculates loss for all the samples in training examples