I got an idea that can running clustering on the dataset group similar images together. To which the model said that its not possible directly but using [[autoencoders]] and [[Pretrained CNN]] So basically what the model said was that we need more sophisticated feature extraction. Once we have feature extraction we can cluster them together.

I continued running the code on mnist dataset. I learned that the notebook took an input later, a hidden later and an output layer. Then they defined a Nn.Neuralnetwork module. Using nn.Linear, we define the input dimension and the output dimension. we then define all the layrs here.

Then we have a forwared method which basically passes in the neurons from the beggning of input to the final output.

Loss Function

What i initially missed here was the loss function. so for the mnist classifier we use cross entropy loss.

– 06/11 – My next experiment is to use multiple layers in the neural network and see the performance difference. That begs the question what performance factors do we look for here ? may be validation accuracy. I am observing they are doing dropout after each layer i wonder why is that

claude mentioned that one of the reason is [[vanishing gradients]]. Since the since the network is deep, the gradients become smaller and smaller as it travels back. But they fixed that problem with relu. And in this case i did use Relu. So thats not the problem

data with 1 hidden layer

  epoch    train_loss    valid_acc    valid_loss     dur
-------  ------------  -----------  ------------  ------
      1        0.8387       0.8800        0.4174  0.8480
      2        0.4332       0.9103        0.3133  0.8536
      3        0.3612       0.9233        0.2684  1.0073
      4        0.3233       0.9309        0.2317  1.0730
      5        0.2938       0.9353        0.2173  0.9843
      6        0.2738       0.9390        0.2039  0.8702
      7        0.2600       0.9454        0.1868  0.8160
      8        0.2427       0.9484        0.1757  0.8462
      9        0.2362       0.9503        0.1683  0.8483
     10        0.2226       0.9512        0.1621  0.8217
     11        0.2184       0.9529        0.1565  0.8385
     12        0.2090       0.9541        0.1508  0.8643
     13        0.2067       0.9570        0.1446  0.8459
     14        0.1978       0.9570        0.1412  0.8514
     15        0.1923       0.9582        0.1392  0.8638
     16        0.1889       0.9582        0.1342  0.8442
     17        0.1855       0.9612        0.1297  1.1141
     18        0.1786       0.9613        0.1266  1.1765
     19        0.1728       0.9615        0.1250  0.9175
     20        0.1698       0.9613        0.1248  0.8558


Data with 3 hidden layers 

 epoch    train_loss    valid_acc    valid_loss     dur
-------  ------------  -----------  ------------  ------
      1        1.9074       0.6777        1.0223  1.5400
      2        1.1250       0.7968        0.6291  1.0226
      3        0.8954       0.8401        0.5521  1.0114
      4        0.7705       0.8960        0.4075  1.0047
      5        0.6980       0.8668        0.4993  1.0224
      6        0.6402       0.8988        0.3935  1.1413
      7        0.5999       0.9228        0.3063  1.3173
      8        0.5682       0.9050        0.3568  1.0877
      9        0.5445       0.9253        0.2782  1.0162
     10        0.5260       0.9330        0.2725  0.9848
     11        0.5055       0.9410        0.2425  1.0241
     12        0.4975       0.9353        0.2522  1.0082
     13        0.4804       0.9438        0.2295  0.9854
     14        0.4697       0.9445        0.2256  0.9934
     15        0.4527       0.9483        0.2123  1.4027
     16        0.4555       0.9460        0.2169  1.3944
     17        0.4455       0.9129        0.3389  1.2980
     18        0.4383       0.9325        0.2741  1.3400
     19        0.4308       0.9499        0.2062  1.0041
     20        0.4261       0.9490        0.2131  1.0330

There is a drastic improvement in model training and performance when using convolutional neural network. 234456