Created
March 13, 2020 10:52
-
-
Save nofreewill42/d7f8f5301760ea98fd62a86604d7c544 to your computer and use it in GitHub Desktop.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I have found the source of the accuracy drop present in cells 30-34.
With the 5x5 kernel size, we are adding 5x5=25 numbers together.
With the 9x9 kernel size, we are adding 9x9=81 numbers together.
The numbers in both case have the same mean and std, so the outputs of the 9x9 kernel will be 81/25 times the outputs of the 5x5 kernel.
This is important because there are biases in the network and ReLUs can go from inactive to active and vice versa.
So, all we have to do is to divide the weights of the 9x9 kernel with (81/25).
Here is the corrected version of the 9x9 kernel without any "fine-tuning". (This was a fresh training, without this "scaling", I got the same "accuracy drop" as before.)