add micro batches

[lukas-blecher]

Proposed in issue #147

[lukas-blecher]

Maybe scale all gradients by args.micro_batchsize/args.batchsize because right now the gradients are summed over for each mirco batch, resulting in a larger gradient norm on average.
But I've tested it out on a toy model and this constant factor did not hinder convergence.
This can also be compensated by choosing other betas and initial learning rate in case of adam optimizer.

Still might be better for consistency.
Add

for p in model.parameters():
    p.grad=p.grad*microbatch/args.batchsize

before

LaTeX-OCR/pix2tex/train.py

Line 57 in 4c94f3d

opt.step()

[TITC]

As always, learned from your code and comment.

I found some code on some websites and they do not directly average gradient but average the loss before backward propagation.

I think they come up to same result. Although it named gradient descent but in fact it's Directional derivative. Because the direction is fixed when the architecture is determined.

I not sure if this is the correct understanding, I want listening your opinion. @lukas-blecher

[lukas-blecher]

Yes, I think you're right it is equivalent.
Scaling the loss would be computationally more efficient