- For DDP
find_unused_parameters=False
- For optimization while training
Optimizer.zero_grad(set_to_none=False)
- CUDNN flags:
torch.backend.cudnn.enabled=True
torch.backend.cudnn.deterministic=False
torch.backend.cudnn.benchmark=True
Created
October 19, 2022 15:41
-
-
Save salmedina/32c9ec91a5bb1288df03ec73fa2bbf95 to your computer and use it in GitHub Desktop.
PyTorch Training Hacks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment