Visualizing gradients tutorial (issue #3186) #3389
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/tutorials/3389
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit b8275ab with merge base ab2aafd ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
github-actions
bot
added
advanced
docathon-h1-2025
|
Generally seems to be headed in the right direction in terms of tone and organization from my perspective. |
There was a problem hiding this comment.
Thanks for the working on this tutorial. Overall I'd say though that this section (prior to the actual visualizing gradients part) can be much shorter.
By the end of this tutorial, you will be able to:
Differentiate between leaf and non-leaf tensors
have a diagram from https://github.com/szagoruyko/pytorchviz, point to the leafs
Know when to use\
retain_gradvs. ``require_grad`
"use requires_grad for leaf, use retain_grad for non-leaf"
|
Thank you for the comments, they were really helpful. Let me know if you think the first section is still too long. Concerning the "visualizing gradients" section with an actual example, I'm not sure if I'm going about retaining the gradients for intermediate tensors correctly. My thought process was to use a forward hook, call Initially I tried using a backward pass hook like RuntimeError: Output 0 of BackwardHookFunctionBackward is a view and is being modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is forbidden. You can fix this by cloning the output of the custom Function. I know that I can plot the gradients for the parameters by just looping through the If anyone sees a problem with my method let me know. The current state of the code isn't doing what I expected so I still have to debug it. EDIT 1: I stumbled upon this issue. Perhaps it's better to switch to using tensor hooks as suggested by alban, instead of storing the outputs through a forward pass and then later accessing their .grad EDIT 2: I decided to not use ResNet but instead a simplified fully connected network as explained in the BatchNorm paper. It is purely for educative purposes, but it actually shows the results I was expecting. With the ResNet implementation, I believe that the residual connections and ReLU non-linearity are muddying the negative effect on the gradients if they don't have BatchNorm. I'll push an updated PR sometime today. |
Still a work in progress, but I significantly reduced the first section and added some helpful images for the computational graph. I also added links for most terms. The WIP section with ResNet I still have to debug. I'm not sure my method for retaining the intermediate gradients is valid. See discussion on pull request.
Instead of using resnet as the example for visualization of the gradients, I decided to use a simple fully-connected network with and without batchnorm. It is a contrived model, but the importance is on illustration of the gradients, not so much on which model to apply it for. I also wanted the positive effect of batch normalization to be clearly shown, and this was not the case with PyTorch's base resnet model.
j-silv
force-pushed
the
3186-gradient-tutorial
branch
from
0b9f56a to
cc1aa32
Compare
j-silv
changed the title
|
@soulitzer what are your thoughts on these new updates? |
| # Because we are using a ``nn.Module`` instead of individual tensors for | ||
| # our forward pass, we need another method to access the intermediate | ||
| # gradients. This is done by `registering a | ||
| # hook <https://www.digitalocean.com/community/tutorials/pytorch-hooks-gradient-clipping-debugging>`__. |
There was a problem hiding this comment.
Cool tutorial, but not totally accurate in its explanations sometimes
Maybe link https://docs.pytorch.org/docs/stable/notes/autograd.html#backward-hooks-execution instead?
|
|
||
|
|
||
| ###################################################################### | ||
| # Because we are using a ``nn.Module`` instead of individual tensors for |
There was a problem hiding this comment.
-> Because we wrapped up the logic and state of our model in a nn.Module
| ###################################################################### | ||
| # Because we are using a ``nn.Module`` instead of individual tensors for | ||
| # our forward pass, we need another method to access the intermediate | ||
| # gradients. This is done by `registering a |
There was a problem hiding this comment.
This is only if we want to avoid modifying nn.Module code directly. I mean we can always directly just attach hooks in forward.
| # Note that using backward pass hooks to probe an intermediate nodes gradient is preferred over using `retain_grad()`. | ||
| # It avoids the memory retention overhead if gradients aren't needed after backpropagation. | ||
| # It also lets you modify and/or clamp gradients during the backward pass, so they don't vanish or explode. | ||
| # However, if in-place operations are performed, you cannot use the backward pass hook |
There was a problem hiding this comment.
This only applies to module backward hooks and not to backward hooks attached tensors.
| # .. warning:: | ||
| # | ||
| # Note that using backward pass hooks to probe an intermediate nodes gradient is preferred over using `retain_grad()`. | ||
| # It avoids the memory retention overhead if gradients aren't needed after backpropagation. |
There was a problem hiding this comment.
Well you'd just have to clear the .grads yourself
There was a problem hiding this comment.
Although I guess in order to access the .grads you'd need to keep the intermediate forward outputs alive...
That seems pretty bad. Maybe we shouldn't use retains_grad for this tutorial LOL (we could say we're just using retains_grad for illustration purposes, but people probably want something pasteable they can use for their own models)
There was a problem hiding this comment.
An alternative way (which seems to be PyTorch's official recommendation) is to attach a forward hook to each module and then during the backward pass, attach a backward tensor hook to the output.
As far as I can tell, this is the only other sure-way to access intermediate gradients without modifying a nn.Module object. This is because any nn.Module with in-place operations will not allow any backward pass hook at the module level.
If we use @albanD 's suggestion though, is that better than keeping the intermediate forward outputs alive with retain_grad()? Is it just as inefficient?
There was a problem hiding this comment.
I think we should use Alban's suggestion here. It should be more efficient than keeping the intermediates alive for sure!
| # | ||
| # .. warning:: | ||
| # | ||
| # Note that using backward pass hooks to probe an intermediate nodes gradient is preferred over using `retain_grad()`. |
There was a problem hiding this comment.
| # Note that using backward pass hooks to probe an intermediate nodes gradient is preferred over using `retain_grad()`. | |
| # Note that using backward pass hooks to probe an intermediate node's gradient is preferred over using `retain_grad()`. |
| # | ||
| # In summary, ``requires_grad`` tells autograd which tensors need to have | ||
| # their gradients calculated for backpropagation to work. This is | ||
| # different from which gradients have to be stored inside the tensor, |
There was a problem hiding this comment.
-> different from which tensor's gradients field is actually populated
|
|
||
| When training neural networks with PyTorch, it’s possible to ignore some | ||
| of the library’s internal mechanisms. For example, running | ||
| backpropagation requires a simple call to ``backward()``. This tutorial |
There was a problem hiding this comment.
Cool intro , but I think we should drop and just begin with what the tutorial does
When training neural networks with PyTorch, it’s possible to ignore some
of the library’s internal mechanisms. For example, running
backpropagation requires a simple call tobackward().
| When training neural networks with PyTorch, it’s possible to ignore some | ||
| of the library’s internal mechanisms. For example, running | ||
| backpropagation requires a simple call to ``backward()``. This tutorial | ||
| dives into how those gradients are calculated and stored in two |
There was a problem hiding this comment.
Maybe something more like "This tutorial explains the subtleties of requires_grad, retains_grad, leaf and non-leaf using a simple example..."
since I feel like that is the highlight of the tutorial rather than "dives into how those gradients are calculated"
| of the library’s internal mechanisms. For example, running | ||
| backpropagation requires a simple call to ``backward()``. This tutorial | ||
| dives into how those gradients are calculated and stored in two | ||
| different kinds of PyTorch tensors: leaf vs. non-leaf. It will also |
There was a problem hiding this comment.
"This tutorial will then use the retains_grad API to extract and visualize gradients across the a networks layers. This can be useful to debug issues like..."
Transition by explicitly mentioning retains_grad since part 1 explains how retains_grad works.
| debug issues that occur during training such as `vanishing or exploding | ||
| gradients <https://arxiv.org/abs/1211.5063>`__. | ||
|
|
||
| By the end of this tutorial, you will be able to: |
There was a problem hiding this comment.
This section can be removed if you explicitly mention what we want to highlight in the previous section.
| ###################################################################### | ||
| # The distinction between leaf and non-leaf determines whether the | ||
| # tensor’s gradient will be stored in the ``grad`` property after the | ||
| # backward pass, and thus be usable for gradient descent optimization. |
There was a problem hiding this comment.
| # backward pass, and thus be usable for gradient descent optimization. | |
| # backward pass, and thus be usable for gradient descent. |
Fixes #3186
Description
Add draft for visualizing gradients tutorial. Link is here but the content is old and the files need to be re-built.
Checklist