Autograd

Code

import torch

x = torch.tensor(2.0, requires_grad=True)
y = torch.tensor(3.0, requires_grad=True)

# Forward: z = 3x^2 + 2y
z = 3 * x ** 2 + 2 * y

# Backward to populate .grad
z.backward()
print(x.grad, y.grad)  # dz/dx=6x=12, dz/dy=2

# Detach from graph
detached = z.detach()

# No-grad context for inference
with torch.no_grad():
    out = x * 2 + y

# Custom gradient via Function (advanced)
class MyReLU(torch.autograd.Function):
    @staticmethod
    def forward(ctx, inp):
        ctx.save_for_backward(inp)
        return inp.clamp(min=0)
    @staticmethod
    def backward(ctx, grad_out):
        inp, = ctx.saved_tensors
        return grad_out * (inp > 0).float()

Explanation

Tensors with requires_grad=True record operations in a computational graph that backward() traverses to populate .grad. detach() returns a tensor disconnected from the graph, useful for logging or frozen models. Wrapping inference in torch.no_grad() skips graph construction to save memory and time.

More PyTorch Snippets