W04: Geometry, gradients, and automatic differentiation

Prerequisites and observable goals#

The central question: how does a composed computation change?#

For a scalar function $f:\R\to\R$, the derivative is a scalar local slope. For $f:\R^n\to\R$, the gradient is a vector of partial derivatives. For $F:\R^n\to\R^m$, the Jacobian is an $m\times n$ matrix whose entry $J_{ij}$ is $\partial F_i/\partial x_j$. These are related objects, not interchangeable notations. The linear approximation is

An $n$-component input perturbation is transformed into an $m$-component output perturbation. Check this shape before manipulating a formula. A transpose placed casually can make a matrix product impossible or compute a different map.

The gradient also has a geometric meaning: the differential of a scalar function is $\dd f=\nabla f(x)^T\dd x$ under the usual Euclidean inner product. Changing coordinates or physical units changes the components. A gradient in pixels is not numerically the same as a gradient in meters, even for the same modeled phenomenon. Optimization must account for representation and scaling.

One residual connects a prediction to its gradient#

Let $z=wx+b$, target $y$, residual $r=z-y$, and loss $L=r^2/2$. Write the execution graph as named intermediate objects:

The local derivatives are $\partial L/\partial r=r$, $\partial r/\partial z=1$, and $\partial z/\partial w=x$, $\partial z/\partial b=1$, $\partial z/\partial x=w$. Multiplying along each dependency path gives

At $x=2,w=2,b=1,y=3$, prediction is 5, residual 2, loss 2, and gradients with respect to $(w,b,x)$ are $(4,2,4)$. The gradient with respect to the input is not an instruction to train the input; it is a sensitivity that may be useful for analysis. Which variables an optimizer updates is an implementation and modeling choice.

For a batch with design matrix $A$ and parameter vector $\theta$, the analogous objective is $F(\theta)=\tfrac12\|A\theta-y\|^2$. Its gradient is $A^T(A\theta-y)$. The residual is a vector in observation space; multiplying by $A^T$ maps its correlations with the columns back into parameter space. At a least-squares optimum, the residual is orthogonal to the column space, not necessarily zero. The geometry explains the formula instead of merely demanding that it be memorized.

Three methods with different meanings#

Symbolic differentiation manipulates expressions into derivative expressions, such as converting $\sin(x^2)$ into $2x\cos(x^2)$. It may simplify and reason about a formula, with domains still needing attention.

Finite differences compare nearby evaluated outputs. A central difference $[f(x+h)-f(x-h)]/(2h)$ is an approximation whose truncation and roundoff depend on step size, smoothness, and precision. A tiny step can subtract nearly equal rounded values and be worse than a moderate one.

Automatic differentiation (AD) augments the computation with the chain rule for each executed operation. It does not choose a finite-difference step or return a universal symbolic formula. With floating-point operations, its derivative evaluations still have roundoff; with branches it follows an execution path and requires care at discontinuities or nondifferentiable points. “Exact derivative up to floating-point error” refers to the modeled smooth arithmetic expression evaluated by the rules, not a proof that every arbitrary program defines a differentiable real function.

Deriving forward mode with a pair of numbers#

Represent a value and a directional derivative by $a+\dot a\varepsilon$, where the formal symbol satisfies $\varepsilon^2=0$. This algebraic rule is not the assertion that a nonzero real number has square zero. It packages first-order information:

The vanished $\varepsilon^2$ term gives the product rule. For $b\ne0$, division has tangent $(\dot a b-a\dot b)/b^2$. A smooth unary function gives

in this first-order algebra. For example, sine propagates $\cos(a)\dot a$; exponential propagates $e^a\dot a$; logarithm requires $a>0$ and propagates $\dot a/a$.

Seed input $x$ with tangent 1 to get its derivative. For $x=2$, computing $x*x$ gives pair $(4,4)$; then sine gives $(\sin4,4\cos4)$. Seed a vector $x$ with tangent direction $v$ to obtain the Jacobian-vector product $Jv$, without constructing all entries of $J$. Recovering every partial separately by forward mode generally requires multiple seed directions.

Why reverse mode is attractive for one scalar loss#

When there are many parameters and one scalar loss, a reverse pass accumulates each intermediate’s influence on the loss. Start with derivative of the loss with respect to itself equal to 1. Each operation sends its incoming sensitivity backward, multiplied by its local derivative; contributions along multiple paths add. The residual example did exactly this. Reverse mode can compute all parameter sensitivities with work proportional to the executed graph’s size, but usually needs stored or recomputed forward intermediates. G13 develops an explicit neural-network backward pass; it is not a call to an opaque framework substituted for understanding.

Verification and nondifferentiable boundaries#

Check a new AD rule against an independently derived analytic derivative and a range of finite-difference steps away from singularities. Agreement at a single symmetric input may conceal a missing term. For $f(x)=|x|$, no derivative exists at zero even if an implementation selects a branch and returns a number. For division and logarithm, maintain the domain restrictions. For iterative solvers, differentiating a finite executed iteration count may differ from differentiating the exact solution of the mathematical fixed-point equation. Name the differentiated object.

Independent practice#

F foundation C standard course work T transfer S optional stretch L runnable lab or project. Attempt the complete set before opening its solutions.

Complete solutions: W04#

Compare the first step where your work differs. A corrected answer is not yet an independent success.

Open every solution

"""W04: forward-mode automatic differentiation; each operation carries its tangent."""
from dataclasses import dataclass
import math


@dataclass(frozen=True)
class Dual:
    value: float
    tangent: float=0.

    def __post_init__(self):
        if not all(type(x) in (int,float) and math.isfinite(x)
                   for x in (self.value,self.tangent)):
            raise ValueError("finite real value and tangent required")

    @staticmethod
    def wrap(other):
        return other if isinstance(other,Dual) else Dual(other)

    def __add__(self,other):
        other=self.wrap(other)
        return Dual(self.value+other.value,self.tangent+other.tangent)
    __radd__=__add__

    def __neg__(self):return Dual(-self.value,-self.tangent)
    def __sub__(self,other):return self+(-self.wrap(other))
    def __rsub__(self,other):return self.wrap(other)+(-self)

    def __mul__(self,other):
        other=self.wrap(other)
        return Dual(self.value*other.value,self.tangent*other.value+self.value*other.tangent)
    __rmul__=__mul__

    def __truediv__(self,other):
        other=self.wrap(other)
        if other.value==0:raise ZeroDivisionError("zero primal denominator")
        value=self.value/other.value
        tangent=(self.tangent-value*other.tangent)/other.value
        return Dual(value,tangent)
    def __rtruediv__(self,other):return self.wrap(other)/self


def sin(x):
    x=Dual.wrap(x)
    return Dual(math.sin(x.value),math.cos(x.value)*x.tangent)


def exp(x):
    x=Dual.wrap(x);value=math.exp(x.value)
    return Dual(value,value*x.tangent)


def log(x):
    x=Dual.wrap(x)
    if x.value<=0:raise ValueError("log requires positive primal input")
    return Dual(math.log(x.value),x.tangent/x.value)


if __name__=='__main__':
    w=Dual(2,1);x=2;b=1;y=3
    loss=(w*x+b-y)*(w*x+b-y)/2
    print(loss)  # value 2; derivative with respect to w is 4

Retention and readiness#

State the input/output dimensions before writing a derivative. Explain how a residual in observation space becomes a gradient in parameter space. Compare forward and reverse mode using the number of inputs, outputs, and stored intermediates rather than calling one universally superior.