Fisher Information: Sensitivity of Distribution to Parameter Changes

pθ = N(μ, σ²)
pθ+δ = N(μ+δ, σ²)
distribution difference
Left: Fμμ = 1/σ² = 4.00
KL ≈ ½Fδ² = 0.500
exact KL = 0.500
Right: Fμμ = 1/σ² = 0.25
KL ≈ ½Fδ² = 0.031
exact KL = 0.031
Both panels apply the same parameter perturbation δ to the mean μ. With small σ (left), the Fisher information F = 1/σ² is large — the distribution is "sharp", so the same δ causes a big distributional change (large KL). With large σ (right), F is small — the distribution is "flat", so the same δ barely changes it (small KL). The Fisher approximation KL ≈ ½Fδ² matches the exact KL = δ²/(2σ²) perfectly for Gaussians with shared variance.