Fisher Information: Sensitivity of Distribution to Parameter Changes
pθ = N(μ, σ²)
pθ+δ = N(μ+δ, σ²)
distribution difference
Left: Fμμ = 1/σ² =4.00
KL ≈ ½Fδ² =0.500
exact KL =0.500
Right: Fμμ = 1/σ² =0.25
KL ≈ ½Fδ² =0.031
exact KL =0.031
Both panels apply the same parameter perturbation δ to the mean μ.
With small σ (left), the Fisher information F = 1/σ² is large — the distribution is "sharp", so the same δ causes a big distributional change (large KL).
With large σ (right), F is small — the distribution is "flat", so the same δ barely changes it (small KL).
The Fisher approximation KL ≈ ½Fδ² matches the exact KL = δ²/(2σ²) perfectly for Gaussians with shared variance.