KL Estimator Numerical Experiments

Numerical Experiment: Bias and Variance of KL Estimators

μ_p: 0.10 Samples: 1M

q = N(0,1), p = N(0.10, 1), True KL = μ²/2 = 0.00500

k₁ = −log r

k₂ = ½(log r)²

k₃ = (r−1) − log r

True KL

Estimator	E[k]	Bias / True KL	Std / True KL	Verdict
k₁ = −log r	—	—	—	—
k₂ = ½(log r)²	—	—	—	—
k₃ = (r−1)−log r	—	—	—	—

q = N(0,1), p = N(μ_p, 1). True KL = μ²/2. Samples x ~ q; log r = μx − μ²/2. Drag μ to see how bias and variance change — notice k₃ stays unbiased everywhere while k₂'s bias grows with μ.