Note: Differentiating expectations of a function of a random variable with respect to location and scale parameters

Consider a real-valued random variable with a known probability distribution $Pr(z)=\varphi (z)$. From $\varphi (z)$, one can generate a scale/location family of probability densities by scaling and shifting $\varphi (z)$:

$$Pr(x;\mu ,\sigma )=\frac{1}{\sigma }\varphi \left(\frac{x-\mu }{\sigma }\right)$$

The most familiar example of such a family is the univariate Gaussian distribution, when $\varphi (z)=[2\pi {]}^{-1/2}\exp (-\frac{1}{2}{z}^2)$. Now, consider the expectation of a function of $⟨f(x)⟩$ with respect to $Pr\left(x\right)$.

$$⟨f(x)⟩={\int }_{\mathbb{R}}f(x)Pr(x)dx={\int }_{\mathbb{R}}f(x)\frac{1}{\sigma }\varphi \left(\frac{x-\mu }{\sigma }\right)dx$$

What are the derivatives of $⟨f(x)⟩$ with respect to $\mu$ and ${\sigma }^2$? The answers are:

$$\begin{array}{}\text{(1)}& \begin{array}{rl}{\partial }_{\mu }⟨f(x)⟩& =⟨f^{\prime }(x)⟩\\ {\partial }_{{\sigma }^2}⟨f(x)⟩& =\frac{1}{2{\sigma }^2}{⟨(x-\mu )f^{\prime }(x)⟩}_x\end{array}\end{array}$$

This question often appears in the special case that $x$ is normally distributed; You'll find derivations elsewhere online given in terms of the cumulative distribution function of the standard normal distribution.

This note outlines a derivation for any scale/location family using elementary calculus. These derivatives can be obtained by considering how perturbing $\mu$ or $\sigma$ shifts and/or scales the probability density.

For the mean, consider the definition of the derivative:

$$\begin{array}{}\text{(2)}& \begin{array}{rl}\frac{d}{d\mu }⟨f(x)⟩& =\underset{ϵ\to 0}{lim}\frac{1}{ϵ}\{{\int }_{\mathbb{R}}f(x)\frac{1}{\sigma }\varphi \left(\frac{x-ϵ-\mu }{\sigma }\right)dx-{\int }_{\mathbb{R}}f(x)\frac{1}{\sigma }\varphi \left(\frac{x-\mu }{\sigma }\right)dx\}\\ & =\underset{ϵ\to 0}{lim}\frac{1}{ϵ}\{{\int }_{\mathbb{R}}f(x)\frac{1}{\sigma }\varphi \left(\frac{x-ϵ-\mu }{\sigma }\right)dx-⟨f(x)⟩\}\end{array}\end{array}$$

Let $y=x-ϵ$. Then perform a change of variables ($dy=dx$ and $x=y+ϵ$):

$$\begin{array}{}\text{(3)}& \begin{array}{rl}\frac{d}{d\mu }⟨f(x)⟩& =\underset{ϵ\to 0}{lim}\frac{1}{ϵ}\{{\int }_{\mathbb{R}}f(y+ϵ)\frac{1}{\sigma }\varphi \left(\frac{y-\mu }{\sigma }\right)dy-⟨f(x)⟩\}\\ & =\underset{ϵ\to 0}{lim}\frac{1}{ϵ}\{⟨f(x+ϵ)⟩-⟨f(x)⟩\}\\ & =\underset{ϵ\to 0}{lim}\frac{1}{ϵ}\{⟨f(x)+ϵf^{\prime }(x)⟩-⟨f(x)⟩\}\\ & =⟨f^{\prime }(x)⟩\end{array}\end{array}$$

For the variance, consider the derivative in $\sigma$:

$$\begin{array}{}\text{(4)}& \begin{array}{rl}\frac{d}{d\sigma }⟨f(x)⟩& =\underset{ϵ\to 0}{lim}\frac{1}{ϵ}\{{\int }_{\mathbb{R}}f(x)\frac{1}{\sigma +ϵ}\varphi \left(\frac{x-\mu }{\sigma +ϵ}\right)dx-⟨f(x)⟩\}\end{array}\end{array}$$

Let $y=\frac{\sigma }{\sigma +ϵ}(x-\mu )+\mu$. This gives the change of variables $$\begin{array}{}\text{(5)}& \begin{array}{rl}dz& =\frac{\sigma +ϵ}{\sigma }dy\\ x& =\frac{\sigma +ϵ}{ϵ}(y-\mu )+\mu =y+\frac{ϵ}{\sigma }(y-\mu )\end{array}\end{array}$$

Substituting, and simplifying:

$$\begin{array}{}\text{(6)}& \begin{array}{rl}\frac{d}{d\sigma }⟨f(x)⟩& =\underset{ϵ\to 0}{lim}\frac{1}{ϵ}\{{\int }_{\mathbb{R}}f(y+\frac{ϵ}{\sigma }(y-\mu ))\frac{1}{\sigma +ϵ}\varphi \left(\frac{y-\mu }{\sigma +ϵ}\right)\frac{\sigma +ϵ}{\sigma }dy-⟨f(x)⟩\}\\ & =\underset{ϵ\to 0}{lim}\frac{1}{ϵ}\{{\int }_{\mathbb{R}}f(y+\frac{ϵ}{\sigma }(y-\mu ))\frac{1}{\sigma }\varphi \left(\frac{y-\mu }{\sigma +ϵ}\right)dy-⟨f(x)⟩\}\\ & =\underset{ϵ\to 0}{lim}\frac{1}{ϵ}\{⟨f(y+\frac{ϵ}{\sigma }(y-\mu ))⟩-⟨f(x)⟩\}\\ & =\underset{ϵ\to 0}{lim}\frac{1}{ϵ}⟨f^{\prime }(x)\frac{ϵ}{\sigma }(x-\mu )⟩\\ & =\frac{1}{\sigma }⟨f^{\prime }(x)(x-\mu )⟩\end{array}\end{array}$$

To get the derivative in terms of the variance ${\sigma }^2$, apply the chain rule

$$\begin{array}{}\text{(7)}& \begin{array}{r}\frac{d}{d{\sigma }^2}⟨f(x)⟩=\frac{d\sigma }{d{\sigma }^2}\frac{d}{d\sigma }⟨f(x)⟩=\frac{1}{2{\sigma }^2}⟨f^{\prime }(x)(x-\mu )⟩\end{array}\end{array}$$