Monday, September 16, 2019

Note: Moment approximations for Bernoulli neurons with sigmoidal nonlinearity

Consider a stochastic, binray, linear-nonlinear unit, with spiking output s, synaptic inputs x, weights w, and bias (threshold) b:

(1)sBernoulli[p=Φ(a)]a=wx+b,

where Φ() is the cumulative distribution function of a standard normal distribution. Note that Φ() can be rescaled to closely approximate the logistic sigmoid if desired. Assuming the mean μ and covariance Σ of x are known, can we obtain the mean and covariance of s?