The Information Theory of Developmental Pruning: Optimizing Global Network Architecture Using Local Synaptic Rules

Another paper from the Hennig lab is out, this one is from Carolin Scholl's master's thesis. Once again, we used an artificial neural network to get intuition about biology. The paper is on BioRiv, and you can also get the PDF here. 

Local learning rules to attenuate forgetting in neural networks

Another paper from our work on Restricted Boltzmann Machines (RBMs) from the Hennig lab. 

[get paper PDF]

Main points:

• We noticed that measures of synaptic importance were available from local firing statistics (at least in Boltzmann machines)
• We look at an artificial neural network that stores memories and is easy to analyze. (Hopfield nets are the zero temperature limit of a Boltzmann machine).
• We evaluated whether this local measure of synaptic importance could help stabilize important weights when networks learn multiple things that interfere with each-other
• Intuition: biological variables, like synapse size, can correlate with useful statistical quantities. This provides tricks for biologically-plausible approximations of algorithms.
• Intuition: in systems that learn, if a parameter takes on an unusual or surprising value, it is likely that this value was set through learning—and you might want to leave it fixed.