Natural inductive biases for artificial intelligence

Abstract

The study of inductive bias is one of the most all encompassing in all of machine learning. Inductive biases define not only the efficiency and speed of learning, but also what is ultimately possible to learn by a given machine learning system. The history of modern machine learning is intertwined with that of psychology, cognitive science and neuroscience, and therefore many of the most impactful inductive biases have come directly from these fields. Examples include convolutional neural networks, stemming from the observed organization of natural visual systems, and artificial neural networks themselves intending to model idolized abstract neural circuits. Given the dramatic successes of machine learning in recent years however, more emphasis has been placed on the engineering challenges faced by scaling up machine learning systems, with less focus on their inductive biases. This thesis will be an attempted step in the reverse direction. To do so, we will cover both naturally relevant learning algorithms, as well as natural structure inherent to neural representations. We will build artificial systems which are modeled after these natural properties, and we will demonstrate how they are both beneficial to computation, and may serve to help us better understand natural intelligence itself