DL generalises as the parameter-function map is biased towards simple functions
Skimmed through a theory paper “ DEEP LEARNING GENERALIZES BECAUSE THE PARAMETER-FUNCTION MAP IS BIASED TOWARDS SIMPLE FUNCTIONS ”. A very interesting paper showing that random initialization of NN parameters (under various initialization distributions) leads to a distribution over the functional space, such that the simpler functions have a higher probability. One of the main tools of the paper is the notion of the ‘parameter-function map’ which maps parameters to the functions realized by these parameters. For example, if there are n boolean variables, the size of the space of functions is 2^(2^n). The parameters-function map then assigns each parameter in R^d (if ‘d’ is the size of the parameters) to one of these functions (note that the space of parameters is continuous and infinite and the space of functions is discrete and finite, in this example). This is a critical notion as it says we should not be deceived by the number of parameters as the measure of complexity, instead...