Abstract: A new algorithm for unsupervised learning of invariances is
presented. The basic idea is to learn a nonlinear input-output function
which extracts slowly varying aspects from the input signal by minimizing
the temporal variation of the output signal. This is a known approach.
The algorithm, however, differs from existing learning rules. Firstly, it
computes the solution in a closed form (like PCA) and is guaranteed to
find the optimum within the considered function class. Secondly, not only
one but many uncorrelated output signal components can be generated
easily, which is important for hierarchical networks.
The algorithm
is then applied to a simple model of the visual system with a
one-dimensional retina. Depending on what stimuli are used for training,
the network can learn translation-, scale-, rotation- (cyclic shift),
contrast-, or illumination-invariance. Relatively few stimulus patterns
are needed for training to achieve good generalization to new patterns.
The representation generated is suitable for pattern recognition. Overall
the model suggests that it may be plausible that our visual system learns
invariances based on fairly limited visual experience.