RBM Training Tip

A sparse stacked RBM (a type of a deep belief network) is supposed to capture robust patterns that can be very useful features for various classification/machine-learning tasks.

However, the model is very dependent on the strength of the visible units. I'm finding that even obvious patterns that humans can easily detect are not captured by the RBM learning if the visible units are scaled down by some reasonable factor. This may be because the energies for a lot of desirable patterns become too small and the model will only be able to learn a few bases that do give strong signals.

It helps to scale up lower layer weights in these cases.