A Quick Look at MNIST Training Accuracy Statistics
| 6-layer NN 784-2500-2000-1500-1000-500-10 (on GPU) [elastic distortions] | none | 0.35 | Ciresan et al. Neural Computation 10, 2010 and arXiv 1003.0358, 2010 |
| committee of 25 NN 784-800-10 [elastic distortions] | width normalization, deslanting | 0.39 | Meier et al. ICDAR 2011 |
In artificial neural networks, the lowest error rates are 0.35 and 0.39 respectively. However, it can be seen that the one with an error rate of 0.39 used only 800 neurons as a single hidden layer, whereas the one with 0.35, although yielding better results, employed a 6-layer structure with 2500, 2000, 1500, 1000, and 500 neurons as hidden layers. This computational load is extremely heavy, making it difficult for ordinary computers to handle. Therefore, relatively speaking, the data that achieved an error rate of 0.39 using only 800 neurons with width normalization and deslanting processing seems to have more practical value.
Normalization refers to regularization, while deslanting refers to anti-aliasing the images, similar to adding noise, or one might call it jagging or aliasing. As for applying elastic distortions to the original images, this is quite natural and can enhance the network's generalization ability.
| large conv. net, unsup pretraining [elastic distortions] | none | 0.39 | Ranzato et al., NIPS 2006 |
| large conv. net, unsup pretraining [no distortions] | none | 0.53 | Jarrett et al., ICCV 2009 |
| large/deep conv. net, 1-20-40-60-80-100-120-120-10 [elastic distortions] | none | 0.35 | Ciresan et al. IJCAI 2011 |
| committee of 7 conv. net, 1-20-P-40-P-150-10 [elastic distortions] | width normalization | 0.27 +-0.02 | Ciresan et al. ICDAR 2011 |
| committee of 35 conv. net, 1-20-P-40-P-150-10 [elastic distortions] | width normalization | 0.23 | Ciresan et al. CVPR 2012 |
The performance of convolutional neural networks is surprisingly good. As seen above, even without elastic distortions, an error rate of 0.53 can be achieved. After adding width normalization, the best error rate reached 0.23, which is absurdly high accuracy. However, this involved using 35 convolutions, resulting in an equally massive computational load.