Reflections on Using SVMs with Small Samples

Today when validating test data, I found that when using cross_val_score for cross-validation, the larger the proportion of the randomly allocated test set, the higher the accuracy—and the smaller the test set, the more likely the average accuracy is to be low.

There are two major pitfalls when deriving models with SVM. One is that after train_test_split, when the data volume is insufficient, the learned sample distributions differ greatly and are extremely unstable; how to find the optimal train_test_split is hard to determine. The other is whether the assessment of the model’s generalization ability is reliable—and that is bound up with the dataset partitioned by train_test_split. This directly means that occasionally scoring high on the test set is merely an illusion.

This is because what SVM actually fits is the data distribution, and what it learns from the samples is also the distribution; but when the number of samples is randomized through sampling, the structure revealed by the distribution also differs.

When the test set is small, the data on which inference errs may at times concentrate together, and thus average performance on the test set suffers.

That is why, in convolutional neural networks for image recognition, images can be sampled at different resolutions; the aim, in fact, is to obtain different distributions that all point toward a single concept—and training in this way can improve the neural network’s generalization ability.

The SVM process fits all at once in a single step, so it is hard to adjust and improve, because each fit is of the entire distribution, making it difficult to realize adjustments as in incremental learning.

Random forests, relatively speaking, have a learning mechanism that splits subtrees for local sampling, so they are less prone to overfitting; whether the overall effect is better, however, depends on the actual situation of the data.

Therefore, for a dataset fitted by SVM, one should make this assumption: among the existing data, if a sub-distribution that can be sampled out happens to possess very good generalization performance, then using grid search over parameters is quite likely to find it.

And better generalization means—for instance, in an actual inference task, if a complete data distribution exists—this well-generalizing subset distribution will overall be closer to that complete data space distribution.

Thus, if the entire dataset is G and the approximating subset is g, then other subsets drawn from G, if relatively small—for example a, b, and c—need not necessarily present distributions similar to g.

Clearly, if an overall distribution is {0,1,0,0,1,0,1,0,0,0}, distributions drawn at a 0.33 ratio might be {0,0,0} and {1,1,1}. This also means that if the dataset is not large and cross-validation is performed on a relatively small test set, the scores obtained will tend to be unobjectively low.

Therefore, when taking K folds for cross-validation, the amount of data used for testing should be comparable to that of the original training set in order to better and more objectively reflect scores such as accuracy, ROC, and F1.

According to experiments: for example, using 0.8 of the data as the training set, a random forest can reach 90% accuracy; but if one again takes a randomly sampled 0.8 portion of the full dataset for K-fold validation, the result drops to around 78%. This is similar to the situation often seen in experiments where the data model looks good, yet 10%–20% is lost once it goes live.