Residual Networks

I came across an explanation on Zhihu saying that the basic form is actually similar to differential amplification, which makes a lot of sense.

Suppose: F'(5)=5.1. The rate of change from 5 to 5.1 is extremely low because the base 5 is too large, so it can be split:

H(5)=5.1, H(5)=5+F(5), F(5)=0.1. At this point, because 0 becomes 0.1 in the split term, the rate of change reaches 10%, thereby amplifying this effect. Since neural networks cannot well identify such a split, artificial structure design is needed to assist.

In theory, more network layers generally mean richer ability to extract details and abstractions. However, when layers are iterated too many times, gradient explosion or gradient vanishing can occur, which can be addressed through regularization.

But even if this problem is solved, a degradation problem still remains: as the number of layers increases, accuracy actually drops.

A trick here is that, when H(x) = F(x) + x and F(x)=0, clearly H(x) = x. If f(x) gets closer and closer to 0, then h(x) approaches x more and more closely.

So we can construct such a form:

Thus, x can skip two layers and serve directly as input, because h(x)=x. f(x) here of course will not equal 0, but ReLU can make as many places in the matrix as possible become 0, thereby achieving residual amplification.

How X changes is unclear, because it is itself a value to be solved. But here the original value of X can be used directly, because if X is changed, the original X structure corresponding to the residual would be destroyed. The residual here is actually not a difference — it is merely a value that can only gradually approach a difference through training.