Deep Neural Network Learning: AutoEncoder Sparse Coding

This is a technology that feels almost magical. Given a set of images, software can scan them and extract features. For humans, this is easy, but for software, the biggest question is: how should it be done?

The technology behind writing software isn't particularly high-tech; ultimately, it relies on human thought. We convert these thoughts into code to solve real-world problems. Programming can be said to be an excellent way to verify ideas. I even believe that Daoists of the new era should each master programming skills.

I've gone off-topic.

So, given a pile of images, how should we implement the extraction of features?

Generally, to solve problems intelligently, the approach is as follows: provide a set of inputs and a set of outputs. For example, in weather forecasting, the input is time, and the output is the weather. Then, assign random values to the neural network, use mathematical operations to see how much the target result differs from the required output data, convert this into a calculus direction, and calculate how to adjust the data in reverse along the computation path to piece together the target result. Then, repeatedly repeat this calculation and piecing process until it approaches the target result. The parameters determined in the neural network that yield a satisfactory answer can be understood as features.

But what if it's a pile of images? How to handle this? How to extract features from images? Here, we must mention a very interesting solution.

It proceeds as follows:

1. Cut the images into various small blocks and convert these blocks into data representations.

2. Use neural networks to compute these data.

But here arises a problem: what is the output result of the computed data?

This is the extremely clever part: the output result undergoes another neural network computation so that it can output the input value.

Input -> Computation (Encoding) -> Computation (Decoding) -> Output

Then compare the input and output. Based on the difference, adjust the intermediate encoding and decoding computations. There is no need to worry about what exactly the encoding and decoding are doing.

Ultimately, when the precision of the input and output results meets satisfactory requirements, what remains are the features.

Why is this the case? The principle is actually simple. The neural network processing in the intermediate encoding layer is equivalent to arbitrarily interfering with the original image, while the neural network processing in the decoding layer is equivalent to restoring the interfered image.

And there is an interesting aspect to this process: because there is a process of interference and de-interference, the output equals the input. After multiple transformations, what remains and can be identified as the original input is the feature.

Similarly, in this line of thinking, we can add an additional confuser, i.e., randomly add a large amount of noise and then eliminate it. This can also enhance the neural network's ability to resist interference. The encoder obtained by this method after interference processing is called a Denoising AutoEncoder.

This is truly an astonishing idea, and it can only be implemented on computers; traditional other computing tools cannot achieve this because it relies on dense computations.

So, is this enough?

Thus, a belief emerges: can the abstraction of these features be made as simple as possible?

Reduce it again and again, until non-action; through non-action, there is nothing left undone?

Is this possible?

Therefore, if we persist in believing that behind even the most complex things there are simple rules, we can do one more thing based on this belief: Sparsification

How to do it? Because behind complexity there are always simple rules, right? Does simple rule mean that after deleting many things, only keeping the most core elements can restore most of the original information?

If so, this means that if we can find a way where most neurons in the neural network do not activate but still successfully produce output from input, then we can achieve this.

Therefore, we can impose restrictions on neurons. Each neuron has an activation value, and it is stipulated that the total activation value of all neurons must not exceed a certain number, such as 0.05. This will cause most neurons to become 0 values, while a very small number of neurons need to calculate their weights more precisely.

How is this implemented?

Let's start from the source. Sparsification is essentially finding a set of "overcomplete" basis vectors to represent sample data more efficiently. The purpose of the sparse coding algorithm is to find a set of basis vectors such that we can represent the input vector as a linear combination of these basis vectors.

First, the input and output must satisfy: Min |I – O|, where I represents input, O represents output, i.e., the difference between input and output is minimized.

If we use images as an example, it means finding a small number of tiny images to combine into this large image. Then, this pile of tiny images is called the dictionary Φ. If we randomly pick out a small image (larger than the tiniest image), we can fit it entirely because it might be composed of, for example, 20% of Φ1, 50% of Φ2, and 50% of Φ3 (since it is larger than the smallest image, the sum will exceed 100%).

So, because "we need to find the smallest image," this adds an additional condition constraint. Since O is the output, O is the target image that needs to be solved for the picked-out image, and the input is I, so it can be expressed as:

I=x₁+x₂+x₃+x₃.....x_i

 O = a₁Φ₁ + a₂Φ₂ + … +a_nΦ_n

Then consider adding a penalty coefficient, which is used to adjust the values in the network.

So, actually solving the problem involves piecing together a and Φ such that the calculation result becomes a minimum value.

 

What follows the plus sign on the right is the added sparse cost function. This function can also use a logarithmic cost function (utilizing Cauchy prior probability), i.e., log(1+a_i²). It is used to restrict values far greater than zero to avoid the values in the network being too different and concentrated, thereby losing the purpose of sparsification.

Because the target minimum value is stable, if the values in Φ or a are too large, the cost function on the right becomes meaningless --- it becomes very small. Therefore, we need to restrict the value of ||Φ||² to be less than a certain value.

In implementation, each iteration is divided into two steps:

a) Fix the dictionary Φ[k], then adjust a[k] to minimize the above equation, i.e., the objective function (i.e., solve the LASSO problem).

b) Then fix a[k], and adjust Φ[k] to minimize the above equation, i.e., the objective function (i.e., solve the convex QP problem).

 Then just repeat this calculation.