Deep Neural Network Learning: Linear Regression and Logistic Regression
Although I roughly understood the method of Sparse Coding, I always felt there were some inconsistencies. According to my master's teaching back then, whenever I encountered a problem that I couldn't figure out, it was always because my foundation was not solid. So, I reviewed linear regression and logistic regression to clarify my understanding.
Linear regression, in concept, is actually quite simple. It can be abstracted into a geometric concept. For example, on a plane with a bunch of points, if you can connect all of them with a curve, then this is a linear problem because the curve can cover them all, and linear problems can be solved through linear regression.
As for the term "regression," if defined using standard terminology, it is: a statistical analysis method that studies the dependent relationship of a random variable Y on another (X) or a set of (X1, X2, ..., Xk) variables. It is a statistical method to study the relationship between one or more random variables Y1, Y2, ..., Yi and other variables X1, X2, ..., Xk. It is also known as multiple regression analysis. Usually, Y1, Y2, ..., Yi are called dependent variables, and X1, X2, ..., Xk are called independent variables.
My understanding of "regression" is that, assuming there is a line that can cover all the points, you keep drawing on a plane until you find this curve; this process is called regression.
For data, it is about trying to find the relationships between data, especially the significant relationships of influence between data.
The methods of regression are usually of two types, one of which is called the least squares method.
The least squares method was independently discovered by the French scientist Legendre in 1806, but it remained unknown to the world and was obscure. Later, Gauss used the least squares method in his "Theory of Celestial Motion," and Legendre and Gauss even had a dispute over who invented it.
This process is very simple. First, assume a target curve is xi, then each attempt to draw is xj, for example, at the 0 point position, and the deviation between them is x0-x0.
Because here Xi and Xj are not two points, but two functions, the correlation between Xi and Xj is minimized.
Obviously, if you sum up the errors of each Xi and Xj, it is the error of the two functions, but this is difficult to solve. So, their squared difference is used as the error. Thus, the total error judgment is defined as: Σ(xi-xj)2
To minimize the total error, of course, it is best if Xi and Xj are equal. The problem is that we are originally trying to find xi, while constantly trying to draw is xj. What we need to find is how to make xj continuously adjust to become Xi.
Because the first xj is like a curve drawn randomly, this problem is transformed into: how to make Xj approach Xi faster.
This is like the following problem: now there are 10 apples, one of which is lighter than the others. Now there is a balance scale. How to weigh to find the lightest apple as quickly as possible.
The method is: put five on the left and five on the right, find the lighter side, then divide the five into two on the left and two on the right, and weigh again. If they weigh the same, the remaining one is the light one. If they don't weigh the same, take the two lighter ones and weigh them on the scale, so the lighter one is found.
Why can 10 apples be found in three times? Because 2 to the power of 3 equals 8, and 2 to the power of 4 equals 16. In the range of 8 to 15 apples, they can all be found in three times. This is the fastest way to find them, and its essence is:
y = 21+22+23+...2i
So using this method, it is easy to think that the fastest way to make xj approximate xi is to continuously approximate the process of the double-multiplication curve, and this can be calculated.
The adjustment made by continuously approximating is called the gradient descent method, which is a local minimum, while finding the global minimum at once is called the least squares method.
Why not find the global minimum at once? Because in most cases, the amount of sample data is too large, and the calculation is very heavy, making it difficult to find the global minimum. Moreover, in reality, data is not so clear and visible. A loss function must be added to ignore potentially incorrect or samples that need to be ignored. The loss function of regression is not a linear least squares problem, which leads to the fact that even after adding noise interference, the linear least squares problem cannot be completely solved.
There should be other reasons, but personally, I think the most important reason is that for computers, the gradient descent algorithm is easy to write code for and is very suitable for this kind of computation.
Then we need to talk about logistic regression. Linear regression is suitable for solving linear problems. A so-called linear problem means something that can directly give you an answer. Logistic regression, on the other hand, tells you what something might be.
What is the significance of this? For example, for classification problems, logistic regression should be used. If it is for calculation, such as how much money invested will produce how much return, linear regression should be used.
To be more specific, for example, whether a person is a man or a woman based on sexual characteristics is a problem solved by logistic regression. But how fat a person becomes based on how much food they eat is a problem solved by linear regression.
The concept of logistic regression is actually like this: in each trial, there are only two possible outcomes, and the occurrence of the two outcomes is mutually exclusive and independent of each other, unrelated to the results of other trials. The probability of the event occurring or not remains unchanged in each independent trial. This series of trials is collectively called n-fold Bernoulli trials. When the number of trials is 1, the binomial distribution is the Bernoulli distribution.
Those that satisfy this use logistic regression.
Then speaking of logistic regression, this feels quite interesting because the most commonly used thing in logistic regression is a sigmoid function. Through the sigmoid function, non-linear classification problems can be solved.
And the graph of this function is like this:
And our family's Taiji diagram is like this:
Isn't it very similar? Why use an S-curve?
Because in terms of distribution, the S-curve is most sensitive to changes in the middle data and can suppress changes at both ends, reflecting changes better.
If the S-line in the Taiji line is transformed to a certain extent using a normal distribution function, the S-line of the Taiji diagram can be obtained. Related papers can be found on Baidu.