Logistic Regression

The logistic regression is a method to calculate the relationships between a nominal categorical dependent variable (Y) and continuous/categorical independent variables (X_i) like the independent variables in the linear regression. For example, the color dependent variable (Y) with the following value: blue, red or green.

Binary Logistic Regression (Go to the calculator)

When the dependent variable is a binary variable, also called a dichotomous variable, you should use the Binary Logistic Regression. The model will calculate the probability for the category to occur based on the independent variables, X_j.
The dependent variable Y may have only two options 1 or 0, such as the win or lose and success or failure. The following model required the accumulated input data, based on i combination of X: (x₁..x_p), there is also a similar commonly used model is based on single events, therefore each data row is a single event. In this case, every row is a single event, and Y_i value may be only 1 or 0.

• y(1)_i: the total 1 occurrences based for i combination of X.
• y(0)_i: the total 0 occurrences based for i combination of X.
• e_i: the total events for i combination of X, e_i=y(1)_i+y(0)_i
• p_i: the observed probability for event=1.
• p̂_i: the predicted probability for event=1 based on the model.

Odds is the ratio between the probability that the event will happen to the probability it won't happen
The odds is actually similar to the probability but from a different angel. $$odds=\frac{p}{1-p}$$ Examples
When P = 1/3 the odds are 1:2 (odds = 0.5).
When P = 1/2 the odds are 1:1 (odds = 1).

H₀: ln(odds) = b₀
H₁: ln(odds) = b₀+b₁X₁+...+b_pX_p

$$ odds(x_1..x_p)=\frac{p(x_1..x_p)}{1-p(x_1..x_p)}=e^{b_0+b_1x_1+...+b_px_p}\\ p(x_1..x_p)=\frac{1}{1+e^{-(b_0+b_1x_1+...+b_px_p)}}$$ The maximize log-likelihood method is used instead of the least squares method that is used in the linear regression. The method is based on the Binomial distribution.

Likelihood is the probability that the sample data will occur and the maximize log-likelihood method finds the P_i that will maximize this probability.
n: The number of X combinations (x₁..x_p) $$L=\prod_{i=1}^n \hat p^{y_i(1)}(1- \hat p_i)^{y_i(0)}$$ $$LL=ln(L)=\sum_{1=1}^n{(y_i(1) ln(\hat p_i) + y_i(0)ln(1- \hat p_i))}$$

Newton's Method

We use the newton's method to find the B vector of parameters that will maximize the Log-likelihood function based on the following iteration formula. The iteration loop will run until the differences between B_R+1 and B_R will limit to zero for each coefficient element. B₀=[0,0,..0]
$$ V=\begin{bmatrix} e_1 \hat p_1(1- \hat p_1) & 0 & 0 & 0\\ 0 & e_2 \hat p_2(1- \hat p_2) & 0 & 0 \\ 0 & 0 & ...& 0\\ 0 & 0 & 0 & e_n \hat p_n(1- \hat p_n) \end{bmatrix}$$ R: iteration.
T: t_i vector.
P: p_i vector. (observed probabilities)
p̂: p̂_i vector (predicted probabilities).
B: b_i vector. (coefficients)
$$ B_{R+1}=B_R+(X'V_RX)^{-1}X'T⊙(P- \hat P_R)$$ All the multiplications are matrix multiplications, except for the last which is an element wise multiplication . Example: [2,3] ⊙ [4,5] =[8,15].

Multinomial Logistic Regression (Go to the calculator)

When the dependent variable can get more than two categorical values, you should use the Multinomial Logistic Regression. The model will calculate the probability for the category to occur based on the independent variables, X_j. The dependent variable Y may have more than two options, for example Red, Blue, Green.

You may use one of the following input data methods:
Single Y column - each data row is a single event, Y column contains one of the categoritcal values.
Several Y columns - each Y column represent one of the categorical values. you may fill single event per row or the accumulate number of events per row
If you use more than one column the accumulated input data, based on i combination of X: (x₁..x_p), there is also a similar commonly used model is based on single events, therefore each data row is a single event. In this case, every row is a single event and Y_i may be only 1 or 0.

• k+1 number of categorical outcomes. h=(0,1,..,k)
• p number of independent variables. j=(0,1,..,p) 0 for the intersect
• n - sample size or number of X combinations (x₁..x_p) i=(1,..,n).
• y_i,h: the observed occurrences of outcome y(h) for i combination of X.
• ŷ_i,h: the predicted occurrences of outcome y(h) for i combination of X.
• e_i: the total occurrences for i combination of X, e_i=y_i,1+...+y_i,k

H₀: ln(P_h/P₀) = b_0,h
H₁: ln(P_h/P₀) = b_0,h+b_1,hX₁+...+b_p,hX_p

As you probably noticed h=0 is missing above, this because the accumulation of all the probabilities equals one, so you calculate it based on the other probabilities. $$\frac{p_h}{p_0}=e^{t_h} =e^{b_{0,h}+b_{1,h}x_1+...+b_{p,h}x_p}\quad,\quad p_0+p_1+...p_k=1 \quad\Rightarrow\\ p_0=\frac{1}{1+e^{t_1}+..+e^{t_k}}\quad p_h=\frac{e^{t_h}}{1+e^{t_1}+..+e^{t_k}} $$ The maximize log-likelihood method is used instead of the least squares method that is used in the linear regression. The method is based on the Binomial distribution.

Likelihood is the probability that the sample data will occur and the maximize log-likelihood method finds the P_i that will maximize this probability.
$$L=\prod_{i=1}^n \prod_{h=0}^r \hat p_{i,h}^{y_{i,h}}$$ $$LL=ln(L)=\sum_{i=1}^n \sum_{h=0}^r y_{i,h}ln(\hat p_{i,h})$$

Newton's Method

We use the newton's method to find the B vector of parameters that will maximize the Log-likelihood function based on the following iteration formula. delta - The tool will run the following iteration until the changes between each element in B_R+1 matrix and B_R matrix in smaller than delta. The default delta=0.00000001 or until arrived to the Maximum iterations. The default is maximum 15 iterations.
Usually you won't need to change the above parameter since after small number of iterations the delta will be very small. $$ B_{R+1}=B_R+(X'V_RX)^{-1}X'T⊙(P- \hat P_R)$$ R: iteration.
E: e_i vector (totals).
P: p_i vector. (observed probabilities).
p̂: p̂_i vector (predicted probabilities).
B: b_i vector. (coefficients).

$$ B_{R+1}=B_R+H^{-1}X'(Y-\hat Y_R)\\ B=\begin{bmatrix}b_{0,1} &b_{0,2} &...&b_{0,k}\\b_{1,1} &b_{1,2} &...&b_{1,k}\\...& ... & ... & ...\\b_{p,1} &b_{p,2} &...&b_{p,k}\\\end{bmatrix} B_0=\begin{bmatrix}0 &0 &...&0\\0 &0 &...&0\\...& ... & ... & ...\\0 &0 &...&0\\\end{bmatrix}\\\\$$ Matrices orders: X: n*(p+1), B: (p+1)*k
$$X=\begin{bmatrix} &1 &X_{11} &X_{12} & .. &X_{1p} \\ &1 &X_{21} &X_{22} & .. &X_{2p} \\ & : & : & : & : & : \\ &1 &X_{n1} &X_{n2} & .. &X_{np} \end{bmatrix} B=\begin{bmatrix}b_{0,1} &b_{0,2} &...&b_{0,k}\\b_{1,1} &b_{1,2} &...&b_{1,k}\\...& ... & ... & ...\\b_{p,1} &b_{p,2} &...&b_{p,k}\\\end{bmatrix} $$ For any h between 1 and k and i between 1 and n:
$$t_{i,1}=b_{0,1}+b_{1,1}x_{i,1}+...+b_{p,1}x_{i,p}\\ t_{i,2}=b_{0,2}+b_{1,2}x_{i,1}+...+b_{p,2}x_{i,p}\\ ...\\ t_{i,k}=b_{0,k}+b_{1,k}x_{i,1}+...+b_{p,k}x_{i,p}\\ T=XB' \quad$$ T matrix order: n*k
$$T=\begin{bmatrix}t_{1,1} &t_{1,2} &...&t_{1,k}\\...& ... & ... & ...\\t_{2,1} &t_{2,2} &...&t_{2,k}\\t_{n,1} &t_{n,2} &...&t_{n,k}\\\end{bmatrix}$$ There are _k*k W_i1i2 diagonal sub-matrices, each W is a matrix of order: n*n
When h₁=h₂
$$W_{h_1,h_2}=\begin{bmatrix} e_1 \hat p_{1,h_1}(1- \hat p_{1,h_1}) & 0 & 0 & 0\\ 0 & e_2 \hat p_{2,h_1}(1- \hat p_{2,h_1}) & 0 & 0 \\ 0 & 0 & ...& 0\\ 0 & 0 & 0 & e_n \hat p_{n,h_1}(1- \hat p_{n,h_1}) \end{bmatrix}$$ When h₁≠h₂
$$W_{h_1,h_2}=\begin{bmatrix} -e_1 \hat p_{1,h_1}\hat p_{1,h_2} & 0 & 0 & 0\\ 0 & -e_2 \hat p_{2,h_1}\hat p_{2,h_2} & 0 & 0 \\ 0 & 0 & ...& 0\\ 0 & 0 & 0 & -e_n \hat p_{n,h_1}\hat p_{n,h_2} \end{bmatrix}$$ There are k*k H_j1j2 sub-matrices, each sub-matrix H_j1j2 is a matrix of order: (p+1)*(p+1)
$$H_{h_1,h_2}=X'WX=\begin{bmatrix}V_{0,1} &V_{0,2} &...&V_{0,p}\\V_{1,1} &V_{1,2} &...&V_{1,p}\\...& ... & ... & ...\\V_{p,1} &V_{p,2} &...&V_{p,p}\\\end{bmatrix}\\$$ H the matrix of order: k(p+1)*k(p+1) (k*k sub-matrices of order (p+1)(p+1) ) $$H=\begin{bmatrix}H_{1,1} &H_{1,2} &...&H_{1,k}\\H_{2,1} &H_{2,2} &...&H_{2,k}\\...& ... & ... & ...\\H_{k,1} &H_{k,2} &...&H_{k,k}\\\end{bmatrix}\\$$

Numeric Example

Ridge Penalty and Step-Halving

The calculators add two refinements on top of the plain Newton's Method described above. Neither changes the result on well-behaved data like the example above — they only matter when the data is difficult to fit.

Ridge (L₂) penalty. Under More options you can set a small ridge value (0 by default, meaning off). It shrinks the coefficients toward zero, which helps when independent variables are highly correlated with each other or when one outcome category has very few observations — both situations that otherwise make the coefficients unstable or huge.

It works by subtracting a penalty from the log-likelihood before maximizing it, based on all coefficients except the intercepts: $$LL_{penalized}=LL-\frac{1}{2}\cdot ridge\sum_{h,j\ne0} b_{h,j}^2$$ Because the Newton step is built from both the first and second derivative of what is being maximized, the ridge term has to enter both the Hessian and the update itself, or the coefficients would not actually shrink: $$H_{h,h}\mathrel{{+}{=}}ridge \quad\text{(on each non-intercept diagonal entry)}\\ B_{R+1}=B_R+H^{-1}\big(X'(Y-\hat Y_R)-ridge\cdot B_R\big)$$ When ridge = 0 both formulas collapse back to the plain Newton's Method shown above.

Step-halving. Sometimes a predictor is on a very different scale than the others — for example, one column in the thousands and another between 0 and 1. In that case a full Newton step can overshoot and make the fit worse instead of better. Before accepting a step, the tool checks whether the (penalized) log-likelihood actually improved; if it did not, it tries half that step, then a quarter, and so on, up to 12 times, until it finds a step that does improve it. This keeps the iteration moving toward the best fit instead of oscillating or diverging on badly-scaled data.