Logistic Regression and Decision Boundaries

This was one of the topics I had left unfinished because I could not fully digest it at the time. Enough time had passed that I could barely follow the notes I had typed during the live class. I searched through the concepts again and worked through the related Codeit lessons before reorganizing the material. I would not say that writing it down means I understand everything yet, but keeping the notes visible and revising them as I revisit the topic should help the learning stick.

The Basic Structure of Logistic Regression

The name logistic regression makes it sound like a regression model that predicts a continuous number. In practice, it is used for classification problems with outcomes such as spam or not spam, or a malignant or benign tumor.

Linear regression predicts a number directly from a linear expression. Logistic regression first calculates a linear score and then converts that score into a probability. The instructor described it in two stages: find a linear boundary that separates the data, then estimate the probability of a class relative to that boundary.

Why is it called regression if it performs classification?

Logistic regression does not directly output a category such as spam or not spam. It first calculates a number such as a 0.92 or 0.37 probability that an email is spam. The name includes regression because this probability is calculated first.

The probability itself is not the final answer. With a threshold of 0.5, a probability of 0.5 or higher becomes spam, while a lower probability becomes not spam. The task it ultimately solves is therefore classification.

Email features
  → calculate spam probability
  → compare it with the threshold
  → classify the email as spam or not spam

For linear regression, a number such as a house price of 820 million won is the final result. Logistic regression calculates a number such as a spam probability of 0.92 and then uses it to select a category. Its name contains regression, but the machine learning task it solves is classification.

Why not use linear regression directly for classification?

It is possible to classify 0 and 1 by applying a threshold of 0.5 to a linear regression prediction. The problem is that linear regression does not restrict its output range, and a point far outside the main input range can move both the fitted line and the resulting classification threshold.

Consider the following input $x$ and label $y$ :

$x$	1	2	3	4
$y$	0	0	1	1

Ordinary least squares fits this line:

\hat{y}=0.4x-0.5

Using 0.5 as the classification threshold places the boundary at $x=2.5$ , which separates all four points correctly. However, the prediction is -0.1 at $x=1$ and 1.1 at $x=4$ , already outside the valid probability range from 0 to 1.

Now add the positive example $(20,1)$ . Extending the original line to $x=20$ produces a prediction of 7.5, far from the label 1. Because linear regression minimizes squared error, it flattens the line substantially to reduce that large residual. Refitting the model gives approximately:

\hat{y}\approx0.036x+0.384

The 0.5 boundary now moves to about $x=3.22$ . The prediction for $x=3$ , whose true label is 1, falls to about 0.492 and is incorrectly classified as 0. In this context, sensitivity to a large value does not mean that the label $y$ is large. It means that an input such as $x=20$ , far from the other observations, has substantial influence over the slope and intercept.

Logistic regression passes its linear score through the sigmoid function, keeping predictions between 0 and 1. Once a positive example lies far enough on the positive side of the boundary, its predicted probability approaches 1 and its loss becomes small. The model does not need to flatten its boundary merely to force that point’s numeric output to equal 1.

Logistic regression is not completely immune to outliers or incorrect labels. It is nevertheless better suited to classification than attaching an arbitrary threshold to linear regression because its output is a valid probability and its loss function is designed for classification.

What the Linear Score z Means

Given an input feature vector $x$ , a weight vector $w$ , and a bias $b$ , the linear result is:

z = w^{\mathsf{T}}x + b

With two features, the same expression becomes:

z = w_1x_1 + w_2x_2 + b

The points satisfying $z=0$ form the boundary between the two classes.

w_1x_1 + w_2x_2 + b = 0

With one feature

Suppose study time $x$ is used to predict whether a student passes an exam. If the trained logistic regression model is:

\hat{y}=\sigma(-4.7+0.1x)

then a threshold of 0.5 places the decision boundary where $\hat{y}=0.5$ . The sigmoid function outputs 0.5 when its input is 0, so the boundary can be found by solving:

-4.7+0.1x=0

x=47

The model therefore assigns a passing probability of 0.5 at 47 hours of study. Because the coefficient of $x$ is positive, values above 47 produce probabilities greater than 0.5 and are classified as passing, while lower values are classified as failing. The value 47 is not chosen manually; it is calculated from the learned weight and bias.

With two features

Now let $x_1$ be study time and $x_2$ be the score on a practice exam. Assume the trained model has the following linear score:

z=-100+2x_1+x_2

With a threshold of 0.5, the decision boundary satisfies $z=0$ :

-100+2x_1+x_2=0

Solving for $x_2$ gives:

x_2=-2x_1+100

This equation is not asking for one unique pair of values for $x_1$ and $x_2$ . After training, the parameters -100, 2, and 1 are fixed. Every combination of study time and practice-exam score that satisfies the equation lies on the same line.

For example, the boundary contains a practice-exam score of 60 when study time is 20 hours, and a score of 40 when study time is 30 hours. Connecting those points gives the decision boundary. In this example, points above the line have $z>0$ and a passing probability greater than 0.5, while points below it have $z<0$ and are more likely to be classified as failing.

With three features, $w_1x_1+w_2x_2+w_3x_3+b=0$ forms a plane in three-dimensional space. With more features, it forms a hyperplane that cannot be drawn directly. Logistic regression on the original features produces a linear boundary, but adding polynomial features can create a curved boundary.

The value $z$ is the model’s linear score for input $x$ . Its sign determines which side of the decision boundary contains the input. When $z=0$ , the input lies on the boundary.

The signed perpendicular distance from $x$ to that boundary is:

\frac{w^{\mathsf{T}}x+b}{\lVert w\rVert} = \frac{z}{\lVert w\rVert}

The score $z$ itself is not a geometric distance. Multiplying both $w$ and $b$ by the same constant leaves the decision boundary unchanged but changes the magnitude of $z$ . Dividing by the norm of the weight vector, $\lVert w\rVert$ , produces the perpendicular distance. The sign of that result still identifies the side of the boundary.

A decision boundary is not unique to logistic regression. Any line or surface that a classification model uses to separate classes is a decision boundary.

Converting the Score into a Probability

The linear score $z$ can range from negative infinity to positive infinity. The sigmoid function maps that value into the interval from 0 to 1.

\sigma(z) = \frac{1}{1+e^{-z}}

Its output changes with $z$ as follows.

Linear score	Sigmoid output
$z < 0$	Less than 0.5
$z = 0$	0.5
$z > 0$	Greater than 0.5
$z \to -\infty$	Approaches 0
$z \to +\infty$	Approaches 1

The logistic regression hypothesis applies the sigmoid function to the linear score.

\hat{y} = \sigma(w^{\mathsf{T}}x+b)

The actual label $y$ is either 0 or 1, while the prediction $\hat{y}$ is a probability between 0 and 1. If $\hat{y}=0.9$ , the model assigns a probability of 0.9 to the positive class for that input.

A threshold converts the probability into a final class. With a threshold of 0.5, a probability of 0.5 or greater becomes class 1, while a lower probability becomes class 0. Since $\sigma(0)=0.5$ , $z=0$ is the decision boundary under that threshold. Changing the threshold can therefore change the final classification even when the predicted probability remains the same.

Logistic regression does not use the linear score itself as the final answer. It passes the score through the sigmoid function, converts it into a probability, and compares that probability with a threshold. The learned weights and bias define the decision boundary, and the final class depends on which side of that boundary contains the input.

I also separated the linear score $z$ from the actual geometric distance to the boundary. The equations are not yet automatic for me, but working from a single-feature cutoff to a two-feature line made the meaning of a decision boundary much clearer.