SVM Margins and Support Vectors

Logistic regression takes a linear score and passes it through a sigmoid function to predict the probability of a class. A support vector machine solves the same kind of linear classification problem from a different angle. Instead of producing a probability, it chooses a decision boundary by looking at the space between that boundary and the training data.

The lesson started with several lines that could separate two classes. More than one line classified the training points correctly, so correct separation alone was not enough to choose between them. An SVM selects the line that stays as far as possible from the closest points on either side.

---

Classifying by the sign of a linear score

The basic linear expression for an SVM has the same form used in logistic regression.

$$f(x)=w^{\mathsf{T}}x+b$$

The decision boundary is where the linear score equals zero.

$$w^{\mathsf{T}}x+b=0$$

A point with a positive score belongs to one class, while a point with a negative score belongs to the other. The basic SVM output is therefore a score relative to the boundary, not a probability between zero and one. The sign of that score determines the final class.

This can be pictured as a step function that assigns a different class to each side of zero. The SVM does not train by differentiating that step function directly. Its optimization is based on the margin between the boundary and the data.

---

A better boundary has a larger margin

When several lines separate the two classes, the SVM measures the distance from each line to its nearest training point. That distance is the margin.

A boundary placed very close to one class may classify the training set correctly, but a small change in a new data point can move it across the line. A boundary with more room on both sides can tolerate a larger change before the predicted class flips.

The SVM searches for the values of $w$ and $b$ that maximize this margin. In the basic linearly separable case, each label can be written as $y_i\in\{-1,1\}$, with the following constraint:

$$y_i(w^{\mathsf{T}}x_i+b)\ge 1$$

Under this constraint, maximizing the margin can be expressed as minimizing the size of the weight vector.

$$\min_{w,b}\frac{1}{2}\lVert w\rVert^2$$

I have not reached the point of solving this optimization problem by hand. The useful idea from this lesson was simpler: among the possible separating lines, choose the one whose nearest training point is farthest away.

---

Support vectors determine the boundary

The training points closest to the decision boundary are called support vectors. These points set the width of the margin and determine where the boundary sits. Points far away from the boundary do not affect it in the same direct way.

Draw several boundaries that separate the classes
  -> measure the closest point to each boundary
  -> choose the boundary with the largest minimum distance
  -> the points that set that distance are the support vectors

The nearest points support the position of the classification boundary. They are the support vectors, and an SVM builds its boundary around them.

---

Logistic regression and SVMs

Both models can produce a linear decision boundary, but they optimize and expose different outputs.

	Logistic regression	SVM
Basic output	Class probability	Score relative to the boundary
Classification rule	Sigmoid probability and threshold	Sign of the linear score
Training focus	Reducing cross-entropy	Maximizing the margin
Data most directly shaping the boundary	Probability error across the data	Support vectors near the boundary

Logistic regression is a direct fit when the output needs to be interpreted as a probability, such as the probability that a message is spam. A basic SVM score is not a probability. Probability calibration can be added separately, but the distinction in this lesson was between probability-based prediction and margin-based classification.

Before deep learning became widespread, SVMs were a standard high-performing choice for many classification problems. At this stage, the most useful lesson was not the full optimization procedure. It was the criterion for a good boundary: separating the classes is only the first requirement, and the distance to the closest data points decides which valid boundary the SVM prefers.