Machine Learning and Linear Algebra

This class started with the definition of machine learning, then moved through linear algebra, calculus, models, cost functions, supervised learning, and unsupervised learning. The volume of new material increased quickly, and I could not absorb all of it in a single pass. I am using this post to review the part I have covered so far.

One step at a time.

The Mathematics Used in Machine Learning

Linear algebra and calculus are used most often in the model-training process, but statistics and probability are part of the same picture.

Field	Main Concepts	Role	Example
Linear algebra	Vectors, matrices, linear transformations	Represents data and weights and performs operations	Input matrix multiplied by a weight vector
Calculus	Rates of change, derivatives, gradients	Finds the direction that reduces the cost	Differentiating the cost with respect to a weight
Statistics	Means, variances, distributions, samples	Describes trends and variation in data	Calculating the mean and standard deviation
Probability	Events, random variables, probability distributions	Expresses uncertainty numerically	Outputting a probability for each class

These four fields work together across EDA, data analysis, preprocessing, and model training. Rather than trying to cover all of the mathematics at once, I focused on the questions I wanted to make clearer first.

Vectors and Matrices

What does it mean for each coordinate of a vector to represent a dimension?

A familiar coordinate plane provides a useful starting point. A point on an x-axis needs one coordinate, while a point on an x-y plane needs two. The pair $(x, y)$ is therefore a two-dimensional vector. Adding more independent coordinates produces an n-dimensional vector, even though dimensions beyond three are difficult to draw directly.

How can apartment size, room count, and location be represented as a vector?

A single apartment could be written conceptually as a four-dimensional row vector such as $[90\text{ m}^2, 3\text{ rooms}, \text{near a station}, \text{7th floor}]$ . This is only an example of collecting four features in one row. Before passing the data to a model, a categorical feature such as proximity to a station must also be encoded numerically.

Matrix definition and notation

A matrix is a rectangular arrangement of numbers or expressions. It can collect multiple vectors and can also represent a linear transformation. Matrices are conventionally written with uppercase letters such as $A$ and $B$ , while vectors are usually written with lowercase letters such as $x$ and $y$ .

Matrix addition and element-wise multiplication

Matrix addition, subtraction, and element-wise multiplication require the two matrices to have the same number of rows and columns. Standard matrix multiplication follows a different rule.

Matrix multiplication

The number of columns in the first matrix must equal the number of rows in the second. A $3 \times 2$ matrix can therefore be multiplied by a $2 \times 4$ matrix because both inner dimensions are 2.

The order also matters. Depending on their shapes, $AB$ may be defined while $BA$ is not. Even when both products are valid, $AB$ applies the transformation represented by $B$ first and then the transformation represented by $A$ . The product $BA$ applies them in the opposite order, so the two results are generally different.

Identity, Transpose, and Inverse Matrices

The identity matrix $I$ has ones on its main diagonal and zeros everywhere else. It is the multiplicative identity for matrices of a compatible size, which gives $AI = IA = A$ .

The transpose $A^{\mathsf{T}}$ exchanges the rows and columns of $A$ . If the rows of a data matrix represent samples and the columns represent features, transposing the matrix swaps those two axes.

Determinant and inverse

Consider the following $2 \times 2$ matrix:

A = \begin{bmatrix} a & b \\ c & d \end{bmatrix}

Its determinant is

\det(A) = ad - bc

When the determinant is not zero, the inverse is

A^{-1} = \frac{1}{ad-bc} \begin{bmatrix} d & -b \\ -c & a \end{bmatrix}

and it satisfies

AA^{-1} = A^{-1}A = I

An inverse matrix plays a role similar to the reciprocal of an ordinary number. Multiplying a nonzero number by its reciprocal gives 1, while multiplying an invertible matrix by its inverse gives the identity matrix $I$ . Deriving the determinant from first principles and then following the elimination steps used to obtain the inverse still feels dense enough to deserve a separate study note.

When $\det(A)=0$ , the inverse does not exist. In the formula above, the determinant appears in the denominator, so a zero determinant would require division by zero.

Expressing the solution to a linear system with an inverse

For this example, assume that $A$ is square and invertible. Starting with $Ax=b$ , multiply both sides by $A^{-1}$ :

\begin{aligned} A^{-1}Ax &= A^{-1}b \\ (A^{-1}A)x &= A^{-1}b \\ Ix &= A^{-1}b \\ x &= A^{-1}b \end{aligned}

Associativity groups the left side so that $A^{-1}A$ becomes the identity matrix $I$ , producing the solution $x=A^{-1}b$ .

Why numerical computing usually avoids calculating the inverse directly

Computers represent real numbers with finite precision, so calculating an inverse introduces rounding error. For a numerically unstable matrix, that error can have a much larger effect on the result. If the only goal is to solve $Ax=b$ , methods based on LU or QR decomposition can solve the system directly without calculating the entire inverse and performing unnecessary work.

Machine-learning problems also do not always use a closed-form solution based on an inverse. Depending on the problem, an optimization method such as gradient descent updates the weights repeatedly instead.

Gaussian-Jordan elimination, a method I vaguely remember encountering in college, transforms a matrix toward the identity by scaling rows and adding one row to another. Processing an $N \times N$ matrix has time complexity $O(N^3)$ . If $N$ is 100,000, the cubic term alone reaches roughly $10^{15}$ operations, which makes direct computation impractical.

Representing a Linear System with Matrices

Several linear equations made from addition and scalar multiplication can be written together in one compact form:

Ax = b

Gaussian-Jordan elimination can solve the system by applying row operations to its augmented matrix. Once the coefficient matrix on the left has been reduced to the identity matrix, the vector on the right contains the values of the unknowns.

The Difference Between a Linear System and Machine Learning

In an ordinary linear system, the coefficient matrix $A$ and result vector $b$ are given, and the goal is to solve for the unknown vector $x$ . When the conditions are appropriate and the system has a unique solution, that $x$ satisfies every equation exactly.

Machine learning uses a similar shape with different roles. The input matrix $X$ and the observed targets $y$ are given, while the unknown is the weight vector $w$ that describes how strongly each feature contributes to the prediction. Real data contains noise, and one expression often cannot fit every sample exactly. Training therefore searches for the $w$ that reduces a cost function and explains the dataset as well as possible.

How Machine Learning Uses Matrices

Suppose a house-price dataset contains $n$ samples and $d$ features for each property. The complete input can be represented by an $n \times d$ matrix $X$ . Each row represents one property, while the columns hold features such as size, room count, and a location score.

X = \begin{bmatrix} x_{11} & x_{12} & \cdots & x_{1d} \\ x_{21} & x_{22} & \cdots & x_{2d} \\ \vdots & \vdots & \ddots & \vdots \\ x_{n1} & x_{n2} & \cdots & x_{nd} \end{bmatrix}

If the weight vector $w$ contains the influence of each feature, the predictions for all samples can be written as

\hat{y} = Xw + b

The dot product of one row of $X$ and $w$ , plus the bias $b$ , produces the predicted price for one property. Multiplying the full matrix $X$ by $w$ performs the same calculation for every sample at once. Here, $X$ is the input matrix containing all samples, while a lowercase $x$ represents the feature vector for one sample.

$\hat{y}$ is read as “y-hat.” The hat marks an estimated value rather than an observed one. The house price recorded in the training data is therefore $y$ , while the price calculated by the model is $\hat{y}$ . In this example, the house price is the target or label.

Where Calculus Enters

Multiplying the input matrix $X$ by the weight vector $w$ produces the model’s predictions. The next problem is deciding how to change $w$ so that the difference between those predictions and the observed values becomes smaller.

Calculus provides the rate of change of the cost function with respect to the weights. I covered the path from partial derivatives to the gradient vector and gradient descent in a separate note.

From Loss Functions to Gradient Descent →

This is where I stopped for the day. I still have the remaining material to organize, but writing out the relationships between vectors, matrices, determinants, and inverses has made the current boundary of my understanding explicit.