Matrices and vectors

What a matrix is, how two of them add and multiply, what a vector's length and direction are, and which directions a matrix only stretches.

August 31, 2026

21 minutes read

4270 words

Contents26

A vector is a column of numbers. A matrix is a rectangle of them, so a vector is the case of a single column. Adding either works one position at a time. Multiplying two matrices builds each entry of the answer from a whole row and a whole column. That one rule is what the rest of the subject rests on.

Assumed knowledge

Arithmetic. Addition, subtraction and multiplication of ordinary numbers, and what a fraction is.

Nothing else. No page here needs to be read first.

Ground covered

Vectors first. What one is, how two are added, what scaling does to one, the length of one and the unit vector along it. The dot product of two, and the test it gives for a right angle.

Matrices second, as the same idea with more than one column. The shape, how an entry is named, addition and scaling again. Multiplication, which does not work position by position, and the rule deciding which shapes can be multiplied at all. Why the order of a product matters. The identity, the inverse that takes the place of division, and the transpose.

Last, what a square matrix does to the plane and to space. The directions it only stretches, which are its eigenvectors. The reflection built from a single vector, how to apply it without writing it down, and why it keeps length. The rotation, first in the plane and then about an axis, and the family both of them belong to.

A reader who can already say why a 2 by 3 matrix times a 3 by 2 matrix gives a 2 by 2 result can skip the page.

Vectors

A vector is a list of numbers written in a column inside brackets. Its entries are named from the top down, so $v_1$ is the first of them.

v = \begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{bmatrix} \tag{1}

A vector of two entries is drawn as an arrow from the origin to the point with those coordinates. One of three entries is an arrow in space. The count of entries is the vector’s dimension. Nothing stops it being larger than three, and only the drawing stops.

Adding and scaling vectors

Entries in matching positions are added, which needs the two vectors to have the same number of them. The answer has that many entries again.

v + w = \begin{bmatrix} v_1 + w_1 \\ v_2 + w_2 \\ \vdots \\ v_n + w_n \end{bmatrix} \tag{2}

Multiplying a vector by a single ordinary number multiplies every entry by it. A single number is called a scalar in this context, to separate it from the vector.

k v = \begin{bmatrix} k v_1 \\ k v_2 \\ \vdots \\ k v_n \end{bmatrix} \tag{3}

Drawn as arrows, both rules have a shape. Adding two vectors adds their coordinates. Carrying the second arrow to the tip of the first does exactly that. The carried arrow keeps its length and its direction, and only its starting point moves. The sum runs from the origin to where the carried arrow ends.

Addition on the left, scaling on the right. On the right the arrow sweeps through k = 2, 1.5, 0.5 and -1, collapsing to a point as k passes zero and coming back the other way. The animation is CSS inside the drawing, so nothing runs to play it.

Order does not matter here, so $v + w$ and $w + v$ reach the same point. Carrying $v$ to the tip of $w$ traces the other two sides of the same parallelogram.

Scaling by $k$ multiplies both coordinates by $k$ , which changes the length of the arrow and leaves its direction alone. A $k$ above 1 lengthens it, and a $k$ between 0 and 1 shortens it. A negative $k$ reverses it, because both coordinates change sign at once.

Every multiple of one vector therefore lies on a single line through the origin. The next section picks out the point on that line at distance 1.

Length of a vector

A column vector of two entries can be drawn as an arrow from the origin to the point with those coordinates. Its length is written $|v|$ , and Pythagoras gives it.

|v| = \sqrt{v_1^2 + v_2^2 + \cdots + v_n^2} \tag{4}

The same formula holds for any number of entries, though only two or three of them can be drawn. It is the modulus of a complex number, written for a list rather than for a pair.

The vector (3, 4) has length 5. The short arrow along the same direction has length 1.

The vector $(3, 4)$ has length $\sqrt{9 + 16}$ , which is 5.

Unit vectors

A vector of length 1 is a unit vector. Dividing any vector by its own length gives one, and the result is written $\hat{v}$ .

\hat{v} = \frac{v}{|v|} \tag{5}

Dividing here means scaling by $1/|v|$ , which is equation (3) with $k = 1/|v|$ . Every entry is divided by the same number, so the direction is untouched and only the length changes.

Dividing $(3, 4)$ by 5 gives $(0.6, 0.8)$ , the short arrow in the figure above. That step is called normalising the vector, and a unit vector is often called a normalised one.

Dot product

Two vectors with the same number of entries can be multiplied to give a single ordinary number. Multiply the entries in matching positions, then add the results.

v \cdot w = v_1 w_1 + v_2 w_2 + \cdots + v_n w_n \tag{6}

The answer is one ordinary number, however many entries the two vectors have. It is written with a raised dot, and it is also called the scalar product for that reason.

The same quantity has a geometric value. It is the two lengths multiplied, times the cosine of the angle between the vectors.

v \cdot w = |v| \, |w| \cos\theta \tag{7}

The dot product read two ways. On the left it is 7, and the angle is 57.5 degrees. On the right it is 0, and the angle is a right angle.

Setting $w = v$ gives $v \cdot v = |v|^2$ , because the angle is then zero and its cosine is 1. So equation (4) can also be written $|v| = \sqrt{v \cdot v}$ .

Orthogonal directions

The cosine of a quarter turn is zero. Two vectors at a right angle therefore have a dot product of zero, and a dot product of zero means a right angle.

v \cdot w = 0 \tag{8}

The right-hand panel above is such a pair. The vectors $(4, 1)$ and $(-1, 4)$ give $-4 + 4$ , which is zero.

Orthogonal is the mathematical word for perpendicular, and it is the one used from here on. A set of unit vectors that are all orthogonal to each other is called orthonormal.

A vector orthogonal to a line, a plane or a surface is called a normal to it. In three dimensions one normal vector fixes a whole plane through the origin. That plane is every vector whose dot product with the normal is zero. The word normal means orthogonal here. Normalising is the separate operation of equation (5), and the two share only their spelling.

Rows and columns

A matrix is a rectangular block of numbers written inside brackets. Its horizontal lines are rows and its vertical lines are columns.

The shape of a matrix is its number of rows and its number of columns, in that order. A matrix with 3 rows and 4 columns is a 3 by 4 matrix, written $3 \times 4$ . The shape is also called its dimensions or its size.

One entry is named by the row and the column it sits in. The entry in row $i$ and column $j$ of a matrix $A$ is written $a_{ij}$ , with the row first. So $a_{23}$ sits in row 2, column 3.

A 3 by 4 matrix. Row 2 and column 3 meet in one entry, and that entry is the one written with the subscript 2 3.

A = \begin{bmatrix} a_{11} & a_{12} & a_{13} & a_{14} \\ a_{21} & a_{22} & a_{23} & a_{24} \\ a_{31} & a_{32} & a_{33} & a_{34} \end{bmatrix} \tag{9}

Two shapes have their own names. A matrix with the same number of rows as columns is square. A matrix with a single column is a column vector, which is the vector of equation (1) written as an $n$ by 1 matrix. One with a single row is a row vector.

Adding and scaling matrices

Equations (2) and (3) extend to a rectangle without changing. Two matrices of the same shape are added by adding the entries that sit in the same position, and the answer has that same shape.

(A + B)_{ij} = a_{ij} + b_{ij} \tag{10}

Addition, position by position. The 5 and the 1 sit in the same place, so their sum sits in that place too.

Subtraction works the same way, with the entries subtracted rather than added.

Shapes that differ cannot be added at all. There is no answer for a 2 by 3 matrix plus a 3 by 2 matrix, because the positions do not correspond.

Scaling works the same way. Multiplying a matrix by a single number multiplies every entry by it.

(kA)_{ij} = k \, a_{ij} \tag{11}

The shape is unchanged. Scaling by $-1$ negates every entry, which is how subtraction can be written as $A + (-1)B$ .

Multiplication

Multiplying two matrices does not work position by position. Each entry of the answer is built from a whole row of the left matrix and a whole column of the right one.

Take row $i$ of $A$ and column $j$ of $B$ . Both are lists of numbers. Multiply the first of one by the first of the other, then the second by the second, taking each pair in turn. Add the products. The total is the entry in row $i$ , column $j$ of the answer.

c_{ij} = \sum_{k} a_{ik} \, b_{kj} \tag{12}

The symbol $\sum$ means add up the terms that follow it, once for each value of $k$ . Here $k$ counts along the row and down the column at the same time, one step in each per term.

Row 2 is spent against column 1, and what they make between them is the entry where that row and that column meet.

Working the marked entry gives $4 \times 7 + 5 \times 9 + 6 \times 11$ , which is $28 + 45 + 66$ , or 139.

The rule only works when the row and the column are the same length. The number of columns of the left matrix must equal the number of rows of the right one. Those two counts vanish in the answer, and the two that remain are its shape.

(m \times n) \; \cdot \; (n \times p) \;=\; (m \times p) \tag{13}

So a 2 by 3 matrix times a 3 by 2 matrix gives a 2 by 2 matrix. The same two matrices in the other order give a 3 by 3 matrix instead. When the inner two counts differ, the product does not exist.

Order matters

Swapping the two factors of a product of ordinary numbers changes nothing, because $3 \times 5$ and $5 \times 3$ are both 15. Matrices do not behave that way, and $AB$ and $BA$ are usually different matrices.

\begin{bmatrix} 1 & 2 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0 \\ 1 & 1 \end{bmatrix} = \begin{bmatrix} 3 & 2 \\ 1 & 1 \end{bmatrix}, \qquad \begin{bmatrix} 1 & 0 \\ 1 & 1 \end{bmatrix} \begin{bmatrix} 1 & 2 \\ 0 & 1 \end{bmatrix} = \begin{bmatrix} 1 & 2 \\ 1 & 3 \end{bmatrix} \tag{14}

Both products exist and both are 2 by 2. They disagree in three of their four entries. So the words times and by are not enough on their own, and $AB$ is read as $A$ times $B$ on the right. The order is part of the statement.

Matrix times a vector

Equation (4) needs no change when the right-hand factor has a single column. The result is another single column.

Each entry of the answer is one row of the matrix spent against the whole input. That makes it a weighted sum of the inputs, with the row supplying the weights.

\begin{bmatrix} 2 & 0 & 1 \\ 1 & 3 & 0 \\ 0 & 1 & 4 \end{bmatrix} \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix} = \begin{bmatrix} 2(1) + 0(2) + 1(3) \\ 1(1) + 3(2) + 0(3) \\ 0(1) + 1(2) + 4(3) \end{bmatrix} = \begin{bmatrix} 5 \\ 7 \\ 14 \end{bmatrix} \tag{15}

Read that way, a matrix is a table of weights. Row $i$ says how much each input contributes to output $i$ , and a zero says that input contributes nothing to that output.

Identity matrix

One square matrix carries 1 down its main diagonal from the top left and 0 everywhere else. That matrix is the identity, written $I$ , or $I_n$ when its size needs saying.

I_3 = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}, \qquad AI = IA = A \tag{16}

It does to matrices what 1 does to ordinary numbers. Reading it through equation (15) says why. Row $i$ has a single 1, in position $i$ , so output $i$ is input $i$ and nothing else.

Division and inverses

There is no division of matrices. Nothing is written $A/B$ , and no rule divides one rectangle of numbers by another.

What takes its place is multiplication by an inverse. The inverse of a square matrix $A$ is the matrix $A^{-1}$ that returns the identity when multiplied by $A$ , in either order.

A A^{-1} = A^{-1} A = I \tag{17}

Because order matters, dividing by $A$ has two meanings, and they are $A^{-1}B$ and $BA^{-1}$ . Writing $B/A$ would not say which.

For a 2 by 2 matrix the inverse is given by a formula. The number $ad - bc$ is the determinant, written $\det A$ .

A = \begin{bmatrix} a & b \\ c & d \end{bmatrix}, \qquad A^{-1} = \frac{1}{ad - bc} \begin{bmatrix} d & -b \\ -c & a \end{bmatrix} \tag{18}

The inverse of $\begin{bmatrix} 2 & 1 \\ 1 & 1 \end{bmatrix}$ has determinant $2 - 1 = 1$ , so it is $\begin{bmatrix} 1 & -1 \\ -1 & 2 \end{bmatrix}$ . Multiplying the two gives $I$ .

Not every square matrix has an inverse. A determinant of zero leaves equation (18) dividing by zero, and no inverse exists. The matrix $\begin{bmatrix} 1 & 2 \\ 2 & 4 \end{bmatrix}$ is one such, because its second row is twice its first and its determinant is $4 - 4 = 0$ . A matrix without an inverse is called singular.

Transpose

The transpose of $A$ is written $A^{\mathsf{T}}$ . It is the matrix whose rows are the columns of $A$ , so the entry at row $i$ , column $j$ comes from row $j$ , column $i$ .

(A^{\mathsf{T}})_{ij} = a_{ji} \tag{19}

Transposing turns an $m \times n$ matrix into an $n \times m$ one. The 2 by 3 matrix of the multiplication figure becomes a 3 by 2 matrix, with rows $(1, 4)$ , $(2, 5)$ and $(3, 6)$ .

A column vector transposes into a row vector. That gives the dot product of equation (6) a second spelling. A 1 by $n$ row times an $n$ by 1 column is a 1 by 1 answer, which is one number.

v \cdot w = v^{\mathsf{T}} w \tag{20}

One vector $v$ can also stand on both sides of a product, as in $v v^{\mathsf{T}}$ . Taken in that order it is $n$ by 1 times 1 by $n$ , so equation (13) makes it a square matrix rather than a number.

Eigenvalues and eigenvectors

A matrix sends most directions somewhere else. A few directions come back pointing exactly where they went in, longer or shorter but not turned.

Such a direction is an eigenvector of the matrix. The factor it is scaled by is the matching eigenvalue, written $\lambda$ .

A v = \lambda v, \qquad v \neq 0 \tag{21}

The condition $v \neq 0$ is there because the vector of all zeros satisfies the equation for every $\lambda$ and says nothing.

The matrix sends every direction of length 1 to the oval. Two directions come back along the line they started on, and those two are its eigenvectors.

The matrix drawn there is $\begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}$ . Its eigenvalues are 3 and 1, with eigenvectors along $(1, 1)$ and $(1, -1)$ .

An $n$ by $n$ matrix has at most $n$ eigenvalues. An eigenvalue of 1 leaves its direction alone. A negative eigenvalue reverses the direction as well as scaling it. An eigenvalue of 0 flattens its direction to nothing, and a matrix with one is singular, which is the condition of equation (18) seen another way.

Reflection matrices

A column vector times its own transpose, written $\hat{v}\hat{v}^{\mathsf{T}}$ , is called an outer product. Subtracting twice it from the identity gives a reflection.

H = I - 2\,\hat{v}\hat{v}^{\mathsf{T}} \tag{22}

The eigenvalues say what $H$ does. Along $\hat{v}$ it returns $-\hat{v}$ , an eigenvalue of $-1$ , because $\hat{v}^{\mathsf{T}}\hat{v}$ is 1. Any $w$ orthogonal to $\hat{v}$ has $\hat{v}^{\mathsf{T}}w = 0$ by equation (8), so $Hw = w$ , an eigenvalue of 1. One direction is reversed and everything orthogonal to it is untouched, which is what a mirror does.

Taking $\hat{v} = (1, 1)/\sqrt{2}$ in two dimensions gives a matrix of whole numbers.

H = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} - 2 \begin{bmatrix} 0.5 & 0.5 \\ 0.5 & 0.5 \end{bmatrix} = \begin{bmatrix} 0 & -1 \\ -1 & 0 \end{bmatrix} \tag{23}

That $H$ sends $(1, 1)$ to $(-1, -1)$ and leaves $(1, -1)$ alone. A matrix of this form is a Householder reflection.

Applying a reflection

The matrix $H$ of equation (22) has $n^2$ entries, and multiplying a vector by it the ordinary way takes $n^2$ multiplications. There is no need to build it at all. Regrouping the product moves the brackets one step to the right.

H x = x - 2\,\hat{v}\,(\hat{v}^{\mathsf{T}} x) = x - 2\,(\hat{v} \cdot x)\,\hat{v} \tag{24}

The bracket is a single ordinary number, the dot product of equation (6). The reflection of $x$ is that number, doubled, times $\hat{v}$ , subtracted from $x$ . One dot product and one scaled subtraction do it, which is $2n$ multiplications in all, and the only thing stored is $\hat{v}$ . For $n = 6$ that is twelve multiplications rather than thirty-six. Moving between two reflections then means moving between two vectors of six entries.

Length and inversion

The length of $Hx$ equals the length of $x$ . Squaring equation (24), and writing $d$ for the number $\hat{v} \cdot x$ , gives three terms.

|Hx|^2 = |x|^2 - 4d\,(\hat{v} \cdot x) + 4d^2\,|\hat{v}|^2 = |x|^2 - 4d^2 + 4d^2 = |x|^2 \tag{25}

The last step uses $|\hat{v}| = 1$ . A mirror image is as long as the original.

Applying $H$ twice gives back $x$ . Multiplying equation (22) by itself and collecting terms gives the identity.

H H = I - 4\hat{v}\hat{v}^{\mathsf{T}} + 4\hat{v}\,(\hat{v}^{\mathsf{T}}\hat{v})\,\hat{v}^{\mathsf{T}} = I - 4\hat{v}\hat{v}^{\mathsf{T}} + 4\hat{v}\hat{v}^{\mathsf{T}} = I \tag{26}

Again the step uses $\hat{v}^{\mathsf{T}}\hat{v} = 1$ . Reflecting a reflection returns the original, so $H$ is its own inverse. The matrix $H$ is symmetric as well, so $H^{\mathsf{T}} H = H H = I$ .

Choosing the vector

Any unit vector gives a reflection, so there is one for every direction. The balanced direction $\hat{v} = (1, 1, \dots, 1)/\sqrt{n}$ gives $H = I - \tfrac{2}{n}J$ , where $J$ is the matrix whose every entry is 1. Each output is then $(1 - \tfrac2n)$ of its own input, minus $\tfrac2n$ of each of the others. That is the most even mixing a reflection can do. The direction along one axis, $\hat{v} = (1, 0, \dots, 0)$ , gives a matrix with $-1$ in the first place of its diagonal and 1 in the rest. That one flips a single entry and mixes nothing. Moving $\hat{v}$ from one of those directions to the other moves the matrix from mixing everything to mixing nothing.

Rotation in two dimensions

Turning the plane about the origin by an angle $\theta$ is a matrix. Its two columns are where the two axis directions land, which is equation (15) read backwards.

R(\theta) = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix} \tag{27}

Both columns have length 1, and their dot product is $-\cos\theta\sin\theta + \sin\theta\cos\theta$ , which is 0. So the columns are orthonormal by equation (8). A square matrix whose columns are orthonormal is called an orthogonal matrix, and a rotation is one.

A rotation moves a vector along the circle it already sits on. In three dimensions the turn needs an axis as well as an angle, and the axis is what stays still.

Nothing is stretched, because every column is a unit vector. The determinant is 1 rather than $-1$ , which is what separates a rotation from a reflection: both keep lengths, and only the reflection turns the plane over.

Rotation in three dimensions

An axis is needed here as well as an angle. Turning about the $z$ axis leaves the third coordinate untouched and does equation (27) to the first two. The two by two block therefore sits in the top left, and the rest is the identity.

R_z(\theta) = \begin{bmatrix} \cos\theta & -\sin\theta & 0 \\ \sin\theta & \cos\theta & 0 \\ 0 & 0 & 1 \end{bmatrix} \tag{28}

Turning about $x$ or about $y$ is the same three by three matrix with the block moved to the rows and columns that turn.

R_x(\theta) = \begin{bmatrix} 1 & 0 & 0 \\ 0 & \cos\theta & -\sin\theta \\ 0 & \sin\theta & \cos\theta \end{bmatrix}, \qquad R_y(\theta) = \begin{bmatrix} \cos\theta & 0 & \sin\theta \\ 0 & 1 & 0 \\ -\sin\theta & 0 & \cos\theta \end{bmatrix} \tag{29}

All three are orthogonal and all three have determinant 1. Two turns one after another are a single matrix, their product, and the order is part of the statement exactly as equation (14) says. Turning about $x$ and then about $z$ does not land where turning about $z$ and then about $x$ does.

Undoing a rotation needs no work at all. Multiplying $R^{\mathsf{T}}R$ takes the dot product of every pair of columns, and those are orthonormal, so the answer is the identity of equation (16). The inverse of a rotation is its transpose.

Orthogonal matrices and length

A square matrix $Q$ with $Q^{\mathsf{T}} Q = I$ is orthogonal, which is the definition the rotations met above. Reflections satisfy it too, by equation (26). Every such matrix keeps lengths, for the reason equation (25) gave in its own case.

|Qx|^2 = (Qx)^{\mathsf{T}}(Qx) = x^{\mathsf{T}} Q^{\mathsf{T}} Q\, x = x^{\mathsf{T}} x = |x|^2 \tag{30}

Dot products survive as well, so $Qx \cdot Qy = x \cdot y$ , and the angle between two vectors comes through unchanged. Take a vector holding several signals. The total energy of those signals is the squared length of the vector, and an orthogonal matrix leaves it alone. That is why matrices of this kind mix the delay lines of a reverb. The mixing shares the sound out among the lines and neither adds energy nor removes it.

Figure sources

Seven of the eight drawings come from one script kept beside this page. One helper draws a matrix as a bracketed grid and tints the cells a figure is about. Another rules the paper for the ones that draw arrows. That is why each group shares a look. The script checks its own arithmetic against NumPy before drawing it, so a wrong entry cannot reach the page. It needs NumPy and Matplotlib.

matrices.py

anim.py writes the animated pair. It emits the SVG itself rather than going through Matplotlib, because the animation is CSS and Matplotlib has no way to write any. Every rule in it is scoped to the drawing’s own root, so nothing it defines reaches the page around it. It needs nothing beyond Python.

anim.py

uv run --with numpy --with matplotlib python3 matrices.py
python3 anim.py