Linear Algebra Cheat Sheet

Introduction

Linear algebra is a branch of mathematics that deals with vectors, vector spaces, linear transformations, and systems of linear equations. It forms the foundation of many fields including machine learning, physics, engineering, and computer graphics. This cheat sheet provides a quick reference for the fundamental concepts and formulas in linear algebra.

Vectors

A vector is an ordered collection of numbers. In , a vector v can be written as:

Vector Operations

Vector Addition

Scalar Multiplication

Dot Product

Vector Norm

Unit Vector

Vector Projections

Orthogonal Components

Matrices

A matrix is a rectangular array of numbers arranged in rows and columns. An mn matrix A can be written as:

Matrix Operations

Matrix Addition

Scalar Multiplication

Matrix Multiplication

Matrix Transpose

Matrix Inverse

Trace

Properties of Matrix Operations

• (A + B) + C = A + (B + C) [Associative]
• A + B = B + A [Commutative]
• (AB)C = A(BC) [Associative]
• A(BC) = (AB)C [Associative]
• (AB) = BA
• (A) = A
• (AB) = BA
• (A) = (A)

Determinants

The determinant is a scalar value that can be computed from the elements of a square matrix and encodes certain properties of the matrix.

Determinant for Common Matrices

22 Matrix

33 Matrix

Determinant Properties

• det(I) = 1 [Identity matrix]
• det(AB) = det(A) det(B)
• det(A) = det(A)
• det(A) = 1/det(A)
• det(A) = det(A) [for an nn matrix]
• If any row or column is zero, det(A) = 0
• If two rows or columns are identical, det(A) = 0
• det(A) = 0 if A is singular (not invertible)

Systems of Linear Equations

A system of linear equations can be represented in matrix form as Ax=b, where A is the coefficient matrix, x is the vector of unknowns, and b is the vector of constants.

Solving Methods

Gaussian Elimination (Row Reduction)

1. Transform the augmented matrix [A|b] to row echelon form.
2. Use back substitution to solve for unknowns.

Gauss-Jordan Elimination

1. Transform the augmented matrix [A|b] to reduced row echelon form.
2. Read the solution directly.

Cramer's Rule

For a system Ax=b with n equations and n unknowns, if det(A) 0, then:

where A is A with the i-th column replaced by b.

Existence and Uniqueness of Solutions

• Unique solution: det(A) 0 and rank(A) = n
• Infinite solutions: det(A) = 0, rank(A) = rank([A|b])
• No solution: rank(A) rank([A|b])

Vector Spaces and Subspaces

A vector space V over a field F is a set of vectors with two operations (addition and scalar multiplication) that satisfy specific axioms.

Vector Space Axioms

• Closed under addition: u, v V u + v V
• Closed under scalar multiplication: v V, F v V
• Commutativity of addition: u + v = v + u
• Associativity of addition: (u + v) + w = u + (v + w)
• Additive identity: 0 V such that v + 0 = v
• Additive inverse: v V, (-v) V such that v + (-v) = 0
• Multiplicative identity: 1v = v
• Distributivity: (u + v) = u + v and ( + )v = v + v
• Associativity of scalar multiplication: (v) = ()v

Subspace

A subset W of V is a subspace if it:

• Contains the zero vector
• Is closed under vector addition
• Is closed under scalar multiplication

Common Subspaces

• Null space (kernel): N(A) = {x : Ax = 0}
• Column space (range): C(A) = {Ax : x }
• Row space: R(A) = {Ax : x }
• Left null space: N(A) = {y : Ay = 0}

Basis and Dimension

Linear Independence

A set of vectors {v, v, ..., v} is linearly independent if:

Basis

A basis for a vector space V is:

• A linearly independent set of vectors in V
• That spans V

Dimension

The dimension of a vector space V, denoted dim(V), is the number of vectors in any basis for V.

Rank-Nullity Theorem

where A is an mn matrix, rank(A) is the dimension of the column space, and nullity(A) is the dimension of the null space.

Linear Transformations

A linear transformation T: V W is a function satisfying:

• T(u + v) = T(u) + T(v) [Additivity]
• T(v) = T(v) [Homogeneity]

Matrix Representation

If T: is a linear transformation, then T(x) = Ax for some unique mn matrix A, where the columns of A are T(e), T(e), ..., T(e).

Kernel and Range

• Kernel (null space): Ker(T) = {v V : T(v) = 0}
• Range (image): Rng(T) = {T(v) : v V}

Properties

• T is one-to-one (injective) iff Ker(T) = {0}
• T is onto (surjective) iff Rng(T) = W
• T is invertible iff T is both one-to-one and onto

Eigenvalues and Eigenvectors

Definition

For a square matrix A, a nonzero vector v is an eigenvector with eigenvalue if:

Characteristic Equation

Finding Eigenvalues

1. Set up the characteristic equation det(A - I) = 0
2. Solve for

Finding Eigenvectors

1. For each eigenvalue , solve (A - I)v = 0
2. The nonzero solutions are the eigenvectors

Properties

• The trace of A equals the sum of eigenvalues: tr(A) =
• The determinant of A equals the product of eigenvalues: det(A) =
• The eigenvalues of A are 1/ (if A is invertible)
• The eigenvalues of A are the same as A

Diagonalization

A matrix A is diagonalizable if A = PDP, where D is a diagonal matrix with eigenvalues on the diagonal, and P is a matrix whose columns are the corresponding eigenvectors.

Orthogonalization and Projections

Gram-Schmidt Orthogonalization

Given linearly independent vectors {v, v, ..., v}:

1. u = v
2. u = v - projv
3. u = v - projv - projv
4. ... continue for each vector

Orthogonal Matrices

A square matrix Q is orthogonal if QQ = QQ = I, meaning its columns form an orthonormal set.

Projections

The projection of vector b onto a subspace spanned by the columns of matrix A is:

Singular Value Decomposition (SVD)

For any mn real matrix A, there exists an SVD: A = UV, where:

• U is an mm orthogonal matrix
• is an mn diagonal matrix with non-negative entries (singular values)
• V is an nn orthogonal matrix

Pseudoinverse

where is the transpose of with reciprocals of non-zero entries.

Applications

• Least squares approximation
• Data compression (e.g., image compression)
• Principal Component Analysis (PCA)
• Matrix approximation

Special Matrices

Key Formulas Summary

Vector Operations

• Dot Product: u v = uv
• Vector Norm: ||v|| = (v v)
• Cross Product (3D): u v = [uv - uv, uv - uv, uv - uv]
• Projection: proju = (uv / vv)v

Matrix Operations

• Matrix Transpose: (A) = a
• Matrix Inverse: AA = I
• Determinant (22): det([a, b; c, d]) = ad - bc
• Trace: tr(A) = a

Eigendecomposition

• Characteristic Equation: det(A - I) = 0
• Eigenvector Equation: (A - I)v = 0
• Diagonalization: A = PDP

SVD

• Singular Value Decomposition: A = UV
• Pseudoinverse: A = VU

Common Applications

Linear algebra is applied in numerous fields:

Computer Graphics

• 3D transformations and rotations
• Image processing
• Rendering and shading

Machine Learning

• Principal Component Analysis (PCA)
• Neural networks (weight matrices)
• Support Vector Machines
• Recommendation systems

Physics and Engineering

• Quantum mechanics (state vectors)
• Structural analysis
• Control systems
• Signal processing

Statistics

• Covariance matrices
• Regression analysis
• Factor analysis