Linear Algebra: A Fundamental Approach

A good way to understand linear algebra is to stop treating vectors as “lists of numbers” and start treating them as directions that can be combined. Once you do that, many later topics - least squares, dimensionality reduction, even quantum measurement - become different views of the same geometry. Subspaces are where “directions stay closed under mixing,” and projections are the mechanism for splitting space into “what lies in the subspace” and “what points away from it.”

In this chapter you build that geometric picture carefully, and you also keep it honest with rigorous proofs. You will see how orthogonality (perpendicularity) is not just a picture detail: it turns “closest point” into a clean statement and makes projections behave predictably.

Core objects: vectors, spans, and subspaces

A vector space (often shortened to “space”) is a set of vectors where you can add vectors and scale them without leaving the set. In plain terms: if the vectors represent directions or signals, the space contains all mixtures you can form using those directions. Formally, a set V is a vector space over a field F if it is closed under addition and scalar multiplication and it satisfies the usual axioms (associativity, commutativity, distributivity, and existence of a zero vector and additive inverses). Ask yourself a quick check: if you take two allowed directions and mix them, do you still stay inside the allowed directions?

Now focus on how spaces are built. Given vectors v1, v2,..., vk in a vector space, the span of these vectors is the set of all linear combinations: c1 v1 + c2 v2 +... + ck vk where each ci is a scalar. The span is the smallest subspace containing the vectors. That “smallest” part matters because it gives you a recipe: if you want a subspace generated by certain directions, span is the construction.

A subspace is a subset W of a vector space V that is itself a vector space under the same operations. So to prove something is a subspace, you do not need every axiom from scratch. Instead, you typically check closure properties and the presence of the zero vector (which is forced by closure). A common workflow is: verify that the zero vector is in W, that u and v in W implies u + v is in W, and that u in W and scalar c implies c u is in W.

One concrete way to anchor this is to think of points in R2 or R3. In R2, a subspace is either the zero vector alone, a line through the origin, or all of R2. In R3, a subspace is either {0}, a line through the origin, a plane through the origin, or all of R3. These are exactly the shapes you get when you take spans of one vector, two non-parallel vectors, or three vectors that generate the whole space.

[INSERT FIGURE: 2D geometric representation of a span and a subspace in R^2; show a line through the origin and the set of all scalar multiples]

Practical takeaway: whenever you see a statement like “the set of all mixtures of these vectors,” translate it into “span,” and whenever you see “subset that is closed under mixing,” translate it into “subspace.”

Orthogonality and orthogonal complements: separating “in” from “out”

To talk about projections, you need orthogonality. Two vectors u and w are orthogonal if their dot product is zero. In Rn with the standard dot product, u · w = 0 means they point in perpendicular directions. This gives you a way to define “perpendicular to a whole subspace,” not just to a single vector.

Given a subspace W inside Rn, the orthogonal complement of W, written W_perp, is the set of all vectors in Rn that are orthogonal to every vector in W. In symbols, x is in W_perp if and only if x · w = 0 for every w in W. The key intuition is: W_perp contains all directions that “do not interact” (via dot product) with anything in W.

Two immediate facts make W_perp useful. First, W_perp is itself a subspace. Second, it captures the “leftover directions” once you separate space into a part that lies in W and a part that lies perpendicular to W. That separation is exactly what projections formalize.

[INSERT FIGURE: 3D vector representations of W and W_perp in R^3; show a plane through the origin and the line perpendicular to it]

Ask yourself: if W is a plane through the origin in R3, what does W_perp look like? It must be all vectors perpendicular to every direction in that plane - so it becomes a line through the origin. If W is a line through the origin, W_perp becomes a plane through the origin. These shapes are the geometry you will reuse constantly.

Practical takeaway: W_perp is not an extra concept; it is the clean definition of “everything perpendicular to the entire subspace.”

Orthogonal projection onto a subspace: the closest-point mechanism

Now you can state the central idea: given a subspace W and a vector x, the orthogonal projection of x onto W is the unique vector p in W such that x - p is orthogonal to W. In words: p is the part of x that lies in W, and the leftover x - p points straight away from W in the perpendicular direction.

This characterization is more than a definition; it is a working tool. If you can describe W and compute the projection, you can solve many problems that later appear as least squares, filtering, and measurement postulates.

The existence and uniqueness of the projection are what you should be able to prove. The proof strategy is based on the idea that any vector can be decomposed into “in W” plus “in W_perp.” If that decomposition is correct, then the projected part is forced to be the W component.

A useful special case is projection onto a line spanned by a nonzero vector u. If W = span(u), then the projection of x onto W is the scalar multiple of u: p = ( (x · u) / (u · u) ) u The formula is simple because “being in W” means “being a multiple of u,” and “being perpendicular to W” means perpendicular to u.

[INSERT FIGURE: Geometric representation of an orthogonal projection in R^2; show x, its projection p on a line, and the right angle between x-p and the line]

Practical takeaway: orthogonal projection is the closest-point split, enforced by perpendicularity.

Step-by-Step Proofs: subspaces, orthogonal complements, and projection uniqueness

You now prove three core facts in a way that you can reuse later.

Step 1: Prove that W_perp is a subspace. Let W_perp be defined as all x such that x · w = 0 for every w in W. Take a and b in W_perp. For any w in W, (a + b) · w = a · w + b · w = 0 + 0 = 0, so a + b is in W_perp. Also for any scalar c, (c a) · w = c (a · w) = c 0 = 0, so c a is in W_perp. The zero vector also satisfies 0 · w = 0, so 0 is in W_perp. Therefore W_perp is a vector space, hence a subspace.

Practical takeaway: orthogonal complements are stable under the same operations as vector spaces.

Step 2: Prove the decomposition x = p + q with p in W and q in W_perp. Here is the key statement you want: for each x, there exists p in W and q in W_perp such that x = p + q. One clean way to justify this is to use the idea of solving a “best fit” problem geometrically. If p is chosen to minimize the distance ||x - p||, then the residual x - p must be perpendicular to W; otherwise you could move a small amount within W to reduce distance. That perpendicularity condition is exactly the definition of being in W_perp. The existence of a minimizer can be argued using coordinates or using the fact that spans of finitely many vectors reduce the problem to a finite-dimensional optimization with a quadratic objective.

Ask yourself: why does perpendicularity show up at the minimizer? Because any direction inside W gives a possible move, and if the residual is not perpendicular, moving a bit improves the dot-product alignment and shrinks the distance.

Practical takeaway: orthogonality is not decoration - it is the necessary condition for the closest point.

Step 3: Prove uniqueness of the orthogonal projection. Suppose p1 and p2 are both projections of x onto W. Then p1 and p2 are in W, and x - p1 is in W_perp while x - p2 is in W_perp. Subtract: (p1 - p2) = (x - p2) - (x - p1). The right-hand side is a difference of two vectors in W_perp, so p1 - p2 is in W_perp. But p1 - p2 is also in W because both p1 and p2 are in W. So p1 - p2 lies in W ∩ W_perp.

Now use the fact that the only vector orthogonal to itself (in a real inner product space) is the zero vector. If y is in W and W_perp, then y · y = 0, so y = 0. Therefore W ∩ W_perp = {0}, which forces p1 - p2 = 0 and hence p1 = p2. Projection is unique.

Practical takeaway: uniqueness comes from the intersection W ∩ W_perp being only the zero vector.

[INSERT FIGURE: Diagram showing W, the perpendicular residual, and the uniqueness argument using the intersection W ∩ W_perp]

Real-World Cybersecurity/Tech Applications: projections as “remove the part that doesn’t belong”

Orthogonal projection looks abstract until you connect it to what you actually compute: separating a signal into components that match a chosen structure versus components that do not. In cybersecurity, you often build a subspace meant to represent “typical behavior” or “allowed patterns,” then treat the perpendicular component as the mismatch.

For example, suppose you model a set of normal network traffic features as lying close to some low-dimensional subspace W (spanned by a set of basis directions learned from data). When a new traffic vector x arrives, projecting onto W gives p, the part consistent with the learned structure. The residual r = x - p captures what does not fit. Large residual magnitudes often flag anomalies: the vector is far from the subspace in the orthogonal direction.

Even in simpler linear systems, this idea appears when you enforce constraints. If you have a “direction set” and you want to strip out the component that violates a constraint, projecting onto the subspace that satisfies the constraint is the cleanest way to do it. The orthogonal complement tells you exactly what kind of violation you are left with.

Practical takeaway: if you can define what “allowed” means as a subspace, projection gives the computational split between allowed behavior and leftover mismatch.

20 Practice Problems (with solutions)

1) Problem: Given vectors v1 and v2 in R3, define W as the span of v1 and v2. What are the only possible dimensions of W? Solution: dim(W) is 0 if both vectors are zero, 1 if v1 and v2 are dependent (one is a scalar multiple of the other), and 2 if they are independent. It cannot exceed 2 because W is generated by two vectors.

2) Problem: Prove that if W is a subspace of V, then 0 is in W. Solution: Take any w in W. Since W is a subspace, it is closed under scalar multiplication, so 0*w is in W. But 0*w is the zero vector, so 0 is in W.

3) Problem: Let W be a subspace of Rn. Show that W_perp is a subspace. Solution: If a and b are in W_perp, then for every w in W, (a+b)·w = a·w + b·w = 0. Similarly (c a)·w = c(a·w)=0. Also 0 is in W_perp. So W_perp is closed under addition and scalar multiplication and contains zero.

4) Problem: If W = {0} in Rn, what is W_perp? Solution: Every x is orthogonal to the only vector 0, since x·0 = 0. So W_perp = Rn.

5) Problem: If W = Rn, what is W_perp? Solution: x is orthogonal to every vector in Rn. In particular, x must be orthogonal to itself, so x·x=0, forcing x=0. Hence W_perp = {0}.

6) Problem: Let u be a nonzero vector in Rn and W = span(u). Derive the projection formula for proj_W(x). Solution: proj_W(x) must be a multiple of u: p = c u. The condition x - p is orthogonal to u gives (x - c u)·u = 0, so x·u - c(u·u)=0. Thus c = (x·u)/(u·u), so p = ((x·u)/(u·u)) u.

7) Problem: Show that if p is the projection of x onto W = span(u), then x - p is orthogonal to u. Solution: Using p = ((x·u)/(u·u))u, compute (x - p)·u = x·u - p·u = x·u - ((x·u)/(u·u)) (u·u) = 0.

8) Problem: Prove uniqueness of orthogonal projection onto a subspace W. Solution: Assume p1 and p2 are both projections. Then p1-p2 is in W and in W_perp. So p1-p2 is orthogonal to itself, giving p1-p2 = 0, hence p1=p2.

9) Problem: Let W be a subspace. Prove that W and W_perp intersect only at {0}. Solution: If y is in both W and W_perp, then y is orthogonal to every vector in W, including itself. So y·y=0, which implies y=0.

10) Problem: Suppose x is already in W. What is its orthogonal projection onto W? Solution: If x is in W, then choose p = x. Then x - p = 0, which is orthogonal to W. So proj_W(x)=x.

11) Problem: Suppose x is in W_perp. What is its orthogonal projection onto W? Solution: If x is in W_perp, then x is orthogonal to every vector in W. The projection p must be in W and make x-p orthogonal to W. Taking p=0 works because x-0=x is orthogonal to W. Uniqueness forces proj_W(x)=0.

12) Problem: Let W be span(u) with u nonzero. Under what condition is proj_W(x)=0? Solution: proj_W(x)=0 means (x·u)/(u·u)=0, which is equivalent to x·u=0. So x must be orthogonal to u.

13) Problem: In R2, let W be the x-axis (all vectors of the form (t,0)). Find W_perp. Solution: W_perp consists of all vectors orthogonal to every (t,0). Dot product with (t,0) gives t times the first

component, so the condition for orthogonality to all (t,0) is that the first coordinate of the vector is 0. Therefore W_perp is the y-axis, consisting of all vectors of the form (0,s).

Introduction: why projections show up everywhere A projection answers one practical question: if you have a target direction or a target flat inside a space, what part of your vector actually “lands” on that target? The clean answer uses orthogonality - perpendicularity - because perpendicular error is what you can’t get rid of by choosing a best approximation inside the subspace.

This chapter builds that idea in a way that later matrix theory can reuse directly. We start with vectors, then we treat sets of vectors like “places” where solutions live. Those places are subspaces. From there we define orthogonal projections and prove the basic facts that make them reliable: existence, uniqueness, and geometric meaning.

Ask yourself a quick checkpoint: if you change the vector slightly, does the projected part change in a predictable way? The linear structure behind subspaces and projections is exactly what makes that predictability possible.

Core Theory: vector spaces and subspaces as sets closed under operations A vector space is a set V where you can add vectors and multiply vectors by scalars from a field (like real numbers), and the usual rules work. For this chapter, you mainly work with V = Rn, where vectors are n-tuples of real numbers and operations are done componentwise.

A subspace W of V is a subset that behaves like a vector space under the same operations. Concretely, W must contain the zero vector, be closed under addition, and be closed under scalar multiplication. These are not “extra conditions”; they are exactly the closure properties you need so that linear combinations of vectors stay inside W.

Geometrically, a subspace is a “flat” through the origin. In R2 it looks like a line through (0,0) or all of R2 or just {0}. In R3 it looks like a plane through the origin, a line through the origin, or all of R3. This “through the origin” part matters because linear combinations of vectors always keep the origin fixed.

A useful companion idea is span. If you start with vectors v1 and v2, then span(v1,v2) is the set of all linear combinations a v1 + b v2. By construction, span(v1,v2) is a subspace. It is the smallest subspace containing those starting vectors.

Takeaway: whenever you see a set described as “all linear combinations,” you should immediately think “subspace,” and whenever you see a subspace, you should think “all linear combinations of some spanning vectors.”

Step-by-Step Proofs: orthogonality and the perpendicular subspace W_perp Orthogonality in Rn is defined using the dot product. Two vectors x and y are orthogonal if x·y = 0. This gives a way to test whether a vector points in a direction perpendicular to another.

Now define W_perp for a subspace W: W_perp is the set of all vectors in Rn that are orthogonal to every vector in W. Formally, x is in W_perp when x·w = 0 for all w in W. This definition looks global - “for every w” - but it becomes simple once you pick a spanning set for W.

Because W is a subspace, you can prove W_perp is also a subspace. The key closure checks use only dot product properties: - If a and b are orthogonal to every w in W, then (a+b)·w = a·w + b·w = 0, so a+b is still orthogonal to W. - If a is orthogonal to every w in W, then (c a)·w = c(a·w)=0, so scalar multiples stay inside W_perp. - The zero vector is orthogonal to everything because 0·w = 0.

Geometrically, W_perp is the set of all directions that have no component “along” W. If W is a line through the origin, W_perp is the line through the origin perpendicular to it. If W is a plane through the origin in R3, W_perp is the line through the origin perpendicular to the plane.

A powerful fact connects W and W_perp: their intersection is only {0}. If y is in both W and W_perp, then y is orthogonal to every vector in W, including itself. That gives y·y = 0, and in real vector spaces that forces y = 0.

Takeaway: W_perp is not a random set; it is the subspace that contains exactly the vectors with zero dot product against the whole target subspace.

[INSERT FIGURE: 3D vector representation of W and its perpendicular directions W_perp]

Core Theory: orthogonal projection as the “best” landing point The orthogonal projection of a vector x onto a subspace W is the vector proj_W(x) in W that matches x in the sense of perpendicular error. The requirement is that the difference x - proj_W(x) is orthogonal to W. That is, the error points purely into W_perp.

This definition is geometric: you split x into two parts, one inside W and one perpendicular to W. In symbols: - proj_W(x) lies in W. - x - proj_W(x) lies in W_perp. So x = (part in W) + (part perpendicular to W).

Why is this the right idea for “best approximation”? Because any other candidate p in W would have a difference x - p whose component along W cannot be eliminated. The perpendicular condition is the one that makes the error as “small” as possible in the direction that matters.

In the special case W = span(u) with u nonzero, the geometry becomes one-dimensional: proj_W(x) must be a scalar multiple of u. Write proj_W(x) = c u. The perpendicular condition says (x - c u)·u = 0. Solving gives c = (x·u)/(u·u), so proj_W(x) = ((x·u)/(u·u)) u.

This formula is not

End of chapter one. 4 more chapters in the full book.