Projection onto a Subspace

The vector v_{‖ S} , which actually lies in S, is called the projection of v onto S, also denoted proj _S ^v. If v₁, v₂, …, v _r form an orthogonal basis for S, then the projection of v onto S is the sum of the projections of v onto the individual basis vectors, a fact that depends critically on the basis vectors being orthogonal:

Example 1: Let S be the 2-dimensional subspace of R³ spanned by the orthogonal vectors v₁ = (1, 2, 1) and v₂ = (1, −1, 1). Write the vector v = (−2, 2, 2) as the sum of a vector in S and a vector orthogonal to S.

From (*), the projection of v onto S is the vector

Therefore, v = v_{‖ S} where v_{‖ S} = (0, 2, 0) and

That v_{⊥ S} = (−2, 0, 2) truly is orthogonal to S is proved by noting that it is orthogonal to both v₁ and v₂:

In summary, then, the unique representation of the vector v as the sum of a vector in S and a vector orthogonal to S reads as follows:

Example 2: Let S be a subspace of a Euclidean vector space V. The collection of all vectors in V that are orthogonal to every vector in S is called the orthogonal complement of S:

( S^⊥ is read “S perp.”) Show that S^⊥ is also a subspace of V.

Proof. First, note that S^⊥ is nonempty, since 0 ∈ S^⊥. In order to prove that S^⊥ is a subspace, closure under vector addition and scalar multiplication must be established. Let v₁ and v₂ be vectors in S^⊥; since v₁ · s = v₂ · s = 0 for every vector s in S,

Example 3: Find the orthogonal complement of the x−y plane in R³.

Example 4: Let P be the subspace of R³ specified by the equation 2 x + y = 2 z = 0. Find the distance between P and the point q = (3, 2, 1).

One way to find the orthogonal component q_{⊥ P} is to find an orthogonal basis for P, use these vectors to project the vector q onto P, and then form the difference q − proj _P q to obtain q_{⊥ P} . A simpler method here is to project q onto a vector that is known to be orthogonal to P. Since the coefficients of x, y, and z in the equation of the plane provide the components of a normal vector to P, n = (2, 1, −2) is orthogonal to P. Now, since

The Gram-Schmidt orthogonalization algorithm. The advantage of an orthonormal basis is clear. The components of a vector relative to an orthonormal basis are very easy to determine: A simple dot product calculation is all that is required. The question is, how do you obtain such a basis? In particular, if B is a basis for a vector space V, how can you transform B into an orthonormal basis for V? The process of projecting a vector v onto a subspace S—then forming the difference v − proj _S v to obtain a vector, v_{⊥ S} , orthogonal to S—is the key to the algorithm.

Example 5: Transform the basis B = { v₁ = (4, 2), v₂ = (1, 2)} for R² into an orthonormal one.

The first step is to keep v₁; it will be normalized later. The second step is to project v₂ onto the subspace spanned by v₁ and then form the difference v₂ − proj_v1 v₂ = v_⊥1 Since

Figure 6

The vectors v₁ and v_⊥1 are now normalized:

Thus, the basis B = { v₁ = (4, 2), v₂ = (1, 2)} is transformed into the orthonormal basis

Figure 7

The preceding example illustrates the Gram-Schmidt orthogonalization algorithm for a basis B consisting of two vectors. It is important to understand that this process not only produces an orthogonal basis B′ for the space, but also preserves the subspaces. That is, the subspace spanned by the first vector in B′ is the same as the subspace spanned by the first vector in B′ and the space spanned by the two vectors in B′ is the same as the subspace spanned by the two vectors in B.

In general, the Gram-Schmidt orthogonalization algorithm, which transforms a basis, B = { v₁, v₂,…, v _r }, for a vector space V into an orthogonal basis, B′ { w₁, w₂,…, w _r }, for V—while preserving the subspaces along the way—proceeds as follows:

Step 1. Set w₁ equal to v₁

Step 2. Project v₂ onto S₁, the space spanned by w₁; then, form the difference v₂ − proj _{S 1} v₂ This is w₂.

Step 3. Project v₃ onto S₂, the space spanned by w₁ and w₂; then, form the difference v₃ − proj _{S 2} v₃. This is w₃.

Step i. Project v _i onto S_i −1, the space spanned by w₁, …, w _i−1 ; then, form the difference v _i − proj _{S i−1} v _i . This is w _i .

This process continues until Step r, when w _r is formed, and the orthogonal basis is complete. If an orthonormal basis is desired, normalize each of the vectors w _i .

Example 6: Let H be the 3-dimensional subspace of R⁴ with basis

Find an orthogonal basis for H and then—by normalizing these vectors—an orthonormal basis for H. What are the components of the vector x = (1, 1, −1, 1) relative to this orthonormal basis? What happens if you attempt to find the componets of the vector y = (1, 1, 1, 1) relative to the orthonormal basis?

The first step is to set w₁ equal to v₁. The second step is to project v₂ onto the subspace spanned by w₁ and then form the difference v₂− proj_W1 v₂ = W₂. Since

Now, for the last step: Project v₃ onto the subspace S₂ spanned by w₁ and w₂ (which is the same as the subspace spanned by v₁ and v₂) and form the difference v₃− proj _{S 2} v₃ to give the vector, w₃, orthogonal to this subspace. Since

This gives

Therefore, the Gram-Schmidt process produces from B the following orthogonal basis for H:

You may verify that these vectors are indeed orthogonal by checking that w₁ · w₂ = w₁ · w₃ = w₂ · w₃ = 0 and that the subspaces are preserved along the way:

An orthonormal basis for H is obtained by normalizing the vectors w₁, w₂, and w₃:

Relative to the orthonormal basis B′′ = { ŵ₁, ŵ₂, ŵ₃}, the vector x = (1, 1, −1, 1) has components

These calculations imply that

If the components of y = (1, 1, 1, 1) relative to this basis are desired, you might proceed exactly as above, finding

These calculations seem to imply that

The problem, however, is that this equation is not true, as the following calculation shows:

What went wrong? The problem is that the vector y is not in H, so no linear combination of the vectors in any basis for H can give y. The linear combination

Example 7: If the rows of a matrix form an orthonormal basis for R ⁿ , then the matrix is said to be orthogonal. (The term orthonormal would have been better, but the terminology is now too well established.) If A is an orthogonal matrix, show that A⁻¹ = A^T.

Let B = { vˆ₁, vˆ₂, …, vˆ _n } be an orthonormal basis for R ⁿ and consider the matrix A whose rows are these basis vectors:

The matrix A^T has these basis vectors as its columns:

Since the vectors vˆ₁, vˆ₂, …, vˆ _n are orthonormal,

Now, because the ( i, j) entry of the product AA^T is the dot product of row i in A and column j in A^T,

Thus, A⁻¹ = A^T. [In fact, the statement A⁻¹ = A^T is sometimes taken as the definition of an orthogonal matrix (from which it is then shown that the rows of A form an orthonormal basis for R ⁿ ).]

An additional fact now follows easily. Assume that A is orthogonal, so A⁻¹ = A^T. Taking the inverse of both sides of this equation gives