Numerical Methods Qualification Exam Problems and Solutions (University of Maryland)/August 2003

Descent Direction
$$\nabla Q(x_k) \cdot d_k < 0 \!\,$$

Optimal step size
Choose $$\alpha_k \!\,$$ such that $$Q(x_k+\alpha_k d_k) \!\,$$ is minimized i.e.

$$\alpha_k : \min_{\alpha_k} Q(x_k+\alpha_k d_k) \!\,$$

$$ Q(x_k+\alpha_kd_k) = \langle x_k + \alpha_k d_k, Ax_k + \alpha_k A d_k \rangle - \langle x_k + \alpha_k d_k, b \rangle \!\,$$

$$\frac{d}{d \alpha} Q(x_k+\alpha_k d_k)= \langle A d_k, x_k \rangle + \alpha_k \langle A d_k, d_k \rangle - \langle d_k, b \rangle \!\,$$

Setting the above expression equal to zero gives the optimal $$\alpha_k \!\,$$:

$$\alpha_k = \frac{\langle d_k, b-Ax_k \rangle}{\langle A d_k, d_k \rangle} \!\,$$

Note that since $$A \!\,$$ is symmetric

$$\langle d_k, A x_k \rangle = \langle A d_k, x_k \rangle \!\,$$

Solution 3b
Note that $$d_k = -\nabla Q(x_k) = b - Ax_k = r_k \!\,$$. Then the minimal $$\alpha_k \!\,$$ is given by $$\frac{\langle r_k, r_k \rangle }{\langle r_k, r_k \rangle_A} \!\,$$

Given $$x_0 \in R^n \!\,$$ For $$ k=0,1,2,\ldots \!\,$$ $$ \begin{align} r_k &= b- Ax_k \\ &\mbox{If } r_k =0, \mbox { then quit } \\ d_k &= r_k \\ \alpha_k &= \frac{\langle r_k, r_k \rangle }{\langle r_k, r_k \rangle_A} \\ x_{k+1}&=x_k + \alpha_k d_k \end{align} $$

Solution 3
Since $$ B \!\,$$ is symmetric, positive definite, $$B=E^TE \!\,$$ where $$E \!\,$$ is upper triangular (Cholesky Factorization).

Then $$B^{-1}=E^{-1}E^{-T} \!\,$$

Hence,

$$ \begin{align} B^{-1} Ax & = B^{-1}b \\ E^{-1}E^{-T} Ax &= E^{-1}E^{-T} b \\ E^{-T} Ax &=E^{-T}b \\ \underbrace{E^{-T}A E^{-1}}_{\tilde{A}} \underbrace{ Ex}_{\tilde{x}} &= \underbrace{E^{-T}b}_{\tilde{b}} \end{align} \!\,$$

$$\tilde{A} \!\,$$ is symmetric:

$$(E^{-T}AE^{-1})^T=E^{-T}A E^{-1} \!\,$$ since $$A \!\,$$ symmetric

$$\tilde{A} \!\,$$ is positive definite:

$$x^TE^{-T}AE^{-1}x= (E^{-1}x)^T A(E^{-1}x) > 0 \!\,$$ since $$A \!\,$$ positive definite

Pseudocode
Given $$x_0 \in R^n \!\,$$ For $$ k=0,1,2,\ldots \!\,$$ $$ \begin{align} \tilde{x_k} &= E x_k \\ r_k &= \tilde{b}- \tilde{A} \tilde{x_k} \\ &\mbox{If } r_k =0, \mbox { then quit } \\ d_k &= r_k \\ \alpha_k &= \frac{\langle r_k, r_k \rangle }{\langle r_k, r_k \rangle_{\tilde{A}}} \\ x_{k+1}&=x_k + \alpha_k d_k \end{align} $$