Numerical Methods Qualification Exam Problems and Solutions (University of Maryland)/August 2008

Solution 1
Since Householder Matrices are orthogonal and hermitian (i.e. symmetric if the matrix is real), we have

$$y = Hx \Rightarrow x = H^Ty = Hy$$

Then the constraint can be written

$$ \begin{align} p^Tx            &= \delta \\ p^T(H^Ty)       &= \delta \\ (Hp)^Ty         &= \delta \\ (\|p\|_2e_1)^Ty &= \delta \\ \|p\|_2e_1^Ty   &= \delta \\ y_1             &= \frac{\delta}{\|p\|_2} \end{align} $$

Hence, the first entry of the column vector $$y \!\,$$, $$y_1 \!\,$$(a scalar), represents our constraint.

Now we want to show that we can write $$ \|Ax-b\|_2 \!\,$$ as a related unconstrained problem. Substituting for $$x\!\,$$ in $$Ax\!\,$$ we get,

$$Ax = AHy\!\,$$

Let $$ AH = B = \begin{bmatrix}B_1 & \hat{B}\end{bmatrix} $$ where $$ B_1 \!\,$$ is the first column of $$ B \!\,$$ and $$\hat{B}\!\,$$ are the remaining columns.

Since,


 * $$ y = \begin{bmatrix}y_1 \\ \hat{y}\end{bmatrix} $$

block matrix multiplication yields,

$$ \begin{align} AHy  &=By \\ &=\begin{bmatrix}B_1 & \hat{B}\end{bmatrix} \begin{bmatrix}y_1 \\ \hat{y}\end{bmatrix} \\ &= y_1B_1 + \hat{B}\hat{y} \end{align} $$

So  $$\min_y \|b - AHy\|_2 = \min_{\hat{y}}\|b - y_1B_1 - \hat{B}\hat{y}\|_2$$, which is unconstrained.

Solution 2a
Substituting into this equation the pairs $$x_i,\,y_i,\; i=1,2,3,\, $$ gives rise to the following matrix equation:



\begin{bmatrix} y_1 \\ y_2 \\ y_3 \end{bmatrix} = \underbrace{\begin{bmatrix} 1 & x_1 & x_1^2 \\ 1 & x_2 & x_2^2 \\ 1 & x_3 & x_3^2 \end{bmatrix}}_{A} \begin{bmatrix} c_1 \\ c_2 \\ c_3 \end{bmatrix} $$

Where A is the Vandermonde matrix, which is know to be non-singular. This proves existence and uniqueness of the coefficients $$c_1,\, c_2,\, c_3 $$.

Solution 2b
Now, substituting the pairs $$x_i,\,y_i,\; i=1,2,3,\, $$ gives:



\begin{bmatrix} y_1 \\ y_2 \\ y_3 \end{bmatrix} = \begin{bmatrix} 1 & x_1^2 & x_1^4 \\ 1 & x_2^2 & x_2^4 \\ 1 & x_3^2 & x_3^4 \end{bmatrix} \begin{bmatrix} c_1 \\ c_2 \\ c_3 \end{bmatrix} $$

Consider the unique set of points $$x_1=1,\ x_2=-1,\ x_3=0.\ $$

The system reduces to



\begin{bmatrix} y_1 \\ y_2 \\ y_3 \end{bmatrix} = \underbrace{\begin{bmatrix} 1 & 1 & 1 \\ 1 & 1 & 1 \\   1 & 0 & 0 \end{bmatrix}}_A \begin{bmatrix} c_1 \\ c_2 \\ c_3 \end{bmatrix} $$

Here, the matrix $$A\!\,$$ is clearly singular.

More generally, the determinant of the matrix $$ A$$ is given by

$$ \left| \begin{array}{ccc} 1 & x_1^2 & x_1^4 \\ 1 & x_2^2 & x_2^4 \\ 1 & x_3^2 & x_3^4 \end{array}\right|=(x_2^2-x_1^2)(x_3^2-x_1^2)(x_3^2-x_2^2)=0 $$ if $$x_j=-x_i$$ for any $$ i\neq j$$

Solution 3a


\begin{align} \langle Q^{-1}Av,w\rangle_Q &= (QQ^{-1}Av,w) \\ &= (Av,w) \\ \langle v,Q^{-1}Aw\rangle_Q &= (Qv,Q^{-1}Aw) \\ &= (Q^{-T}Qv,Aw) \\ &= (Q^{-1}Qv,Aw) \\ &= (v,Aw) \\ &= (A^Tv,w) \\ &= (Av,w) \\ \\ \langle Q^{-1}Av,v\rangle_Q &= (QQ^{-1}Av,v) \\ &= (Av,v) \\ &> 0 \\ \end{align} $$

Solution 3b
Choose $$\!\,x_0$$ $$\!\,r_0=b-Ax_0$$ $$\tilde{r}_0$$ : solve $$Q\tilde{r}_0 = r_0\!\,$$ $$\!\,p_0=\tilde{r}_0$$ for $$\!\,k=0,1,2,...$$ until convergence $$\!\,\alpha_k=(r_k,\tilde{r}_k)/(p_k,Ap_k)$$ $$\!\,x_{k+1}=x_k+\alpha_kp_k$$ $$\!\,r_{k+1}=r_k-\alpha_kAp_k$$  $$\tilde{r}_{k+1}$$ : solve $$Q\tilde{r}_{k+1} = r_{k+1}\!\,$$ $$\!\,\beta_k=(r_{k+1},\tilde{r}_{k+1})/(r_k,\tilde{r}_k)$$ $$\!\,p_{k+1}=\tilde{r}_{k+1}+\beta_kp_k$$ end

One additional cost is computing $$Q^{-1}\,\!$$.

Another one-time cost is multiplying $$Q^{-1}A\,\!$$ which has cost $$n^3\,\!$$ and $$Q^{-1}b\,\!$$ which has cost $$n^2\,\!$$.

Solution 3c
We want $$Q^{-1}A\!\,$$ to have eigenvalues whose values are close together. This accelerates convergence. Furthermore, if there are only $$k\!\,$$ distinct eigenvalues, the algorithm will terminate in $$k\!\,$$ steps.