Numerical Methods Qualification Exam Problems and Solutions (University of Maryland)/August 2006

Solution 1a
Integrating $$\int K(x)f''(x)dx\!\,$$ by parts over arbitrary points $$p \!\,$$ and $$q \!\,$$ gives


 * $$ \int_p^q K(x)f''(x)dx = \left[(x-x_j)f(x) - \frac{1}{2}(x-x_j)^2f'(x)\right]_{x=p}^{x=q} - \int_p^q f(x)dx \!\,$$

Since $$ K(x) \!\,$$ is defined on $$ [x_{j-\frac{1}{2}}, x_{j+\frac{1}{2}}] \!\,$$ we use


 * $$[p,q]=[x_j, x_{j+\frac{1}{2}}] \!\,$$

and


 * $$[p,q]=[x_{j-\frac{1}{2}}, x_j] \!\,$$

Using the first interval we get


 * $$ \frac{1}{2}\left[-\frac{1}{4}(x_{j+1}-x_j)^2f'(x_{j+\frac{1}{2}}) + (x_{j+1}-x_j)f(x_{j+\frac{1}{2}})\right] \qquad \mathbf{(1)}\!\,$$

And for the second we get


 * $$ \frac{1}{2}\left[\frac{1}{4}(x_{j-1}-x_j)^2f'(x_{j-\frac{1}{2}}) + (x_j-x_{j-1})f(x_{j-\frac{1}{2}})\right] \qquad \mathbf{(2)}\!\,$$

Since these apply to arbitrary half-subintervals, we can rewrite equation $$ \mathbf{(2)} \!\,$$ with the its indecies shifted by one unit. The equation for the interval $$[x_{j+\frac{1}{2}}, x_{j+1}] \!\,$$ is


 * $$ \frac{1}{2}\left[\frac{1}{4}(x_{j}-x_{j+1})^2f'(x_{j+\frac{1}{2}}) + (x_{j+1}-x_{j})f(x_{j+\frac{1}{2}})\right] \qquad \mathbf{(2')}\!\,$$

Combining $$ \mathbf{(1)} \!\,$$ and $$ \mathbf{(2')} \!\,$$ and writing it in the same form as the integration by parts, we have


 * $$ \int_{x_j}^{x_{j+1}} K(x)f''(x)dx = (x_{j+1}-x_j)f(x_{j+\frac{1}{2}}) - \int_{x_j}^{x_{j+1}} f(x)dx \!\,$$

Then our last step is to sum this over all of our $$n\!\,$$ subintervals, noting that $$x_{j+1}-x_j=(b-a)/n\!\,$$



\begin{align} \sum_{j=0}^{n-1}\int_{x_j}^{x_{j+1}} K(x)f''(x)dx &= \sum_{j=0}^{n-1}(x_{j+1}-x_j)f(x_{j+\frac{1}{2}}) - \sum_{j=0}^{n-1}\int_{x_j}^{x_{j+1}} f(x)dx \\ \int_{a}^{b} K(x)f''(x)dx &= \frac{b-a}{n} \sum_{j=0}^{n-1}f((x_{j}+x_{x+1})/2) - \int_{a}^{b} f(x)dx \\ \int_{a}^{b} K(x)f''(x)dx &= Q_n(f) - I(f) \end{align} $$

Solution 1b
Applying the result of part (a), the triangle inequality, and pulling out the constant $$\|f(x)\|_\infty \!\,$$, we have,

$$ \begin{align} &\leq \int_a^b |K(x)| |f^{\prime\prime}(x)| dx \\ &\leq \|f^{\prime\prime}(x)\|_\infty \underbrace{\int_a^b |K(x)| dx}_{M_n} \end{align} $$
 * E_n(f)| &= | \int_a^b K(x) f^{\prime\prime}(x) dx | \\

$$M_n \!\,$$ is some constant less than infinity since $$[a,b] \!\,$$ is compact and $$|K(x)| \!\,$$ is continuous on each of the finite number of intervals for which it is defined.

The above inequality becomes an equality when


 * $$f''(x)=c,$$

where $$c$$ is any constant.

Solution 2a
The QR algorithm produces a sequence of similar matrices $$ \{A_i\} \!\,$$ whose limit tends towards being upper triangular or nearly upper triangular. This is advantageous since the eigenvalues of an upper triangular matrix lie on its diagonal.

i = 0 A_1 = A while ( error > tolerance  ) A_i=Q_i R_i (QR decomposition/factorization) A_{i+1}=R_i Q_i (multiply R and Q, the reverse multiplication) i=i+1 (increment)

Solution 2b
From the factor step (QR decomposition) of the algorithm, we have


 * $$ A_i=Q_iR_i \!\,$$

which implies


 * $$ Q_i^{-1}A_i = R_i \!\,$$

Substituting into the reverse multiply step, we have

$$ \begin{align} A_{i+1} &=R_i Q_i   \\ &= Q_i^{-1} A_i Q_i \end{align} $$

Solution 2c
A series of Given's Rotations matrices pre-multiplying $$ A \!\,$$, a upper Heisenberg matrix, yield an upper triangular matrix$$R \!\,$$ i.e.

$$ G_{(n-1,n)}\cdots G_{(2,3)}G_{(1,2)}A = R \!\,$$

Since Givens Rotations matrices are each orthogonal, we can write


 * $$ A = \underbrace{(G_{(n-1,n)}\cdots G_{(2,3)}G_{(1,2)})^T}_QR \!\,$$

i.e.


 * $$ A = QR \!\,$$

If we let $$ A_0 := A \!\,$$, we have $$ A_0 = Q_0R_0 \!\,$$, or more generally for $$k=1,2,3,\ldots \!\,$$


 * $$ A_k = Q_kR_k \!\,$$.

In each case, the sequence of Given's Rotations matrices that compose $$ Q \!\,$$ have the following structure

$$ \begin{align} Q&=G_{(1,2)}^TG_{2,3}^T\cdots G_{(n-1,n)}^T \\ &= \begin{pmatrix} *  &   *   &   0   &   0   & \cdots &   0   \\ \!\,*  &   *   &   0   &   0   & \cdots &   0   \\ 0  &   0   &   1   &   0   & \cdots &   0   \\ 0  &   0   &   0   &   1   & \cdots &   0   \\ \vdots &\vdots &\vdots &\ddots & \ddots &\vdots \\ 0  &   0   &\cdots &\cdots &    0   &   1   \\ \end{pmatrix}

\begin{pmatrix} 1  &   0   &   0   &   0   & \cdots &   0   \\ 0  &   *   &   *   &   0   & \cdots &   0   \\ 0  &   *   &   *   &   0   & \cdots &   0   \\ 0  &   0   &   0   &   1   & \cdots &   0   \\ \vdots &\vdots &\vdots &\ddots & \ddots &\vdots \\ 0  &   0   &\cdots &\cdots &    0   &   1   \\

\end{pmatrix}

\cdots

\begin{pmatrix} 1  &   0   &   0   &   0   & \cdots &   0   \\ 0  &   1   &   0   &   0   & \cdots &   0   \\ \vdots &\ddots &\ddots &\ddots & \ddots &  0   \\ 0  &   0   &   0   &   1   &   0    &   0   \\    0   &   0   &   0   &   0   &   *    &   *   \\    0   &   0   &   0   &   0   &   *    &   *   \\ \end{pmatrix}  \\ &= \begin{pmatrix} *  &   *   &   *   &\cdots &   *   \\ \!\,*  &   *   &   *   &\cdots &   *   \\ 0  &   *   &   *   &\cdots &   *   \\ \vdots &\ddots &\ddots &\ddots &\vdots \\ 0  &   0   &   0   &   *   &   *   \\ \end{pmatrix} \end{align} $$

So $$ Q \!\,$$ is upper Hessenberg.

From the algorithm, we have

$$ \begin{align} A_{k+1}&=R_kQ_k\\ &= \begin{pmatrix} *  &   *   &   *   &\cdots &   *   \\ 0  &   *   &   *   &\cdots &   *   \\ 0  &   0   &   *   &\cdots &   *   \\ \vdots &\vdots &\ddots &\ddots &\vdots \\ 0  &   0   &   0   &   0   &   *   \\ \end{pmatrix} \begin{pmatrix} *  &   *   &   *   &\cdots &   *   \\ \!\,*  &   *   &   *   &\cdots &   *   \\ 0  &   *   &   *   &\cdots &   *   \\ \vdots &\ddots &\ddots &\ddots &\vdots \\ 0  &   0   &   0   &   *   &   *   \\ \end{pmatrix}

\end{align} $$

We conclude $$A_{k+1} \!\,$$ is upper Hessenberg because for $$j=1,2,\ldots,n-2 \!\,$$ the $$j \!\,$$th column of $$A_{k+1} \!\,$$ is a linear combination of the first $$j+1 \!\,$$ columns of $$R_k \!\,$$ since $$Q_k \!\,$$ is also upper Hessenberg.

Solution 2d
The eigenvalues of $$A \!\,$$ can be computed. They are $$6 \!\,$$ and $$2 \!\,$$. Furthermore, the result of matrix multiplies in the algorithm shows that the diagonal difference, $$(a_{1,2}-a_{2,1})\!\,$$, is constant for all $$i \!\,$$.

Since the limit of $$A \!\,$$ is an upper triangular matrix with the eigenvalues of $$A \!\,$$ on the diagonal, the limit is


 * $$\begin{pmatrix}6 & 2 \\ 0 & 2\end{pmatrix} \!\,$$

Solution 3a
We know that $$\|x^{(n)}-x\|_A\leq\|y-x\|_A\!\,$$ for every $$y\in x^{(0)}+\mathcal{K}_n(r^{(0)},A)\!\,$$, so if we can choose a $$y\in x^{(0)}+\mathcal{K}_n(r^{(0)},A)\!\,$$ such that


 * $$\|x^{(n)}-x\|_A\leq\|y-x\|_A\leq\|p_n(A)\|_A\|x^{(0)}-x\|_A\!\,$$,

then we can solve part a.

Rewrite r^(0)
First note from definition of $$r^{(0)}\!\,$$

$$ \begin{align} r^{(0)} &= Ax^{(0)}-b \\ &= Ax^{(0)}-Ax \\ &= A(x^{(0)}-x) \end{align} $$

Rewrite Krylov space
Therefore, we can rewrite $$\mathcal{K}_n(r^{(0)},A) \!\,$$ as follows:

$$ \begin{align} \mathcal{K}_n(r^{(0)},A) &= \mathcal{K}_n(A(x^{(0)}-x),A) \\ &= \mbox{span} \{A(x^{(0)}-x),A^2(x^{(0)}-x), \ldots, A^n(x^{(0)}-x) \} \end{align} $$

Write y explicitly
We can then write $$y \!\,$$ explicitly as follows:

$$ \begin{align} y &\in x_0 + \mathcal{K}_n(r^{(0)},A) \\ &\in x_0 + \mathcal{K}_n(A(x^{(0)}-x),A)\\ &= x_0 + \alpha_1 A(x^{(0)}-x) + \alpha_2 A^2(x^{(0)}-x)+\ldots+\alpha_n A^n(x^{(0)}-x)  \\ & \mbox{for arbitrary real numbers } \alpha_i \quad i=1,2,\ldots,n \end{align} $$

Substitute and Apply Inequality
We substitute $$y \!\,$$ into the hypothesis inequality and apply a norm inequality of matrix norms to get the desired result.

$$ \begin{align} \|x^{(n)}-x\|_A &\leq \|y-x\|_A \\ &= \|(x_0-x) +\alpha_1 A(x^{(0)}-x)+\alpha_2A^2(x^{(0)}-x)+\ldots+\alpha_nA^n(x^{(0)}-x) \|_A \\ &= \| \alpha_nA^n(x^{(0)}-x)+ \alpha_{n-1}A^{n-1}(x^{(0)}-x) +\ldots+\alpha_2A^2(x^{(0)}-x)+\alpha_1A(x^{(0)}-x)+(x_0-x) \|_A \\ &= \|P(A)(x^{(0)}-x)\|_A \\ &\leq \|P(A)\|_A \|x^{(0)}-x\|_A \end{align} $$

Overview
We want to show that

$$\|p_n(A)\|_A=\frac{1}{T_n(\frac{\lambda_\max+\lambda_\min}{\lambda_\max-\lambda_\min})} \!\,$$

Maximize Numerator of p_n(z)
By hypothesis,


 * $$\|p_n(A)\|_A=\max\{|p_n(z)| : z\in Sp\{A\}\}\!\,$$

Since only the numerator of $$p_n(z) \!\,$$ depends on $$ z \!\,$$, we only need to maximize the numerator in order to maximize $$|p_n(z)| \!\,$$. That is find


 * $$ z : \max_{z \in [\lambda_\min,\lambda_\max]}\left|T_n\left(\frac{\lambda_\max+\lambda_\min-2z}{\lambda_\max-\lambda_\min}\right)\right| \!\,$$

Rewrite T_n
Let $$\cos(\theta)=x \!\,$$. Then


 * $$\theta = \arccos(x) \!\,$$

Hence,

$$ \begin{align} T_n(\cos\theta) &= T_n(x) \\ &= \cos( n \theta) \\ &= \cos (n \arccos (x)) \end{align} $$

so


 * $$ T_n(x) = \cos(n \arccos(x)) \!\,$$

Max of T_n is 1
Denote the argument of $$T_n \!\,$$ as $$ x(z) \!\,$$ since the argument depends on $$ z \!\,$$. Hence,


 * $$ x(z) = \frac{\lambda_{\max}+\lambda_{\min}-2z}{\lambda_{\max}-\lambda_{\min}} \!\,$$,

Then,


 * $$ x(\lambda_\min) = \frac{\lambda_{\max}-\lambda_{\min}}{\lambda_{\max}-\lambda_{\min}} = 1 \!\, $$


 * $$ x(\lambda_\max) = \frac{\lambda_{\min}-\lambda_{\max}}{\lambda_{\max}-\lambda_{\min}} = -1 \!\, $$

Thus $$ x(z) \in [-1,1] \!\,$$.

Now, since $$ \; \arccos(x)\; $$ is real for $$ x \in [-1,1] \!\,$$,


 * $$ -1 \leq \underbrace{\cos(n\overbrace{\arccos(x)}^{\in \mathbb{R}})}_{T_n(x)} \leq 1 \!\,$$

Hence,


 * $$ \max_{x \in [-1,1]} T_n(x) = \max_{x \in [-1,1]} |\cos(n\arccos(x))| = 1 \!\,$$

Show T_n(1)=1
Let $$z=\lambda_\min \!\,$$, then

$$\left|T_n\left(\frac{\lambda_\max+\lambda_\min-2(\lambda_\min)}{\lambda_\max-\lambda_\min}\right)\right| = |T_n(1)| \!\,$$

Using our formula we have,

$$ \begin{align} &= |\cos(n \cdot 2\pi k ) | \quad k \in \mathbf{Z} \\ &= 1 \end{align} $$
 * T_n(1)| &= |\cos (n \cdot \arccos(1)) | \\

In other words, if $$z=\lambda_\min \!\,$$, $$ T_n \!\,$$ achieves its maximum value of $$1 \!\,$$.