Numerical Methods Qualification Exam Problems and Solutions (University of Maryland)/August 2007

Forward Direction
We want to prove the following statement:

If $$\{g_1,\ldots,g_n\}\!\,$$ is a Chebyshev system, then for any $$n \!\,$$ distinct points $$ x_1,\ldots,x_n \in [a,b] \!\,$$ the matrix $$A \!\,$$ with $$a_{i,j}=g_j(x_i), 1 \leq i, j\leq n \!\,$$ is non-singular.

Writing out the matrix $$A \!\,$$ yields:

$$ A= \begin{pmatrix} g_1(x_1) & g_2(x_1) & \cdots & g_n(x_1) \\ g_1(x_2) & g_2(x_2) & \cdots & g_n(x_2) \\ \vdots & \vdots  & \ddots & \vdots \\ g_1(x_n) & g_2(x_n) & \cdots & g_n(x_n) \end{pmatrix} $$

Since the set $$\{g_1,\ldots,g_n\}\!\,$$ is linearly independent, there does not exist any non-zero sets of constants of $$ \alpha_i \!\,$$ such that $$\sum_{i=1}^n \alpha_i g_i(x)=0 \!\,$$ for any $$x \in [a,b]\!\,$$. Hence, the columns of the matrix $$ A \!\,$$ are linearly independent which implies that $$ A \!\,$$ is non-singular.

Proof of (i)
Assume $$A\!\,$$ is non-singular.

This implies the columns of



A= \begin{pmatrix} g_1(x_1) & g_2(x_1) & \cdots & g_n(x_1) \\ g_1(x_2) & g_2(x_2) & \cdots & g_n(x_2) \\ \vdots & \vdots  & \ddots & \vdots \\ g_1(x_n) & g_2(x_n) & \cdots & g_n(x_n) \end{pmatrix} $$

are linearly independent. Since $$A\!\,$$ is non-singular for any choice of $$\{ x_1, x_2, \ldots, x_n \}\!\,$$, $$\{g_1,g_2,\dots,g_n\}\!\,$$ is a linearly independent set and we have shown $$ (i) \!\,$$.

Proof of (ii)
By hypothesis, $$g(x) \!\,$$ is a linear combination of $$\{g_1,g_2,\dots,g_n\}\!\,$$ i.e.


 * $$g(x)=\alpha_1g_1(x)+\alpha_2g_2(x)+\cdots+\alpha_ng_n(x) \!\,$$

for $$\alpha_i\!\,$$ not all zero.

Assume for the sake of contradiction that $$g(x) \!\,$$ has $$ n \!\,$$ zeros at $$\hat{x_1}, \hat{x_2}, \ldots, \hat{x_n} \!\,$$

This implies the following $$ n \!\,$$ equations:

$$ \begin{align} g(\hat{x_1}) &=0=\alpha_1 g_1(\hat{x_1})+\alpha_2 g_2 (\hat{x_1}) + \cdots+ \alpha_n g_n(\hat{x_1})\\ g(\hat{x_2}) &=0=\alpha_1 g_1(\hat{x_2})+\alpha_2 g_2 (\hat{x_2}) + \cdots+ \alpha_n g_n(\hat{x_2})\\ &\vdots \\ g(\hat{x_n}) &=0=\alpha_1 g_1(\hat{x_n})+\alpha_2 g_2 (\hat{x_n}) + \cdots+ \alpha_n g_n(\hat{x_n}) \end{align} $$

Rewriting the equations in matrix form yields

$$ \begin{pmatrix} g_1(\hat{x_1}) & g_2(\hat{x_1}) & \cdots & g_n(\hat{x_1}) \\ g_1(\hat{x_2}) & g_2(\hat{x_2}) & \cdots & g_n(\hat{x_2}) \\ \vdots      &  \vdots        & \ddots &  \vdots        \\ g_1(\hat{x_n}) & g_2(\hat{x_n}) & \cdots & g_n(\hat{x_n}) \end{pmatrix} \begin{pmatrix} \alpha_1 \\ \alpha_2 \\ \vdots \\ \alpha_n \end{pmatrix} =\begin{pmatrix} 0 \\ 0 \\ \vdots \\ 0 \end{pmatrix} $$

Since $$ \alpha_i \!\,$$ are not all zero, this implies that the columns of $$ A \!\,$$, $$ \{ g_1, g_2, \ldots, g_n \} $$, are linearly dependent, a contradiction.

Hence, $$g \!\,$$ has at most $$n-1\!\,$$ zeros, and we have shown $$ (ii) \!\,$$.

Solution 1b
We have to prove:

(i) $$\{1,x,x^2,\dots,f\}\!\,$$ is a linearly independent set

(ii)any linear combination of this set has at most $$m\!\,$$ zeros.

Proof of (i)
If we evaluate $$g_1,g_2,\dots,g_{m+1}\!\,$$ at $$ m+1\!\,$$ distinct points, $$ \{x_1, x_2, \ldots, x_{m+1} \} $$, we have the following Van Der Monde matrix whose determinant is non-zero.

$$ \begin{pmatrix} 1     & 1      & \cdots & 1 \\ x_1   & x_1^2  & \cdots & x_1^m \\ \vdots & \vdots & \ddots & \vdots \\ x_{m+1}   & \cdots & \cdots & x_{m+1}^m \end{pmatrix} $$

Hence, $$g_1,g_2,\dots,g_{m+1}\!\,$$ are linearly independent.

Assume for the sake of contradiction that $$f\!\,$$ is a linear combination of $$g_1,g_2,\dots,g_{m+1}\!\,$$, that is

$$ \begin{align} f(x)&=\beta_0g_1(x)+\beta_1g_2(x)+\dots+\beta_mg_{m+1}(x) \\ &=\beta_0\cdot 1+\beta_1x+\beta_2x^2+\dots+\beta_mx^m \end{align} $$.

Hence, $$ f(x) \!\,$$ is a polynomial of degree $$ m \!\,$$. Taking $$ m+1 \!\,$$ derivatives of $$ f(x) \!\,$$ yields


 * $$f^{(m+1)}(x) = 0\!\,$$

which contradicts the hypothesis. Therefore $$\{1,x,x^2,\dots,f\}\!\,$$ is a linearly independent set.

Proof of (ii)
Let $$h(x)=\beta_0g_1(x)+\beta_1g_2(x)+\dots+\beta_mg_{m+1}(x)+\beta_{m+1}f(x)\!\,$$.

Suppose $$h\!\,$$ has $$m+2\!\,$$ (or more) zeros in $$[a,b]\!\,$$. By Rolle's Theorem, the (n+1)st derivative of f vanishes on the given interval, a contradiction.

(i) and (ii) show that $$\{1,x,x^2,\dots,f\}\!\,$$ is a Chebyshev system.

Solution 2a
By the Weierstrass Approximation Theorem, given $$ \epsilon >0 \!\,$$, there exists a polynomial $$ p(x) \!\,$$ such that


 * $$ \max_{a \leq x \leq b} |f(x)- p(x)| \leq \min \{ \frac{\epsilon}{2} \cdot \frac{1}{M}, \frac{\epsilon}{2} \cdot \frac{1}{b-a} \} \quad \quad (3)$$

Let


 * $$ I(f) = \int_a^b f(x) dx $$

Adding and subtracting $$I(p) \!\,$$ and $$ I_n(p) \!\, $$, yields,


 * $$ I(f)-I_n(f)=[I(f)-I(p)]+[I(p)-I_n(p)]+[I_n(p)-I_n(f)] \!\,$$

By the triangle inequality and equation (2) and (3),

$$ \begin{align} &\leq \int_a^b |f(x)-p(x)|dx + |I(p)-I_n(p)| + \sum_{k=1}^n |w_{n,k}| |f(x_n,k)-p(x_n,k)| \\ &\leq \frac{\epsilon}{2(b-a)} \int_a^b dx + |I(p)-I_n(p)| + \frac{\epsilon}{2M} \sum_{k=1}^n |w_{n,k}| \\ &\leq \frac{\epsilon}{2(b-a)} (b-a) + |I(p)-I_n(p)| + \frac{\epsilon}{2M} M \\ &= \epsilon + |I(p)-I_n(p)|
 * I(f)-I_n(f)| &\leq |I(f-p)|+|I(p)-I_n(p)| + |I_n(p-f)| \\

\end{align} $$

By equation (1), when $$ n \rightarrow \infty \!\,$$ ,


 * $$|I(p)-I_n(p)|=0 \!\,$$

Hence for arbitrary small $$ \epsilon >0 \!\,$$ as $$ n \rightarrow \infty \!\,$$,


 * $$ |I(f)-I_n(f)| \leq \epsilon $$

i.e.
 * $$ I(f)=\lim_{n \rightarrow \infty} I_n(f) \!\, $$

Solution 2b
For $$k=0 \!\,$$, equation (1) yields,

$$ \begin{align} \lim_{n \rightarrow \infty} I_n(x^0)&=\int_a^b x^0 dx \\ \lim_{n \rightarrow \infty} I_n(1) &=\int_a^b 1 \cdot dx \\ &=(b-a) \end{align} $$

Letting $$f(x) \!\,$$ in equation (0) yields,
 * $$ I_n(1)=\sum_{k=1}^n w_{n,k}\cdot 1=\sum_{k=1}^n w_{n,k} $$

Combining the two above results yields,

$$ \begin{align} \lim_{n \rightarrow \infty} I_n(1) &= \lim_{n \rightarrow \infty} \sum_{k=1}^n w_{n,k} \\ &= (b-a) \\ &\leq M \end{align} $$

Since $$(b-a) \!\,$$ is finite, there exists some number $$ M \!\,$$ such that $$ (b-a)\leq M $$.

Since $$ w_{n,k} >0 \!\, $$,


 * $$ \lim_{n \rightarrow \infty} \sum_{k=1}^n |w_{n,k}| \leq M $$

i.e. equation (2).

Solution 3a
Substituting for $$ M \!\,$$ in $$ (v,Mv) \!\,$$ and expanding the inner product we have,

$$ \begin{align} (v,Mv) &= (v,\frac{1}{2}(A+A^T)v)\\ &= (v,\frac{1}{2}Av+\frac{1}{2}A^Tv) \\ &= (v,\frac{1}{2}Av)+(v,\frac{1}{2}A^Tv) \\ &= \frac{1}{2}(v,Av)+\frac{1}{2}(v,A^Tv) \end{align} $$

From properties of inner products we have,


 * $$ (v,A^Tv)=(Av,v)=(v,Av) \!\, $$

Hence,

$$ \begin{align} (v,Mv) &= \frac{1}{2}(v,Av)+\frac{1}{2}(v,A^Tv) \\ &= \frac{1}{2}(v,Av)+\frac{1}{2}(v,Av) \\ &= (v,Av) \end{align} $$

First Inequality


\frac{(v,Av)}{(v,v)} = \frac{(v,Mv)}{(v,v)} =\frac{v^TMv}{v^Tv} $$

Since $$M\,\!$$ is symmetric, it has a eigenvalue decomposition


 * $$ M=Q^T\Lambda Q \!\,$$

where $$Q\,\!$$ is orthogonal and $$\Lambda\,\!$$ is a diagonal matrix containing all the eigenvalues.

Substitution yields,


 * $$ \frac{(v,Av)}{(v,v)} = \frac{v^TQ^T\Lambda Qv}{v^Tv} $$

Let


 * $$ Qv=x \!\,$$

This implies the following three relationships:



\begin{align} v   &= Q^Tx \\ v^TQ^T &= x^T \\ v^T   &= x^T Q \end{align} $$

Substituting,

\begin{align} \frac{v^TQ^T\Lambda Qv}{v^Tv} &= \frac{x^T\Lambda x}{x^TQQ^Tx} \\ &= \frac{x^T\Lambda x}{x^T x} \end{align} $$

Expanding the numerator we have,

$$ \begin{align} x^T\Lambda x &= \begin{bmatrix} x_1 x_2 \ldots x_n \end{bmatrix} \begin{bmatrix} \lambda_1 & & & \\ & \lambda_2 &&\\ & & \ddots &\\ & & & \lambda_n \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix}\\ &=\begin{bmatrix} x_1 x_2 \ldots x_n \end{bmatrix} \begin{bmatrix} \lambda_1 x_1 \\ \lambda_2 x_2 \\ \vdots \\ \lambda_n x_n \\ \end{bmatrix}\\ &= \lambda_1x_1^2+\lambda_2x^2+\ldots+\lambda_n^2x_n^2 \\ &= \sum_{i=1}^n \lambda_i x_i^2 \end{align} $$

Expanding the denominator yield

x^Tx = \sum_{i=1}^n x_i^2 $$

Substituting,

\begin{align} \frac{x^T\Lambda x}{x^T x} &=\frac{\sum_{i=1}^n \lambda_i x_i^2}{\sum_{i=1}^n x_i^2} \\ &\geq\lambda_{\min}(M) \frac{\sum_{i=1}^n x_i^2}{\sum_{i=1}^n x_i^2} \\ &= \lambda_{\min}(M) \end{align} $$

Second Inequality
From part(a)


 * $$(v,Av)=(v,Mv) >0 \!\,$$

for all real $$v\!\,$$.

Therefore $$M \!\,$$ is positive definite which implies all its eigenvalues are positive. In particular,


 * $$ \lambda_{\min}(M) > 0 \!\, $$

Solution 3c
First, we want to write $$\|r_{k+1}\|^2$$ as a function of $$\alpha \!\,$$ i.e.


 * $$ f(\alpha)= \|r_{k+1}\|^2 \!\,$$

Changing the indices of equation $$r_k \!\,$$ from $$k \!\,$$ to $$k+1 \!\,$$,substituting for $$ b-Ax_k \!\,$$, and applying the definition of norm yields,

$$ \begin{align} \|r_{k+1}\|^2 &=\|b-Ax_{k+1} \|^2 \\ &=\|b-Ax_k-\alpha Ar_k \|^2 \\ &=\| r_k - \alpha A r_k \|^2 \\ &=(r_k-\alpha Ar_k, r_k -\alpha Ar_k ) \\ &=(r_k,r_k)-\alpha(Ar_k,r_k)-\alpha(r_k,Ar_k)+\alpha^2(Ar_k,Ar_k) \end{align} $$

From a property of inner products and since $$ A \!\,$$ is symmetric,


 * $$ (r_k,Ar_k)= (A^Tr_k,r_k)=(Ar_k,r_k) \!\,$$

Hence,


 * $$f(\alpha)=\|r_{k+1} \|^2=(r_k,r_k)-2\alpha(Ar_k,r_k)+\alpha^2(Ar_k,Ar_k) \!\,$$

Next we want to minimize $$ f(\alpha) \!\,$$ as a function of $$\alpha \!\,$$. Taking its derivative with respect to $$\alpha \!\, $$ yields,


 * $$f^{\prime}(\alpha)=-2(Ar_k,r_k)+2\alpha(Ar_k,Ar_k) \!\,$$

Setting $$f^{\prime}(\alpha)=0$$ and solving for $$\alpha \!\,$$ gives

$$ \begin{align} 0         &=-2(Ar_k,r_k)+2\alpha(Ar_k,Ar_k) \\ 0         &=-(Ar_k,r_k)+\alpha(Ar_k,Ar_k) \\ (Ar_k,r_k) &=\alpha(Ar_k,Ar_k) \\ \alpha    &=\frac{(Ar_k,r_k)}{(Ar_k,Ar_k)} \end{align} $$

Substituting for $$\alpha \!\,$$ into $$ \|r_{k+1} \|^2 \!\,$$ gives,

$$ \begin{align} \|r_{k+1}\|^2  &=(r_k,r_k)-2\alpha(Ar_k,r_k)+\alpha^2(Ar_k,Ar_k)\\ &=(r_k,r_k)-2\frac{(Ar_k,r_k)}{(Ar_k,Ar_k)}(Ar_k,r_k)+\frac{(Ar_k,r_k)^2}{(Ar_k,Ar_k)^2}(Ar_k,Ar_k) \\ &=(r_k,r_k)-2\frac{(Ar_k,r_k)^2}{(Ar_k,Ar_k)}+\frac{(Ar_k,r_k)^2}{(Ar_k,Ar_k)} \\ &=(r_k,r_k)-\frac{(Ar_k,r_k)^2}{(Ar_k,Ar_k)} \end{align} $$

By definition of norm,


 * $$ \|r_k\|^2=(r_k,r_k) \!\, $$

Hence


 * $$ \frac{\|r_{k+1}\|^2}{\|r_k\|^2}= 1-\frac{(Ar_k,r_k)^2}{(r_k,r_k)(Ar_k,Ar_k)} \!\,$$

Multiplying and dividing the second term on the right hand side of the above equation by $$(r_k,r_k)\!\,$$ and applying a property of inner product yields,


 * $$ \frac{\|r_{k+1}\|^2}{\|r_k\|^2}= 1-\frac{(Ar_k,r_k)^2(r_k,r_k)}{(r_k,r_k)^2(r_k,A^TAr_k)} \!\,$$

From the result of part (b), we have our desired result:


 * $$ \frac{\|r_{k+1}\|_2}{\|r_k\|_2} \leq \left( 1- \frac{\lambda_{\min}(M)^2}{\lambda_{\max}(A^TA)} \right)^{\frac{1}{2}} \!\,$$