Numerical Methods Qualification Exam Problems and Solutions (University of Maryland)/Jan05 667

Steepest Descent Method
$$x_{k+1}=x_k-\alpha_k \nabla f(x_k) \!\,$$

Choose $$\alpha_k \in (0,1] \!\,$$ such that $$ \alpha_k \!\,$$ minimizes $$f(x_k-\alpha_k \nabla f(x_k)) \!\,$$ i.e.

$$\alpha_k : \min_{0 < \alpha_k \leq 1} f(x_k - \alpha_k \nabla f(x_k)) \!\,$$

Directional Derivative Negative
To satisfy (2), the directional derivative should be negative i.e.

$$\nabla_{d_k} f(x_k) = \lim_{\alpha_k \rightarrow 0} \frac{f(x_k+\alpha_k d_k)-f(x_k)}{\alpha_k} <0 \!\,$$

which implies

$$\underbrace{f(x_k+\alpha_kd_k)}_{f(x_{k+1})} < f(x_k)\!\,$$

since $$\alpha_k \in (0,1]\!\,$$

Newton's Method
$$x_{k+1}=x_k-H^{-1}(f(x_k))\nabla f(x_k) \!\,$$

Directional Derivative Negative
To have descent, we want the directional derivative to be negative i.e.

$$ \nabla_{\alpha_k d_k} f(x_k) = \nabla f(x_k) \cdot \alpha_k d_k=              - \nabla f(x_k) \cdot H^{-1}(f(x_k)) \nabla f(x_k) < 0 \!\,$$

which implies

$$\nabla f(x_k)^T H^{-1}(f(x_k)) \nabla f(x_k) > 0 \!\,$$

Therefore $$H^{-1}(f(x_k)) \!\,$$ is positive definite and therefore $$H(f(x_k)) \!\,$$ is positive definite.

Solution 4c
We now want $$\tilde{H}(f(x_k))=H(f(x_k))+\gamma_kI \!\,$$ to be positive definite.

Let $$\lambda_i \!\,$$, $$i=1,2,\ldots,n \!\,$$ be the eigenvalues of $$H(f(x_k)) \!\,$$.

Then $$ \lambda_i + \gamma_k \cdot 1\!\,$$, $$ i=1,2,\ldots,n\!\,$$ are eigenvalues of $$\tilde{H}(f(x_k)) \!\,$$.

Since we want $$\tilde{H}(f(x_k)) \!\,$$ to be positive definite, we equivalently have for $$i=1,2,\ldots,n \!\,$$

$$\lambda_i + \gamma_k > 0 \!\,$$

i.e.

$$\gamma_k > -\lambda_i \quad i=1,2,\ldots,n \!\,$$

Solution 5a
For $$x \in [x_n, x_{n+1}] \!\,$$,

$$y(x_{n+1})-y(x_n)=\int_{x_n}^{x_{n+1}} f(y(x))dx \approx h \cdot f\left(\frac{y_{n+1}+y_n}{2}\right) \!\,$$

Solution 5b
$$ \begin{align} &\leq |y(t_n) - y_n| + h \underbrace{\left| f \left( \frac{ y(t_{n+1}) - y(t_{n}) }{2} \right) - f \left( \frac{ y_{n+1} - y_{n} }{2}\right) \right| }_{\leq \frac{\gamma}{2}|y(t_{n+1}) - y(t_{n}) - y_{n+1} + y_{n}|} + \mathcal{O}(h^3) \\ &\leq |e_n| + \frac{h\gamma}{2} \left( |e_{n+1}| + |e_{n}| \right) + \mathcal{O}(h^3) \end{align} $$
 * e_{n+1}| &= |y(t_{n+1}) - y_{n+1}| \\

where $$\gamma \!\,$$ is the Lipschitz constant of $$f \!\,$$. Rearranging terms we get

$$|e_{n+1}| \leq \underbrace{\frac{1+\frac{h\gamma}{2}}{1-\frac{h\gamma}{2}}}_{C}|e_n| + \mathcal{O}(h^3) \!\,$$

In particular, $$|e_1| \leq C \underbrace{|e_0|}_{0} +\mathcal{O}(h^3) \!\,$$

Then $$e_{N} \!\,$$ is given by

$$ \begin{align} e_{N} &= \left( C^{N-1} + C^{N-2} + \cdots + C + 1 \right) \mathcal{O}(h^3)  \\ &= \frac{C^N-1}{C-1} \mathcal{O}(h^3) \\ &\leq K \mathcal{O}(h^2)

\end{align} \!\,$$

Solution 5c
The midpoint method may be rewritten as follows:

$$y(t_{n+1})=y(t_n)+\frac{h}{2}y'(t_{n+1})+\frac{h}{2}y'(t_n) + T_n \!\,$$

which implies

$$y(t_{n+1})-y(t_n)-\frac{h}{2}y'(t_{n+1})-\frac{h}{2}y'(t_n) = T_n \!\,$$

Expanding each term around $$\tau_n=t_n+\frac{h}{2} \!\,$$ yields $$ T_n\!\,$$.

$$ \begin{array}{|c|c|c|c|c|c|} \mbox{Order } h & y(t_{n+1}) & -y(t_n) & -\frac{h}{2}y'(t_{n+1}) & -\frac{h}{2} y'(t_n) & \Sigma \\ \hline &&&&&\\ 0& y(t_n+\frac{h}{2}) & -y(t_n+\frac{h}{2}) & --- &  --- & 0 \\ &&&&&\\ 1& y'(t_n+\frac{h}{2})\frac{h}{2} & -y'(t_n+\frac{h}{2})(-\frac{h}{2}) & -\frac{h}{2}y'(t_n+\frac{h}{2}) & -\frac{h}{2}y'(t_n+\frac{h}{2}) & 0 \\ &&&&&\\ 2&\frac{y(t_n+\frac{h}{2})}{2} \frac{h^2}{4}&-\frac{y(t_n+\frac{h}{2}}{2}\frac{h^2}{4}&-\frac{h}{2}y(t_n+\frac{h}{2})\frac{h}{2}& -\frac{h}{2}y(t_n+\frac{h}{2})(-\frac{h}{2})&0\\ &&&&&\\ 3&O(h^3)&O(h^3)&O(h^3)&O(h^3)&O(h^3)\\ \hline \end{array} $$

Weak Variational Formulation
Multiplying the given equation by a test function $$v \in H_0^1 [0,1] \!\,$$ and integrating from 0 to 1 gives the equivalent weak variational formulation:

Find $$u \in H_0^1 [0,1] \!\,$$ such that for all $$v \in H_0^1 [0,1] \!\,$$ the following holds


 * $$\underbrace{\epsilon \int_0^1 u' v' dx + b \int_0^1 u' v dx}_{a(u,v)} = \underbrace{\int_0^1 v dx}_{f(v)} \!\,$$

Discrete Variational Formulation
Let $$V_h = \{ v \in H_0^1 [0,1] : \left. v_h \right|_{[x_{i-1},x_i]}\mbox{ linear } \!\,$$ for $$i=1,2,\ldots,n \} \!\,$$

Then we have the discrete variational formulation which is an approximation to the weak variational formulation.

Find $$u_h \in V_h \!\,$$ such that for all $$v_h \in V_h \!\,$$


 * $$a(u_h,v_h)=f(v_h) \!\,$$

Define basis for V_h
Let $$\{ \phi_i \}_{i=1}^n \!\,$$ be the linear "hat" functions which defines a basis for $$V_h \!\,$$.

$$ \phi_i(x)= \begin{cases} \frac{x-x_{i-1}}{h} & \mbox{ if } x\in[x_{i-1},x_i] \\ \frac{x_{i+1}-x}{h} & \mbox{ if } x\in[x_i,x_{i+1}] \\ 0                  & \mbox{ otherwise } \end{cases} \!\,$$

Then calculation yields the following: (draw pictures)

$$\int_0^1 \phi_i(x) dx = h \!\,$$

$$ \phi_i'(x)= \begin{cases} \frac{1}{h} & \mbox{ if } x\in[x_{i-1},x_i] \\ -\frac{1}{h} & \mbox{ if } x\in[x_i,x_{i+1}] \\ 0                  & \mbox{ otherwise } \end{cases} \!\,$$

$$ \int_0^1 \phi_i'(x)\phi_j'(x) dx= \begin{cases} \frac{2}{h} & \mbox{ if } i=j \\ -\frac{1}{h} & \mbox{ if } |i-j|=1 \\ 0                  & \mbox{ otherwise } \end{cases} \!\,$$

$$ \int_0^1 \phi_i(x)\phi_j'(x)dx= \begin{cases} 0       & \mbox{ if } i=j \\ \frac12 & \mbox{ if } i-j=1 \\ -\frac12 & \mbox{ if } i-j=-1 \\ 0       &  \mbox{ otherwise } \end{cases} \!\,$$

Discrete Variational Formulation in Matrix Form
Since $$\{ \phi_i \}_{i=1}^n \!\,$$ is a basis for $$V_h \!\,$$,


 * $$u_h=\sum_{i=1}^n u_{hi}\phi_i \!\,$$

Also, the discrete variational formulation may be expressed as


 * $$\epsilon \sum_{i=1}^n u_{hi} \int_0^1 \phi_i'(x)\phi_j'(x)dx+ b \sum_{i=1}^n u_{hi} \int_0^1 \phi_i'(x) \phi_j(x)dx = \int_0^1 \phi_j(x) dx \mbox{ for }j=1,2,\ldots,n\!\,$$

which in matrix form is

$$\underbrace{ \begin{bmatrix} 2\frac{\epsilon}{h} & -\frac{\epsilon}{h}+\frac{b}{2} & 0 & \cdots &\cdots & 0 \\ -\frac{\epsilon}{h}-\frac{b}{2} & 2\frac{\epsilon}{h} & -\frac{\epsilon}{h}+\frac{b}{2} & 0 & \cdots & 0 \\ 0 & \ddots &\ddots & \ddots & \cdots & 0 \\ 0 & \cdots &\cdots & 0 & -\frac{\epsilon}{h}-\frac{b}{2} & 2\frac{\epsilon}{h} \end{bmatrix}}_A \begin{bmatrix} u_1 \\ u_2 \\ \vdots \\ u_n \end{bmatrix} = \begin{bmatrix} h\\ h\\ \vdots\\ h \end{bmatrix} \!\,$$

$$A \!\,$$ is not symmetric. $$A \!\,$$ is positive definite if


 * $$\frac{\epsilon}{h}> \frac{b}{2} \!\,$$

Solution 6b
The first, second, and last rows all yield the same inequality for $$A \!\,$$ to be an $$M \!\,$$-matrix:


 * $$\frac{\epsilon}{h} > \frac{b}{2} \!\,$$

Solution 6c
Substituting $$\epsilon+\frac{b}{2}h \!\,$$ for $$\epsilon \!\,$$ yields the following matrix $$A \!\,$$:

$$ \begin{bmatrix} 2\frac{\epsilon}{h} +b & -\frac{\epsilon}{h} & 0 & \cdots &\cdots & 0 \\ -\frac{\epsilon}{h}-b & 2\frac{\epsilon}{h}+b & -\frac{\epsilon}{h}  & 0 & \cdots & 0 \\ 0 & \ddots &\ddots & \ddots & \cdots & 0 \\ 0 & \cdots &\cdots & 0 & -\frac{\epsilon}{h}-b & 2\frac{\epsilon}{h}+b \end{bmatrix} \!\,$$

All off diagonal entries are $$\leq 0 \!\,$$. Diagonal entries are $$ >0\!\,$$

The first row meets the last condition since


 * $$ 2\frac{\epsilon}{h} +b \geq \frac{\epsilon}{h}\!\,$$

The second row through (n-1) rows meets the last condition since


 * $$ 2\frac{\epsilon}{h} +b \geq \frac{\epsilon}{h}+b + \frac{\epsilon}{b}\!\,$$

The last row meets the last condition since


 * $$ 2\frac{\epsilon}{h} +b \geq \frac{\epsilon}{h}+b\!\,$$