What is the logic behind Jacobi iterative method? The book I follow and on net also, all that I can find is the algorithm to find the solution, but I don't quite understand the physical significance or logic behind the algorithm.
Can someone please help.
 A: Let me give you an example. 
Suppose we want to solve the following equation in an iterative way:
$$\begin{cases}2x+y=6\\
x+2y=6\end{cases}$$
We write it in the following form to get a fixed point iteration:
$$\begin{cases}x=-\frac{1}{2}y+3\\
y=-\frac{1}{2}x+3\end{cases}$$
Given initial guess $x^{(0)}=\begin{pmatrix}0\\0\end{pmatrix}$, we plug it into the right hand side of the system to get the next approximate solution $x^{(1)}$ and continue. You will eventually get close to the true solution $\begin{pmatrix}2\\2\end{pmatrix}$.
Here is a picture to illustrate the iteration:

If we switch the equations of the original problem,
$$\begin{cases}x+2y=6\\
2x+y=6\end{cases}$$
it wouldn't work. You can try again using the initial guess $x^{(0)}=\begin{pmatrix}0\\0\end{pmatrix}$. 
The following picture shows the second one:

The problem is the slopes $-2$ and $-2$ amplify the solutions and make them go to infinity. 
So the Jacobi method is really a fixed point iteration, where we want the "slope" (derivative) of the right hand side less than 1. 
Basically, we solve each equation in the system for its diagonal element, then iteratively plug in the old numbers to get new numbers. 
If the "slope" is less than 1 (strictly diagonally dominant), then the method converges to the true solution. 
I hope this helps.
A: You are trying to solve $A \textbf{x} = \textbf{b}$.
The matrix $A$ could be written
$A = D + N$ where $D$ is diagonal.
If the sequence $\textbf{x}^k$ generated by $D\textbf{x}^{k+1}  = b - N\textbf{x}^k$ converges, then you have $D\textbf{x}^{\infty}  = b - N\textbf{x}^{\infty}$ or $A \textbf{x}^{\infty} = \textbf{b}^{\infty}$.
If $A$ is diagonal, then $N=0$ and there is no iteration to do.
The condition for convergence is $|| D^{-1} N || < 1$. Intuitively, not super accurate but helpful, think that in the iteration $\textbf{x}^{k+1} = D^{-1} b- D^{-1} N\textbf{x}^{k}$, the error term $D^{-1} N\textbf{x}^{k}$ gets smaller after each iteration and the equation approaches $\textbf{x} \approx D^{-1} b $.  There is a theorem for this, look up if you want rigor.
