What are the benefits of iterative method against LU decomposition? I am learning the iterative approach to solve a linear system. I am wondering about its benefits. It looks like it performs better when the system is very large. 
Could anyone help explain a bit more about the benefits of the iterative method against LU?
 A: LU decomposition is a direct method. In absence of rounding errors (and other types of errors involving float-point operations), Gaussian elimination would always produce an exact answer. Clearly, the more (float-point) operations you do, the more rounding errors you have.
Iterative methods are mainly used to solve large sparse systems of equations (i.e. where most entries are zero).
A few drawbacks of direct methods, according to the book "A first course in numerical methods" (chapter 7, by Ascher and Greif), are:

  
*
  
*The Gaussian elimination (or LU decomposition) process may introduce
  fill-in, i.e., $L$ and $U$ may have nonzero elements in locations where the original matrix $A$ has zeros. If the amount of fill-in is significant, then applying the direct method may become costly. This, in fact, occurs often, in particular when a banded matrix is sparse within its band.
  
*Sometimes we do not really need to solve the system exactly. For example, a few methods for nonlinear systems of equations, in which each iteration involves a linear system to solve, frequently, it is sufficient to solve the linear system within the nonlinear iteration only to a low degree of accuracy. 
Direct methods cannot accomplish this because, by definition, to obtain a solution the process must be completed: there is no notion of an early
  termination or an inexact (yet acceptable) solution.
  
*Sometimes we have a pretty good idea of an approximate guess for the solution. For example, in time-dependent problems, we may solve a linear system at a certain time level and then move on to the next time level. Often the solution for the previous time level is quite close to the solution at the current time level, and it is definitely worth using it as an initial guess. This
  is called a warm start. 
Direct methods cannot make good use of such information, especially
  when $A$ also changes slightly from one-time instance to the next.
  
*Sometimes only matrix-vector products are given. In other words, the matrix is not available explicitly or is very expensive to compute. 
For example, in digital signal processing applications, it is often the case that only input and output signals are given, without the transformation
  itself being explicitly formulated and available.

A: The running time of $LU$ is always $N^3$ and the storage always $N^2$ because all matrix elements are represented and acted on. The system is called dense. For $N=1000$, this is affordable. But for larger systems, you start getting into trouble.
The iterative methods have the advantage that they exploit the sparseness of the matrix (in large physical systems, you don't have a coupling between every pair of variables): storage is proportional to the number of nonzero entries and running-time proportional to the number of nonzero entries times the number of iterations. This can be much lower than $N^2$ and $N^3$ and makes systems of equations with a huge $N$ accessible.
