intuitive explanation of sparsity / references

I know it is a vague question, but I am confused by why/when we actually want sparsity of a matrix. For example, interior-point methods work better when constraint matrix is sparse. Similarly, it is easier to invert a sparse matrix. We also make use of this property to reduce dimensions.

I would be glad if anyone could provide a general explanation regarding different applications, or point me to references. Thank you..

-