Matrices are compact representations of linear systems of equations.
These types of problems are called "linear" because they are closely related to straight lines (and flat planes in higher dimensions).
Note: Matrices can be used in a large variety of ways, but in this answer, I will focus on their historical relationship to simple algebra problems. It should be noted, however, that some properties of matrices are easier to understand from other perspectives (i.e. from their applications in geometry, etc.).
All of your questions will be answered by the end, but it will take a little time to motivate and justify those answers. So please bear with me.
THE PROBLEM
Remember in grade school when you first learned to solve problems like the following?
$$
7x+2y=5 \\
3x-4y=7
$$
Well, how exactly did you solve a problem like this?
By Graphing
One way was to solve both equations for $y$, plot them as two straight lines on a Cartesian plane, find their intersection point, and list the ordered pair corresponding to that point as the answer. Here is a Wolfram Alpha page doing exactly that. The intersection point is $(1,-1)$; therefore, the solution is $x=1$ and $y=-1$.
From this perspective, the "linear" nature of the problem is fairly obvious.
By Substitution
Another standard way to solve this problem is by "substitution" - which involves solving one equation for $y$ in terms of $x$ and then plugging it into the other equation to find $x$ (and then $y$). Like this:
$$
7x+2y=5\\2y=5-7x\\y=\frac{5}{2}-\frac{7}{2}x\\ \ \ \ \\ \ \ \ \\ 3x-4y=7\\3x-4(\frac{5}{2}-\frac{7}{2}x)=7\\3x-10+14x=7\\17x=17\\x=1\\ \ \ \ \\ y=\frac{5}{2}-\frac{7}{2}(1)\\y=-1
$$
This method has the advantage of being less involved than the graphing method, but it is also more abstract. Calculating the answer is more straightforward, but the connection to geometry is much less apparent. This will be a recurring theme from here on: We will continue to trade obviousness and simplicity for elegant calculations.
By Row Operations
The last method that is normally taught is to perform operations on an entire equation (like multiplying by a number) and then to add it to the other equation. When done thoughtfully, this method dramatically speeds up the problem-solving process. Here's how it could work in this case:
$$
7x+2y=5 \\
3x-4y=7\\
\ \ \ \\ 2(7x+2y=5) \rightarrow 14x+4y=10\\ \ \ \\(14x+4y=10)\\+ (3x-4y=7)\\ \rule{4 cm}{0.4pt} \\17x=17\\ \ \ \\ x=1, \text{etc....}
$$
Matrices: Gauss-Jordan Elimination
If you have studied a little bit of matrix algebra, then that last method should look familiar. The "Row Operations" method is exactly the same idea as Gauss-Jordan Elimination on an augmented matrix.
Gauss-Jordan Elimination is significantly more abstract than the previous methods because the variables $x$ and $y$ no longer appear in the problem itself. However, all of the coefficents are still there, and that is what matters. The objective in this case is to get the matrix into Reduced-Row Echelon Form. Here is a quick demonstration:
$$\text{Start:}\ \left(\begin{array}{cc|c}7&2&5\\ 3 & -4 & 7 \end{array}\right)\\ \text{Top Row x2:} \ \left(\begin{array}{cc|c} 14 & 4 & 10 \\ 3 & -4 & 7 \end{array}\right)\\ \text{Add Top to Bottom:} \ \left(\begin{array}{cc|c} 14 & 4 & 10 \\ 17 & 0 & 17 \end{array}\right)\\ \text{Bottom Row $\div$17:} \ \left(\begin{array}{cc|c}14 & 4 & 10 \\ 1 & 0 & 1 \end{array}\right)\\ \text{Bottom Row x14:} \ \left(\begin{array}{cc|c}14 & 4 & 10 \\ 14 & 0 & 14 \end{array}\right)\\ \text{Subtract Bottom from Top:} \ \left(\begin{array}{cc|c} 0 & 4 & -4 \\ 14 & 0 & 14 \end{array}\right)\\ \text{Top Row $\div$4:} \ \left(\begin{array}{cc|c} 0 & 1 & -1 \\ 14 & 0 & 14 \end{array}\right)\\ \text{Bottom Row $\div$14:} \ \left(\begin{array}{cc|c} 0 & 1 & -1 \\ 1 & 0 & 1 \end{array}\right)\\ \text{Switch Rows:} \ \left(\begin{array}{cc|c} 1 & 0 & 1 \\ 0 & 1 & -1 \end{array}\right)
$$
Matrices: By Inversion
Now that we can see some connection between matrices and linear systems of equations, we might naturally ask how to represent this problem as a matrix equation and whether that matrix equation can be easily solved.
First, let's set up the matrix equation:
$$ \textbf{A}\vec{x}=\vec{b}\\ \ \ \\ \text{Let, } \textbf{A}=\left(\begin{array}{cc} 7 & 2 \\ 3 & -4 \end{array}\right), \ \vec{x}=\left(\begin{array}{c} x \\ y \end{array}\right), \ \text{and} \ \vec{b}=\left(\begin{array}{c} 5 \\ 7 \end{array}\right)\\ \ \ \\ \therefore \ \left(\begin{array}{cc} 7 & 2 \\ 3 & -4 \end{array}\right) \left(\begin{array}{c} x \\ y \end{array}\right) = \left(\begin{array}{c} 5 \\ 7 \end{array}\right)
$$
From here, it is obvious that performing matrix multiplication between the matrix and vector on the left-hand side of the equation results in the original system of equations from the very beginning:
$$\left(\begin{array}{cc} 7 & 2 \\ 3 & -4 \end{array}\right) \left(\begin{array}{c} x \\ y \end{array}\right) = \left(\begin{array}{c} 5 \\ 7 \end{array}\right)\\ \left(\begin{array}{c} 7x+2y \\ 3x-4y \end{array}\right) = \left(\begin{array}{c} 5 \\ 7 \end{array}\right)
$$
Therefore, one can see that a vector is just an ordered pair turned on its side. The top entry is the $x$-coordinate while the bottom entry is the $y$-coordinate. Thus, our goal in this problem is to determine the components of the unknown vector $\vec{x}=\left(\begin{array}{c} x \\ y \end{array}\right)$. To do that, we must isolate $\vec{x}$ (just like we would if it were a normal variable rather than a variable vector).
Isolating $\vec{x}$ means finding the inverse of $\textbf{A}$. The inverse of a matrix is unique, so I will write down the general form of the inverse for a 2x2 matrix and that will suffice to cover our example.
$$\textbf{A}=\left(\begin{array}{cc} a & b \\ c & d \end{array}\right)\\ \textbf{A}^{-1}=\frac{1}{ad-bc}\left(\begin{array}{cc} d & -b \\ -c & a \end{array}\right)\\
$$
I recommend checking for yourself to see that
$$\textbf{A}\textbf{A}^{-1}=\textbf{A}^{-1}\textbf{A}=\textbf{I}\\ \ \ \\
\frac{1}{ad-bc}\left(\begin{array}{cc} d & -b \\ -c & a \end{array}\right)\left(\begin{array}{cc} a & b \\ c & d \end{array}\right)=\left(\begin{array}{cc} 1 & 0 \\ 0 & 1 \end{array}\right)
$$
For convenience (and because it has many other applications), we define $ad-bc$ to be the "determinant of $\textbf{A}$."
In this situation, it is relevant because it is the factor by which the inverse matrix must be divided so as to return the identity matrix when multiplied by $\textbf{A}$.
Now, finally, to solve our problem we multiple the original equation by the inverse matrix and we are done.
$$\textbf{A}\vec{x}=\vec{b}\\ \ \ \\
\textbf{A}^{-1}\textbf{A}\vec{x}=\textbf{A}^{-1}\vec{b}\\ \ \ \\
\textbf{I} \vec{x}=\textbf{A}^{-1}\vec{b}\\ \ \ \\
\left(\begin{array}{cc} 1 & 0 \\ 0 & 1 \end{array}\right) \left(\begin{array}{c} x \\ y \end{array}\right)=-\frac{1}{34}\left(\begin{array}{cc} -4 & -2 \\ -3 & 7 \end{array}\right)\left(\begin{array}{c} 5 \\ 7 \end{array}\right)\\ \ \ \\
\left(\begin{array}{c} x \\ y \end{array}\right)=\left(\begin{array}{c} 1 \\ -1 \end{array}\right)
$$
As you can see, the algebra is now much more complicated than the basic methods that you learned in grade school. However, this trade-off comes with the advantage of being a much more elegant-looking solution.
But Why?
There are two main reasons for using matrix algebra to solve linear systems of equations. First, the theory of matrices is very broad. It generalizes specific problems into much larger classes of problem types. By generalizing in this way, we can identify similarities between apparently unrelated problems and, therefore, relate their solutions to one another. Second, computers are really good with matrices. If a problem can be solved with a matrix, then a computer can create an approximate answer very, very quickly. And, guess what, many of the world's most pressing problems can be modeled with matrices.