This is what helped me understand it most:
from here

primarily the picture on the right side.
Let the pink left side of the rectangle be a vector v you're projecting onto.
Let the black line labeled with a '1' be a vector u that you're projecting onto v.
The bit of pink on the top left is the orthogonal "line of projection" from the tip of u to the span of v.
The angle $\alpha - \beta$ in the pink triangle is the angle between v and u.
Imagine you're tilting the diagram a bit clockwise so that the long side ($cos\beta$) of the orange triangle is horizontal.
Now the $cos\beta$ side of the orange triangle is the x component of u in an imagined cartesian coordinate system, and the $sin\beta$ side of the orange triangle is the y component.
The blue side on the right is equal in length to the pink side on the left since this is a rectangle. On the blue side we can see the length is made up of $cos\beta \cdot cos\alpha + sin\beta \cdot sin\alpha$, which, if u and v are unit length, is the dot product.
Here's another diagram of the same scenario with a "normal" axis, and then also a rotated grid corresponding to v, the orange vector being projected onto

$cos\beta$ is the length of the x component of u, a unit vector $\beta$ counter-clockwise from the x axis.
$cos\alpha$ is the same for v. It's also the factor by which a vector pointing along the x axis (such as the x part of u) is scaled when it's projected onto v. This is because: say you have a right triangle with hypotenuse of length 1 lying on the x axis, with another side aligned with v. That side will have angle $\alpha$ with the hypotenuse (because v is $\alpha$ counter-clockwise from the x axis) and so its length will be $cos\alpha$.
So the x component of u is of length $cos\beta$, then when being "scaled" by projecting onto $v$, it becomes $cos\beta \cdot cos\alpha$.
The case with y components and $sin$ is analogous.