Contrary to popular belief, it is actually possible to "divide" by something that could be $0$ and get meaningful answers, but doing so requires that you be very, very careful. So careful, in fact, that it isn't worth it and you should just analyze what happens in the cases when $x=0$ and when $x\neq0$ (Incidentally, if you can avoid dividing by something that you don't know is not $0$, it's worth making sure there isn't something else you could do to figure out $x$).
Nevertheless, if you want to feel badass and divide by things that could be $0$, what you need to do is understand well how fractions work.
What is a fraction? A fraction is an expression of the form $\dfrac ba$ where $b$ and $a$ are numbers. We say that a fraction $\dfrac ba$ stands for a number if when you multiply that number by $a$, you get $b$. There are three types of fractions:
- $\dfrac00$ which can stand for any number $a$ since $0a=0$ for all numbers $a$. We say that $\dfrac00$ is an "indeterminate" expression.
- $\dfrac b0$ where $b\neq0$, which cannot stand for any number since $0x=0\neq b$ for all numbers $a$. We might say that $\frac b0$ is an "undefined" expression.
- $\dfrac ba$ where $a\neq0$, which stands for exactly one number since if $ax=b$ and $ay=b$, then subtracting one equation from the other gives $a(y-x)=0$, and since $a\neq0$ we must have $y-x=0$, so $y=x$.
(the reason why "division by $0$" is usually disallowed outright is that the fractions with zeroes in the denominator are all of cases 1. and 2., neither of which stand for exactly one number, whereas the fractions with non-zero denominator are case 3.: each such fraction stands for exactly one number).
Now, when you did your division algebra and divided by $5x=8x$ by $x$ (which is only "allowed" if $x$ is non-zero), it seemed like you got a contradiction that $5=8$. However, if you pay attention, you will notice that the contradiction comes not from assuming that $x\neq0$ so that division would be allowed, but from assuming that the resulting fractions $\dfrac{5x}x=\dfrac{8x}x$ are of type 3., and so assuming that each should stand for a unique number.
If we don't assume that the resulting fractions are of type 3. (why should we assume such a thing? we don't want to assume anything about $x$, we want to find it!), then there is nothing contradictory about the fact that you get $5=\dfrac{5x}x=\dfrac{8x}x=8$. There is nothing contradictory because all that says is that each of the two fractions stands for two different numbers, and this can only happen if both fractions are of type 1. Hence, both fractions are really just the indeterminate fraction $\dfrac00$, which means that both fractions are really just the indeterminate fraction $\dfrac00$, so we get $x=0$ and $5x=0=8x$.
Again, doing this kind of argument right requires that you are very careful, and for schoolwork you should just check separately what happens in the case when what you are dividing by is $0$ and the case when it is not.